7.3 Problem Submission via Condor

HTCondor, formerly known as Condor is

“a specialized workload management system for compute-intensive jobs.”

This tutorial shows how to submit optimization problem to a HTCondor server via OptServer.

The idea is very simple: since OptServer executes MOSEK using a simple Python script (solve.py), we can instruct OptServer to use a different script that will interface with HTCondor. To this extent we use the script as in Listing 7.11.

Listing 7.11 An example of script to off-load a job from OptServer to a HTCondor server.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import sys
import os,os.path
import subprocess

if __name__ == '__main__':
  workdir  = sys.argv[1]
  probfile = sys.argv[2]
  pidfile = os.path.join(workdir,"PID")

  with open(pidfile,'wt', encoding='ascii')  as f:
     f.write(str(os.getpid))

  r = 1
  try:
      r = subprocess.call(['condor_run',
                           os.path.abspath(os.path.join(os.path.dirname(__file__),"solve.py")),
                           workdir,
                           probfile,
                           '-noPID'])
  finally:
      try:
          os.remove(pidfile)
     except:
          pass

 sys.exit(r)

The script operates as follows:

  • lines 10-11: the job PID is stored in a text file called PID in the working directory;
  • lines 14-24 : a HTCondor process is created, responsible to run the solve.py script.

To tell OptServer to use the script in Listing 7.11 instead of the default solve.py, the cmd option (see Section 9) in the configuration file server.conf must be modified accordingly. In this case the script is available in the script directory of the OptServer distribution. Therefore the configuration file can be simply modified changing the cmd option to

"cmd"   : "${CONFIGDIR}/script/tocondor.py ${TASK}",