10 Technical guidelines¶
This section contains some more in-depth technical guidelines for Optimizer API for Python, not strictly necessary for basic use of MOSEK.
10.1 Memory management and garbage collection¶
Users who experience memory leaks, especially:
memory usage not decreasing after the solver terminates,
memory usage increasing when solving a sequence of problems,
should make sure that the Task
objects are properly garbage collected. Since each Task
object links to a MOSEK task resource in a linked library, it is sometimes the case that the garbage collector is unable to reclaim it automatically. This means that substantial amounts of memory may be leaked. For this reason it is very important to make sure that the Task
object is disposed of, either automatically or manually, when it is not used any more.
The Task
class supports the Context Manager protocol, so it will be destroyed properly when used in a with
statement:
with mosek.Env() as env:
with env.Task(0, 0) as task:
# Build an optimization problem
# ...
If this is not possible, then the necessary cleanup is performed by the methods Task.__del__
and Env.__del__
which should be called explicitly.
10.2 Names¶
All elements of an optimization problem in MOSEK (objective, constraints, variables, etc.) can be given names. Assigning meaningful names to variables and constraints makes it much easier to understand and debug optimization problems dumped to a file. On the other hand, note that assigning names can substantially increase setup time, so it should be avoided in time-critical applications.
Names of various elements of the problem can be set and retrieved using various functions listed in the Names section of Sec. 15.2 (Functions grouped by topic).
Note that file formats impose various restrictions on names, so not all names can be written verbatim to each type of file. If at least one name cannot be written to a given format then generic names and substitutions of offending characters will be used when saving to a file, resulting in a transformation of all names in the problem. See Sec. 16 (Supported File Formats).
10.3 Multithreading¶
Thread safety
Sharing a task between threads is safe, as long as it is not accessed from more than one thread at a time. Multiple tasks can be created and used in parallel without any problems.
Parallelization
The interior-point and mixed-integer optimizers in MOSEK are parallelized. By default MOSEK will automatically select the number of threads. However, the maximum number of threads allowed can be changed by setting the parameter iparam.num_threads
and related parameters. This should never exceed the number of cores.
The speed-up obtained when using multiple threads is highly problem and hardware dependent. We recommend experimenting with various thread numbers to determine the optimal settings. For small problems using multiple threads may be counter-productive because of the associated overhead. Note also that not all parts of the algorithm can be parallelized, so there are times when CPU utilization is only 1 even if more cores are available.
Determinism
By default the optimizer is run-to-run deterministic, which means that it will return the same answer each time it is run on the same machine with the same input, the same parameter settings (including number of threads) and no time limits.
Setting the number of threads
The number of threads the optimizer uses can be changed with the parameter iparam.num_threads
.
The parameter iparam.num_threads
affects only the optimizer. It may be the case that numpy
is consuming more threads. In most cases this can be limited by setting the environment variable MKL_NUM_THREADS
. See the numpy
documentation for more details.
10.4 Efficiency¶
Although MOSEK is implemented to handle memory efficiently, the user may have valuable knowledge about a problem, which could be used to improve the performance of MOSEK This section discusses some tricks and general advice that hopefully make MOSEK process your problem faster.
Reduce the number of function calls and avoid input loops
For example, instead of setting the entries in the linear constraint matrix one by one (Task.putaij
) define them all at once (Task.putaijlist
) or in convenient large chunks (Task.putacollist
etc.)
Use one environment only
If possible share the environment between several tasks. For most applications you need to create only a single environment.
Read part of the solution
When fetching the solution, data has to be copied from the optimizer to the user’s data structures. Instead of fetching the whole solution, consider fetching only the interesting part (see for example Task.getxxslice
and similar).
Avoiding memory fragmentation
MOSEK stores the optimization problem in internal data structures in the memory. Initially MOSEK will allocate structures of a certain size, and as more items are added to the problem the structures are reallocated. For large problems the same structures may be reallocated many times causing memory fragmentation. One way to avoid this is to give MOSEK an estimated size of your problem using the functions:
Task.putmaxnumvar
. Estimate for the number of variables.Task.putmaxnumcon
. Estimate for the number of constraints.Task.putmaxnumbarvar
. Estimate for the number of semidefinite matrix variables.Task.putmaxnumanz
. Estimate for the number of non-zeros in \(A\).Task.putmaxnumqnz
. Estimate for the number of non-zeros in the quadratic terms.
None of these functions changes the problem, they only serve as hints. If the problem ends up growing larger, the estimates are automatically increased.
Do not mix put-
and get-
functions
MOSEK will queue put-
requests internally until a get-
function is called. If put-
and get-
calls are interleaved, the queue will have to be flushed more frequently, decreasing efficiency.
In general get-
commands should not be called often (or at all) during problem setup.
Use the LIFO principle
When removing constraints and variables, try to use a LIFO (Last In First Out) approach. MOSEK can more efficiently remove constraints and variables with a high index than a small index.
An alternative to removing a constraint or a variable is to fix it at 0, and set all relevant coefficients to 0. Generally this will not have any impact on the optimization speed.
Add more constraints and variables than you need (now)
The cost of adding one constraint or one variable is about the same as adding many of them. Therefore, it may be worthwhile to add many variables instead of one. Initially fix the unused variable at zero, and then later unfix them as needed. Similarly, you can add multiple free constraints and then use them as needed.
Do not remove basic variables
When performing re-optimizations, instead of removing a basic variable it may be more efficient to fix the variable at zero and then remove it when the problem is re-optimized and it has left the basis. This makes it easier for MOSEK to restart the simplex optimizer.
10.5 The license system¶
MOSEK is a commercial product that always needs a valid license to work. MOSEK uses a third party license manager to implement license checking. The number of license tokens provided determines the number of optimizations that can be run simultaneously.
By default a license token remains checked out from the first optimization until the end of the MOSEK session, i.e.
a license token is checked out when
Task.optimize
is first called, andit is returned when the MOSEK environment is deleted.
Calling Task.optimize
from different threads using the same MOSEK environment only consumes one license token.
Starting the optimization when no license tokens are available will result in an error.
Default behaviour of the license system can be changed in several ways:
Setting the parameter
iparam.cache_license
toonoffkey.off
will force MOSEK to return the license token immediately after the optimization completed.Setting the license wait flag with the parameter
iparam.license_wait
will force MOSEK to wait until a license token becomes available instead of returning with an error. The wait time between checks can be set withEnv.putlicensewait
.Additional license checkouts and checkins can be performed with the functions
Env.checkinlicense
andEnv.checkoutlicense
.Usually the license system is stopped automatically when the MOSEK library is unloaded. However, when the user explicitly unloads the library (using e.g. FreeLibrary), the license system must be stopped before the library is unloaded. This can be done by calling the function
Env.licensecleanup
as the last function call to MOSEK.
10.6 Deployment¶
When redistributing a Python application using the MOSEK Optimizer API for Python 10.2.5, the following shared libraries from the MOSEK bin
folder are required:
Linux :
libmosek64
,libmosekxx
,libtbb
,Windows :
mosek64
,mosekxx
,tbb
,svml_dispmd
,OSX :
libmosek64
,libmosekxx
,libtbb
.
Furthermore, the folder python/3/mosek
must be in the PYTHONPATH
.
By default the MOSEK Python API will look for the binary libraries in the MOSEK module directory, i.e. the directory containing __init__.py
. Alternatively, if the binary libraries reside in another directory, the application can pre-load the mosekxx
library from another location before mosek
is imported, e.g. like this
import ctypes ; ctypes.CDLL('my/path/to/mosekxx.dll')