from mosek.fusion import *
print("Using Mosek version {ver}".format(ver=Model.getVersion()))
Using Mosek version 9.3.10
N, gamma = 8, np.sqrt(0.05)
N, m, S, gamma
(8,
array([0.072 , 0.1552, 0.1754, 0.0898, 0.429 , 0.3929, 0.3217, 0.1838]),
array([[0.0946, 0.0374, 0.0349, 0.0348, 0.0542, 0.0368, 0.0321, 0.0327],
[0.0374, 0.0775, 0.0387, 0.0367, 0.0382, 0.0363, 0.0356, 0.0342],
[0.0349, 0.0387, 0.0624, 0.0336, 0.0395, 0.0369, 0.0338, 0.0243],
[0.0348, 0.0367, 0.0336, 0.0682, 0.0402, 0.0335, 0.0436, 0.0371],
[0.0542, 0.0382, 0.0395, 0.0402, 0.1724, 0.0789, 0.07 , 0.0501],
[0.0368, 0.0363, 0.0369, 0.0335, 0.0789, 0.0909, 0.0536, 0.0449],
[0.0321, 0.0356, 0.0338, 0.0436, 0.07 , 0.0536, 0.0965, 0.0442],
[0.0327, 0.0342, 0.0243, 0.0371, 0.0501, 0.0449, 0.0442, 0.0816]]),
0.22360679774997896)
Recall that we cast the risk constraint in conic form using any $G$ such that
Then
So:
G = np.linalg.cholesky(S)
print(G)
print( np.max(np.abs(S - G @ G.T)) )
[[0.3076 0. 0. 0. 0. 0. 0. 0. ] [0.1216 0.2504 0. 0. 0. 0. 0. 0. ] [0.1135 0.0994 0.1991 0. 0. 0. 0. 0. ] [0.1131 0.0916 0.0585 0.2088 0. 0. 0. 0. ] [0.1762 0.067 0.0645 0.0496 0.3609 0. 0. 0. ] [0.1196 0.0869 0.0738 0.0368 0.1258 0.2154 0. 0. ] [0.1044 0.0915 0.0646 0.094 0.1016 0.0564 0.2252 0. ] [0.1063 0.0849 0.019 0.0775 0.0571 0.062 0.0334 0.2202]] 2.7755575615628914e-17
The quadratic cone (second-order cone, SOC) $\mathcal{Q}^k$ in dimension $k$ is the convex set defined as:
Therefore the bound
in conic form becomes
M = Model("simple-MVO")
x = M.variable("x", N, Domain.greaterThan(0.0))
M.constraint('budget', Expr.sum(x), Domain.equalsTo(1))
mosek.fusion.LinearConstraint
M.constraint('risk', Expr.vstack(gamma, Expr.mul(G.T, x)), Domain.inQCone())
mosek.fusion.ConicConstraint
M.objective('max-return', ObjectiveSense.Maximize, Expr.dot(m, x))
simpleMVO = M.clone()
M.solve()
# Always check the status
print(M.getPrimalSolutionStatus())
# Actual solution values
X = x.level()
print("Optimal x = {}".format(X))
SolutionStatus.Optimal Optimal x = [6.4748e-08 9.1128e-02 2.6889e-01 1.1835e-07 2.5027e-02 3.2222e-01 1.7692e-01 1.1581e-01]
Let us check that the constraints of the original problem before conic reformulation
are satisfied:
print("budget = {}".format(np.sum(X)))
print("risk = {0}, gamma^2 = {1}".format( X.T @ S @ X , gamma**2))
print("min(x) = {}".format(np.min(X)))
budget = 0.9999999928119866 risk = 0.050000002124506414, gamma^2 = 0.049999999999999996 min(x) = 6.474806707598256e-08
M.writeTask("simple-mvo.ptf")
!cat simple-mvo.ptf
Task simple-MVO
Written by MOSEK v9.3.10
problemtype: Conic Problem
numvar: 18
numcon: 10
numcone: 1
numanz: 54
Objective max-return
Maximize + 7.2e-2 'x[0]' + 1.552e-1 'x[1]' + 1.754e-1 'x[2]' + 8.98e-2 'x[3]'
+ 4.29e-1 'x[4]' + 3.929e-1 'x[5]' + 3.217e-1 'x[6]' + 1.838e-1 'x[7]'
Constraints
'budget[]' [1] + 'x[0]' + 'x[1]' + 'x[2]' + 'x[3]' + 'x[4]' + 'x[5]' + 'x[6]'
+ 'x[7]'
[SOC(9)]
'risk[0]' [] + 2.23606797749979e-1 '1.0'
'risk[1]' [] + 3.07571129984594e-1 'x[0]' + 1.2159788859856e-1 'x[1]' + 1.13469687489031e-1 'x[2]'
+ 1.1314455944465e-1 'x[3]' + 1.76219400054598e-1 'x[4]' + 1.19647120332273e-1 'x[5]'
+ 1.04366102246358e-1 'x[6]' + 1.06316870512645e-1 'x[7]'
'risk[2]' [] + 2.50427541393458e-1 'x[1]' + 9.94392447525178e-2 'x[2]' + 9.16107722715342e-2 'x[3]'
+ 6.6973835744785e-2 'x[4]' + 8.68561128287541e-2 'x[5]' + 9.14807620523957e-2 'x[6]'
+ 8.49431053185731e-2 'x[7]'
'risk[3]' [] + 1.99089092177825e-1 'x[2]' + 5.85256382727785e-2 'x[3]' + 6.45169052766933e-2 'x[4]'
+ 7.37698495906825e-2 'x[5]' + 6.45983813834224e-2 'x[6]' + 1.90345922363295e-2 'x[7]'
'risk[4]' [] + 2.08759490171472e-1 'x[3]' + 4.95800968447033e-2 'x[4]' + 3.68280080083627e-2 'x[5]'
+ 9.40328919312598e-2 'x[6]' + 7.74820314695962e-2 'x[7]'
'risk[5]' [] + 3.60888641135207e-1 'x[4]' + 1.25837794334399e-1 'x[5]' + 1.01560422871808e-1 'x[6]'
+ 5.70988313908233e-2 'x[7]'
'risk[6]' [] + 2.15423162392141e-1 'x[5]' + 5.64407109199148e-2 'x[6]' + 6.20118358186446e-2 'x[7]'
'risk[7]' [] + 2.25219399454967e-1 'x[6]' + 3.33853377327792e-2 'x[7]'
'risk[8]' [] + 2.20216452377311e-1 'x[7]'
Variables
'1.0' [1]
'x[0]' [0;+inf]
'x[1]' [0;+inf]
'x[2]' [0;+inf]
'x[3]' [0;+inf]
'x[4]' [0;+inf]
'x[5]' [0;+inf]
'x[6]' [0;+inf]
'x[7]' [0;+inf]
Solutions
Solution interior
SolutionStatus optimal optimal
ProblemStatus feasible feasible
Objective 2.76845242001663e-1 2.76845247668972e-1
Variables
'1.0' [fixed] 1 0 -6.32357185643455e-1
'x[0]' [at_lower] 6.47480670759826e-8 -2.11268152548724e-2 0
'x[1]' [super_basic] 9.11283996669759e-2 -2.93211981939527e-8 0
'x[2]' [super_basic] 2.68889911532839e-1 -9.56047292922639e-9 0
'x[3]' [at_lower] 1.18347549518961e-7 -1.23829757177462e-2 0
'x[4]' [super_basic] 2.50271241067324e-2 -1.22981025395267e-7 0
'x[5]' [super_basic] 3.22220843219148e-1 -7.86978423070218e-9 0
'x[6]' [super_basic] 1.76919362041914e-1 -1.68888857245084e-8 0
'x[7]' [super_basic] 1.15814169148761e-1 -2.55461049385523e-8 0
Constraints
'budget[]' [fixed] 1 -3.55511937974483e-1 0
'risk[0]' [fixed] 0 -2.82798730631844 0
'risk[1]' [fixed] 0 0 -1.45865040302802
'risk[2]' [fixed] 0 0 -1.33109623292075
'risk[3]' [fixed] 0 0 -1.17051182585188
'risk[4]' [fixed] 0 0 -4.89600976458603e-1
'risk[5]' [fixed] 0 0 -9.38041920660027e-1
'risk[6]' [fixed] 0 0 -1.09484114524428
'risk[7]' [fixed] 0 0 -5.52773367795219e-1
'risk[8]' [fixed] 0 0 -3.22596896217229e-1
@K0
'risk[0]' [fixed] -2.23606797749979e-1 0 0 -2.82798730631844
'risk[1]' [fixed] -1.15332353577058e-1 0 0 1.45865040309823
'risk[2]' [fixed] -1.0524462728839e-1 0 0 1.33109623298474
'risk[3]' [fixed] -9.25510909520995e-2 0 0 1.17051182590811
'risk[4]' [fixed] -3.87173800877269e-2 0 0 4.89600976482131e-1
'risk[5]' [fixed] -7.41604239497834e-2 0 0 9.38041920705145e-1
'risk[6]' [fixed] -8.65811368468448e-2 0 0 1.09484114529686
'risk[7]' [fixed] -4.37121676223292e-2 0 0 5.527733678218e-1
'risk[8]' [fixed] -2.5504185464966e-2 0 0 3.22596890233254e-1
M = simpleMVO.clone()
M.setLogHandler(sys.stdout)
M.solve()
Problem Name : simple-MVO Objective sense : max Type : CONIC (conic optimization problem) Constraints : 10 Cones : 1 Scalar variables : 18 Matrix variables : 0 Integer variables : 0 Optimizer started. Presolve started. Linear dependency checker started. Linear dependency checker terminated. Eliminator started. Freed constraints in eliminator : 0 Eliminator terminated. Eliminator - tries : 1 time : 0.00 Lin. dep. - tries : 1 time : 0.00 Lin. dep. - number : 0 Presolve terminated. Time: 0.00 Problem Name : simple-MVO Objective sense : max Type : CONIC (conic optimization problem) Constraints : 10 Cones : 1 Scalar variables : 18 Matrix variables : 0 Integer variables : 0 Optimizer - threads : 64 Optimizer - solved problem : the primal Optimizer - Constraints : 8 Optimizer - Cones : 1 Optimizer - Scalar variables : 16 conic : 9 Optimizer - Semi-definite variables: 0 scalarized : 0 Factor - setup time : 0.00 dense det. time : 0.00 Factor - ML order time : 0.00 GP order time : 0.00 Factor - nonzeros before factor : 36 after factor : 36 Factor - dense dim. : 0 flops : 6.00e+02 ITE PFEAS DFEAS GFEAS PRSTATUS POBJ DOBJ MU TIME 0 1.0e+00 1.6e+00 1.2e+00 0.00e+00 0.000000000e+00 2.236067977e-01 1.0e+00 0.01 1 1.9e-01 3.1e-01 5.2e-02 7.17e-01 3.603706929e-01 5.403824676e-01 1.9e-01 0.01 2 7.2e-02 1.1e-01 1.1e-02 2.13e+00 3.359510718e-01 3.727924129e-01 7.2e-02 0.01 3 3.9e-02 6.1e-02 4.4e-03 1.55e+00 2.941386998e-01 3.098471290e-01 3.9e-02 0.01 4 9.6e-03 1.5e-02 5.2e-04 1.34e+00 2.835138976e-01 2.868601152e-01 9.6e-03 0.01 5 3.2e-03 5.1e-03 1.0e-04 1.09e+00 2.790266665e-01 2.801001191e-01 3.2e-03 0.01 6 5.2e-04 8.2e-04 6.6e-06 9.93e-01 2.771525777e-01 2.773245159e-01 5.2e-04 0.01 7 1.1e-04 1.8e-04 6.6e-07 9.95e-01 2.768913976e-01 2.769285069e-01 1.1e-04 0.02 8 1.6e-06 2.6e-06 1.1e-09 1.00e+00 2.768465817e-01 2.768471545e-01 1.6e-06 0.02 9 1.6e-08 2.6e-08 1.1e-12 1.00e+00 2.768452420e-01 2.768452477e-01 1.6e-08 0.02 Optimizer terminated. Time: 0.02 Interior-point solution summary Problem status : PRIMAL_AND_DUAL_FEASIBLE Solution status : OPTIMAL Primal. obj: 2.7684524200e-01 nrm: 1e+00 Viol. con: 7e-09 var: 0e+00 cones: 3e-09 Dual. obj: 2.7684524767e-01 nrm: 3e+00 Viol. con: 0e+00 var: 6e-09 cones: 0e+00
There are modeling tools that allow writing the quadratic model directly and still use MOSEK.
import cvxpy as cp
x = cp.Variable(N)
prob = cp.Problem( cp.Maximize(m.T @ x),
[ cp.sum(x) == 1,
cp.quad_form(x, S) <= gamma**2,
x >= 0 ] )
prob.solve(cp.MOSEK, save_file="simple-mvo-cvxpy.ptf")
print("cvxpy {}".format(x.value)) # CVXPY + MOSEK solution
print("Fusion {}".format(X)) # Fusion solution for comparison
cvxpy [1.9719e-08 9.1143e-02 2.6889e-01 3.5566e-08 2.5071e-02 3.2219e-01 1.7690e-01 1.1581e-01] Fusion [6.4748e-08 9.1128e-02 2.6889e-01 1.1835e-07 2.5027e-02 3.2222e-01 1.7692e-01 1.1581e-01]
!cat simple-mvo-cvxpy.ptf
Task ''
Written by MOSEK v9.3.10
problemtype: Conic Problem
numvar: 20
numcon: 9
numcone: 1
numanz: 83
Objective ''
Maximize - @x0 - 5e-2 @x1 - @x10 - @x11
Constraints
[-7.2e-2] - @x0 + @x2 - 1.86165059661116e-2 @x12 + 9.06978859816884e-2 @x13 + 1.54910178426094e-1 @x14
+ 1.81495037818318e-1 @x15 + 2.50037506030729e-1 @x16 - 5.93612377619193e-1 @x17
+ 2.79933922890889e-1 @x18 - 6.27336232076546e-1 @x19
[-1.552e-1] - @x0 + @x3 - 1.02505079402013e-1 @x12 + 2.84957784278348e-1 @x13
- 2.35521024259424e-1 @x14 - 2.5684719619188e-1 @x15 - 2.48491298410675e-1 @x16
- 3.1598167698713e-2 @x17 + 4.28379310648829e-1 @x18 - 5.66367595023283e-1 @x19
[-1.754e-1] - @x0 + @x4 + 3.3437349955976e-1 @x12 - 2.18661325204699e-1 @x13
+ 9.66000610769484e-2 @x14 - 5.17898642170584e-2 @x15 - 3.20285188752809e-1 @x16
- 8.0950528224872e-2 @x17 + 2.78211064206838e-1 @x18 - 5.19613907547152e-1 @x19
[-8.98e-2] - @x0 + @x5 - 2.36447361970562e-1 @x12 - 3.41604736229777e-1 @x13
- 1.87237545386354e-1 @x14 + 2.09687625804829e-1 @x15 - 8.08250070409317e-4 @x16
+ 1.25424176974319e-1 @x17 + 3.22093451563683e-1 @x18 - 5.61650375333107e-1 @x19
[-4.29e-1] - @x0 + @x6 + 2.68149331615919e-2 @x12 - 2.99612315627908e-2 @x13
- 2.32667096216671e-1 @x14 - 1.45073467254737e-2 @x15 - 5.68109825658349e-2 @x16
- 1.96951758225218e-1 @x17 - 6.87239252048446e-1 @x18 - 1.07946207944086 @x19
[-3.929e-1] - @x0 + @x7 - 1.9698009544702e-1 @x12 - 6.06521118962487e-2 @x13
+ 4.1680553882257e-1 @x14 - 2.78263173173759e-1 @x15 - 5.00156989238449e-2 @x16
+ 1.26760827897988e-1 @x17 - 1.24385574096319e-1 @x18 - 7.66415414647846e-1 @x19
[-3.217e-1] - @x0 + @x8 + 6.41283207339116e-2 @x12 + 2.35607595927053e-1 @x13
+ 1.13336205115967e-1 @x14 + 4.0631576178505e-1 @x15 - 7.59083691968833e-2 @x16
+ 4.02771893115513e-1 @x17 - 7.70958353367759e-3 @x18 - 7.52221818571036e-1 @x19
[-1.838e-1] - @x0 + @x9 + 1.66013536893901e-1 @x12 - 1.81422434329357e-2 @x13
- 1.00939071307121e-1 @x14 - 2.20052748341652e-1 @x15 + 4.97707226872002e-1 @x16
+ 2.82238967205627e-1 @x17 + 1.60681966480785e-1 @x18 - 6.17866174109235e-1 @x19
[0] - 3.97346076206104e-1 @x1 + @x10 - @x11
Variables
@x0
@x1 [0;+inf]
@x2 [0;+inf]
@x3 [0;+inf]
@x4 [0;+inf]
@x5 [0;+inf]
@x6 [0;+inf]
@x7 [0;+inf]
@x8 [0;+inf]
@x9 [0;+inf]
@K0 [SOC(10)]
@x10
@x11
@x12
@x13
@x14
@x15
@x16
@x17
@x18
@x19
with a risk bound
where
with open('factor/factor.npy', 'rb') as f:
alpha = np.load(f) # Objective
min_trade = np.load(f) # Bounds for trades
max_trade = np.load(f)
A = np.load(f) # Some linear constraints (see later)
bl = np.load(f)
bu = np.load(f)
beta = np.load(f) # Factor exposures
fac_cov = np.load(f) # Factor covariance
spec_risk = np.load(f) # Specific risk
holding = np.load(f) # Initial holdings
gamma = np.load(f)[0] # Risk bound parameter
n_assets = len(alpha) # Number of assets (excl. cash)
n_factors = fac_cov.shape[0] # Number of factors
print(f"Number of assets {n_assets}")
print(f"Number of factors {n_factors}")
print("Dim of beta {}".format(beta.shape))
print("Dim of fac_cov {}".format(fac_cov.shape))
Number of assets 3969 Number of factors 43 Dim of beta (3969, 43) Dim of fac_cov (43, 43)
The linear part
has a direct model:
M = Model()
x = M.variable("x", n_assets + 1, Domain.inRange(min_trade, max_trade))
trades, cash = x.slice(0, n_assets), x.index(n_assets)
M.constraint("balance", Expr.sum(x), Domain.equalsTo(0))
M.constraint("bounds", Expr.mul(A, trades), Domain.inRange(bl, bu))
M.objective("return", ObjectiveSense.Maximize, Expr.dot(alpha, trades))
# Remember this model for later
basicModel = M.clone()
We proceed with the risk bound constraint
Sigma = beta @ fac_cov @ beta.T + np.diag(spec_risk)
print(Sigma.shape)
(3969, 3969)
G = np.linalg.cholesky(Sigma)
--------------------------------------------------------------------------- LinAlgError Traceback (most recent call last) /tmp/ipykernel_634016/2992681275.py in <module> ----> 1 G = np.linalg.cholesky(Sigma) <__array_function__ internals> in cholesky(*args, **kwargs) ~/anaconda3/envs/pres/lib/python3.9/site-packages/numpy/linalg/linalg.py in cholesky(a) 761 t, result_t = _commonType(a) 762 signature = 'D->D' if isComplexType(t) else 'd->d' --> 763 r = gufunc(a, signature=signature, extobj=extobj) 764 return wrap(r.astype(result_t, copy=False)) 765 ~/anaconda3/envs/pres/lib/python3.9/site-packages/numpy/linalg/linalg.py in _raise_linalgerror_nonposdef(err, flag) 89 90 def _raise_linalgerror_nonposdef(err, flag): ---> 91 raise LinAlgError("Matrix is not positive definite") 92 93 def _raise_linalgerror_eigenvalues_nonconvergence(err, flag): LinAlgError: Matrix is not positive definite
The reason is:
np.linalg.eigvalsh(Sigma)[:10]
array([-2.2824e-16, -2.1635e-16, 1.2673e-04, 2.3408e-04, 2.8771e-04, 3.0525e-04, 3.5774e-04,
3.9572e-04, 3.9724e-04, 4.1947e-04])
a small negative eigenvalue caused by precision issues. Typically one tries to correct for it by boosting the diagonal a bit:
G = np.linalg.cholesky(Sigma + 1e-12*np.eye(n_assets))
G.shape
(3969, 3969)
Going back to the original optimization problem, we can now write the risk bound
M = basicModel.clone()
M.constraint("risk", Expr.vstack(gamma,
Expr.mul(G.T, Expr.add(trades, holding))),
Domain.inQCone())
fullModel = M.clone()
M.setLogHandler(sys.stdout)
M.solve()
Problem Name : Objective sense : max Type : CONIC (conic optimization problem) Constraints : 4033 Cones : 1 Scalar variables : 7941 Matrix variables : 0 Integer variables : 0 Optimizer started. Presolve started. Linear dependency checker started. Linear dependency checker terminated. Eliminator started. Freed constraints in eliminator : 0 Eliminator terminated. Eliminator - tries : 1 time : 0.00 Lin. dep. - tries : 1 time : 0.03 Lin. dep. - number : 0 Presolve terminated. Time: 0.97 Problem Name : Objective sense : max Type : CONIC (conic optimization problem) Constraints : 4033 Cones : 1 Scalar variables : 7941 Matrix variables : 0 Integer variables : 0 Optimizer - threads : 64 Optimizer - solved problem : the primal Optimizer - Constraints : 4031 Optimizer - Cones : 1 Optimizer - Scalar variables : 8001 conic : 3970 Optimizer - Semi-definite variables: 0 scalarized : 0 Factor - setup time : 1.22 dense det. time : 0.25 Factor - ML order time : 0.22 GP order time : 0.00 Factor - nonzeros before factor : 8.12e+06 after factor : 8.13e+06 Factor - dense dim. : 0 flops : 4.28e+10 ITE PFEAS DFEAS GFEAS PRSTATUS POBJ DOBJ MU TIME 0 3.5e+05 4.7e+00 2.5e+05 0.00e+00 0.000000000e+00 2.490130070e+05 1.0e+00 2.61 1 1.4e+05 1.8e+00 1.6e+05 -1.00e+00 -1.451840212e+01 2.489961721e+05 3.9e-01 3.09 2 2.5e+04 3.4e-01 6.7e+04 -1.00e+00 -1.413064381e+02 2.488453306e+05 7.2e-02 3.54 3 3.4e+03 4.6e-02 2.5e+04 -1.00e+00 -1.009686394e+03 2.477832517e+05 9.7e-03 4.26 4 1.9e+03 2.6e-02 1.8e+04 -9.99e-01 -2.141509197e+03 2.464474242e+05 5.5e-03 4.77 5 5.4e+02 7.3e-03 9.7e+03 -9.98e-01 -1.000183115e+04 2.372767435e+05 1.5e-03 5.43 6 4.6e+01 6.3e-04 2.8e+03 -9.90e-01 -1.370666468e+05 9.010151055e+04 1.3e-04 5.93 7 1.1e+00 1.4e-05 1.5e+02 -8.79e-01 -1.645039837e+06 -1.621500814e+06 3.0e-06 6.70 8 2.3e-01 3.2e-06 1.8e+01 4.75e-01 -1.100346180e+06 -1.094190930e+06 6.6e-07 7.36 9 1.0e-01 1.4e-06 5.2e+00 8.56e-01 2.943600402e+05 2.978146518e+05 3.0e-07 7.91 10 6.2e-02 8.5e-07 2.4e+00 9.33e-01 1.548355696e+06 1.550573920e+06 1.8e-07 8.38 11 3.9e-02 5.3e-07 1.2e+00 9.59e-01 2.216787023e+06 2.218248870e+06 1.1e-07 8.86 12 2.2e-02 3.0e-07 4.9e-01 9.74e-01 3.291832558e+06 3.292691823e+06 6.3e-08 9.37 13 1.9e-02 2.6e-07 3.9e-01 9.85e-01 3.679422385e+06 3.680173306e+06 5.5e-08 9.86 14 1.1e-02 1.5e-07 1.6e-01 9.87e-01 4.454953846e+06 4.455387087e+06 3.1e-08 10.40 15 6.1e-03 8.4e-08 7.1e-02 9.93e-01 4.970792256e+06 4.971046756e+06 1.8e-08 10.85 16 4.0e-03 5.5e-08 3.7e-02 9.96e-01 5.236993681e+06 5.237162289e+06 1.2e-08 11.30 17 8.3e-04 1.1e-08 3.5e-03 9.97e-01 5.592734468e+06 5.592769860e+06 2.4e-09 11.86 18 1.0e-04 1.4e-09 1.4e-04 9.99e-01 5.681420765e+06 5.681425282e+06 2.9e-10 12.35 19 3.7e-05 5.1e-10 3.2e-05 1.00e+00 5.691120633e+06 5.691122294e+06 1.1e-10 12.85 20 4.8e-06 6.5e-11 1.5e-06 1.00e+00 5.696121613e+06 5.696121826e+06 1.4e-11 13.54 21 2.2e-06 3.0e-11 4.6e-07 1.00e+00 5.696525300e+06 5.696525397e+06 6.3e-12 14.34 22 4.1e-07 5.6e-12 3.8e-08 1.00e+00 5.696799559e+06 5.696799578e+06 1.2e-12 14.75 23 1.1e-08 1.5e-13 1.6e-10 1.00e+00 5.696862050e+06 5.696862051e+06 3.1e-14 15.20 Optimizer terminated. Time: 15.37 Interior-point solution summary Problem status : PRIMAL_AND_DUAL_FEASIBLE Solution status : OPTIMAL Primal. obj: 5.6968620502e+06 nrm: 7e+05 Viol. con: 9e-04 var: 0e+00 cones: 0e+00 Dual. obj: 5.6968620488e+06 nrm: 3e+06 Viol. con: 1e-09 var: 2e-08 cones: 0e+00
Let us instead exploit the fact that we already (almost) have a factorization of on our hands.
If $ \Sigma_F = P\cdot P^T $ then $\beta\Sigma_F\beta^T = \beta P P^T \beta^T = (\beta P)(\beta P)^T$
and finally
M = basicModel.clone()
P = np.linalg.cholesky(fac_cov)
H1, H2 = P.T @ beta.T, np.diag(np.sqrt(spec_risk))
H = np.vstack(( H1, H2 ))
print(H1.shape)
M.constraint("risk", Expr.vstack(gamma,
Expr.mul(H, Expr.add(trades, holding))),
Domain.inQCone())
factorModel = M.clone()
(43, 3969)
M.setLogHandler(sys.stdout)
M.solve()
Problem Name : Objective sense : max Type : CONIC (conic optimization problem) Constraints : 4076 Cones : 1 Scalar variables : 7984 Matrix variables : 0 Integer variables : 0 Optimizer started. Presolve started. Linear dependency checker started. Linear dependency checker terminated. Eliminator started. Freed constraints in eliminator : 0 Eliminator terminated. Eliminator - tries : 1 time : 0.00 Lin. dep. - tries : 1 time : 0.00 Lin. dep. - number : 0 Presolve terminated. Time: 0.03 Problem Name : Objective sense : max Type : CONIC (conic optimization problem) Constraints : 4076 Cones : 1 Scalar variables : 7984 Matrix variables : 0 Integer variables : 0 Optimizer - threads : 64 Optimizer - solved problem : the primal Optimizer - Constraints : 106 Optimizer - Cones : 1 Optimizer - Scalar variables : 4076 conic : 4011 Optimizer - Semi-definite variables: 0 scalarized : 0 Factor - setup time : 0.01 dense det. time : 0.00 Factor - ML order time : 0.00 GP order time : 0.00 Factor - nonzeros before factor : 5671 after factor : 5671 Factor - dense dim. : 0 flops : 9.74e+06 ITE PFEAS DFEAS GFEAS PRSTATUS POBJ DOBJ MU TIME 0 2.5e+05 1.3e+03 2.5e+05 0.00e+00 -2.897480425e+06 -2.648467418e+06 1.0e+00 0.05 1 2.4e+05 1.3e+03 2.5e+05 -1.00e+00 -2.897476069e+06 -2.648463172e+06 9.8e-01 0.07 2 7.6e+04 4.1e+02 1.4e+05 -1.00e+00 -2.897505003e+06 -2.648501283e+06 3.1e-01 0.08 3 3.2e+04 1.7e+02 9.0e+04 -1.00e+00 -2.897595770e+06 -2.648610977e+06 1.3e-01 0.09 4 2.2e+04 1.2e+02 7.4e+04 -1.00e+00 -2.897511742e+06 -2.648542456e+06 8.9e-02 0.10 5 8.6e+03 4.6e+01 4.6e+04 -1.00e+00 -2.897313011e+06 -2.648422359e+06 3.5e-02 0.10 6 4.6e+03 2.5e+01 3.4e+04 -9.99e-01 -2.897050254e+06 -2.648278698e+06 1.8e-02 0.11 7 2.9e+03 1.5e+01 2.6e+04 -9.98e-01 -2.896512230e+06 -2.647897282e+06 1.1e-02 0.12 8 1.6e+03 8.4e+00 2.0e+04 -9.97e-01 -2.895323315e+06 -2.647068807e+06 6.4e-03 0.13 9 8.6e+02 4.6e+00 1.4e+04 -9.95e-01 -2.894896373e+06 -2.647364244e+06 3.5e-03 0.13 10 1.1e+02 5.8e-01 4.9e+03 -9.89e-01 -2.872432606e+06 -2.637178221e+06 4.4e-04 0.14 11 5.6e+00 3.0e-02 7.5e+02 -8.93e-01 -2.603970799e+06 -2.503252609e+06 2.2e-05 0.15 12 1.8e+00 9.6e-03 1.9e+02 5.40e-02 -2.157057010e+06 -2.112178716e+06 7.2e-06 0.16 13 9.1e-01 4.9e-03 7.0e+01 5.53e-01 -8.962472525e+05 -8.703314249e+05 3.7e-06 0.17 14 7.0e-01 3.7e-03 4.8e+01 7.46e-01 2.176641830e+05 2.383892733e+05 2.8e-06 0.18 15 4.2e-01 2.3e-03 2.2e+01 7.98e-01 9.436222866e+05 9.567544368e+05 1.7e-06 0.18 16 2.2e-01 1.2e-03 8.6e+00 8.73e-01 2.048131675e+06 2.055396701e+06 9.0e-07 0.19 17 1.9e-01 1.0e-03 6.6e+00 9.30e-01 2.628091716e+06 2.634250629e+06 7.6e-07 0.20 18 1.4e-01 7.3e-04 4.0e+00 9.41e-01 3.230923330e+06 3.235386735e+06 5.5e-07 0.21 19 8.8e-02 4.7e-04 2.1e+00 9.57e-01 4.027097745e+06 4.030010692e+06 3.5e-07 0.21 20 6.0e-02 3.2e-04 1.2e+00 9.72e-01 4.518136638e+06 4.520136844e+06 2.4e-07 0.22 21 4.0e-02 2.1e-04 6.5e-01 9.81e-01 4.883990200e+06 4.885333054e+06 1.6e-07 0.23 22 1.7e-02 9.3e-05 1.9e-01 9.87e-01 5.346954165e+06 5.347541499e+06 7.0e-08 0.23 23 5.3e-03 2.8e-05 3.1e-02 9.94e-01 5.584911106e+06 5.585089214e+06 2.1e-08 0.24 24 1.0e-03 5.5e-06 2.7e-03 9.98e-01 5.671131694e+06 5.671166510e+06 4.1e-09 0.25 25 2.3e-05 1.2e-07 9.0e-06 1.00e+00 5.696271335e+06 5.696272121e+06 9.4e-11 0.26 26 3.8e-07 2.0e-09 1.9e-08 1.00e+00 5.696854015e+06 5.696854028e+06 1.5e-12 0.26 27 1.9e-09 1.9e-09 3.2e-11 1.00e+00 5.696863727e+06 5.696863730e+06 1.4e-16 0.27 Optimizer terminated. Time: 0.29 Interior-point solution summary Problem status : PRIMAL_AND_DUAL_FEASIBLE Solution status : OPTIMAL Primal. obj: 5.6968637273e+06 nrm: 7e+05 Viol. con: 8e-05 var: 5e-07 cones: 0e+00 Dual. obj: 5.6968637302e+06 nrm: 3e+06 Viol. con: 1e-14 var: 2e-10 cones: 0e+00
As always we can analyze the solution and make some checks:
tradeVal = M.getVariable("x").level()[:-1]
print("risk bound = {gamma}, actual risk = {risk}".format(gamma=gamma,
risk=np.sqrt((tradeVal.T+holding.T) @ Sigma @ (tradeVal+holding))))
shorts = tradeVal[tradeVal < 0]
longs = tradeVal[tradeVal > 0]
print("Total short positions {}".format(-sum(shorts)))
print("Total long positions {}".format(sum(longs)))
totalAbs = sum(np.abs(tradeVal))
print("Total cash flow {}".format(totalAbs))
risk bound = 249013.00700000004, actual risk = 249013.0069221636 Total short positions 3552537.585477736 Total long positions 3689760.3450614978 Total cash flow 7242297.930539243
plt.hist(tradeVal, bins=30, log=True);
How many trades are actually substantial?
bigTrades = np.where(np.abs(tradeVal) > 0.001 * totalAbs)[0]
print(bigTrades)
print("count = {}".format(len(bigTrades)))
[ 73 82 94 96 99 107 123 154 192 223 261 312 352 406 449 573 576 617 632 660 669 692 782 787 801 880 1048 1059 1100 1173 1205 1207 1233 1254 1267 1288 1295 1400 1459 1523 1552 1629 1708 1723 1740 1743 1813 1853 1864 1923 1930 1963 1978 1991 1997 2036 2102 2111 2127 2133 2151 2189 2200 2222 2299 2311 2360 2417 2441 2451 2530 2552 2567 2615 2627 2628 2645 2656 2692 2695 2704 2737 2745 2793 2803 2810 2814 2817 2915 2983 2986 2997 3015 3037 3055 3064 3069 3114 3122 3166 3208 3210 3286 3318 3330 3401 3465 3477 3526 3556 3613 3637 3675 3696 3738 3740 3745 3768 3769 3849 3858 3907 3931] count = 123
A typical slice of the trades looks like:
print(tradeVal[130:160])
[ 9.0216e-08 7.5190e-08 7.5490e-08 2.4318e-06 8.2513e-08 8.2141e-08 1.9282e-07 1.0761e-07 1.5009e-07 7.3495e-08 9.3916e-08 2.1961e-07 1.2923e-07 1.5395e-07 5.3223e-08 1.1344e-05 7.7962e-08 5.1486e-08 4.8990e-08 3.2322e-07 7.5978e-08 3.8211e-07 4.0998e-06 7.9010e-08 -9.4188e+04 6.3841e-08 1.0578e-07 3.1015e-07 1.0507e-07 6.8921e-08]
Here it is the user's job to interpret and post-process the solution if necessary.
We could for example choose to
ignores = list(np.where(np.abs(tradeVal) < 0.001 * totalAbs)[0])
M.constraint( trades.pick(ignores), Domain.equalsTo(0.0) )
M.solve()
Optimizer started. Optimizer terminated. Time: 0.03 Interior-point solution summary Problem status : PRIMAL_AND_DUAL_FEASIBLE Solution status : OPTIMAL Primal. obj: 5.6830884988e+06 nrm: 7e+05 Viol. con: 1e-04 var: 0e+00 cones: 0e+00 Dual. obj: 5.6830884882e+06 nrm: 3e+06 Viol. con: 1e-10 var: 4e-08 cones: 0e+00
M.getVariable("x").level()[130:160]
array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., -94188., 0., 0., 0., 0., 0.])
fullModel
factorModel
sparsityPattern(factorModel)
sparsityPattern(fullModel)
if ... then implicationsLet us extend the previous model with a cardinality constraint for the number of traded assets:
on) if we trade $i$-th asset ($x_i\neq 0$) and $0$ (off) otherwise.Then we can express 2. with
and the number of assets for which trading is on equals $\sum_i z_i$.
M = factorModel.clone()
z = M.variable(n_assets, Domain.binary()) # The binary variable indicating traded / not traded
bigM = M.parameter() # The big-M constant in the model
k = M.parameter() # Cardinality bound
# The constraints involving z
M.constraint(Expr.sub( Expr.mul(bigM,z), trades ), Domain.greaterThan(0)) # trades <= bigM*z
M.constraint(Expr.add( Expr.mul(bigM,z), trades ), Domain.greaterThan(0)) # -trades <= bigM*z
# The actual cardinality bound sum(z) <= k
M.constraint(Expr.sub( Expr.sum(z), k ), Domain.lessThan(0));
bigM.setValue(3e+6)
k.setValue(100)
M.setSolverParam("optimizerMaxTime", 80) # seconds
M.setLogHandler(sys.stdout)
M.solve()
Problem Name : Objective sense : max Type : CONIC (conic optimization problem) Constraints : 12015 Cones : 1 Scalar variables : 11953 Matrix variables : 0 Integer variables : 3969 Optimizer started. Mixed integer optimizer started. Threads used: 64 Presolve started. Presolve terminated. Time = 0.82 Presolved problem: 7982 variables, 4155 constraints, 202526 non-zeros Presolved problem: 0 general integer, 3969 binary, 4013 continuous Clique table size: 0 BRANCHES RELAXS ACT_NDS DEPTH BEST_INT_OBJ BEST_RELAX_OBJ REL_GAP(%) TIME 0 1 1 0 NA 5.6968637101e+06 NA 1.7 0 1 1 0 3.6608495077e+06 5.6968637101e+06 55.62 20.8 0 1 1 0 5.6180147901e+06 5.6968637101e+06 1.40 40.4 Cut generation started. 0 1 1 0 5.6180147901e+06 5.6968637083e+06 1.40 41.6 Cut generation terminated. Time = 0.17 10 14 11 5 5.6180147901e+06 5.6968615936e+06 1.40 45.4 25 29 26 7 5.6180147901e+06 5.6968615936e+06 1.40 46.6 44 48 39 8 5.6180147901e+06 5.6968615936e+06 1.40 47.6 59 63 50 9 5.6180147901e+06 5.6968615936e+06 1.40 48.0 74 78 61 10 5.6180147901e+06 5.6968615936e+06 1.40 48.5 95 99 76 11 5.6180147901e+06 5.6968615936e+06 1.40 49.0 120 124 97 12 5.6180147901e+06 5.6968615936e+06 1.40 49.6 151 155 104 13 5.6180147901e+06 5.6968615936e+06 1.40 50.2 191 195 114 14 5.6180147901e+06 5.6968615936e+06 1.40 51.7 231 235 154 15 5.6180147901e+06 5.6968615936e+06 1.40 52.5 322 326 213 17 5.6180147901e+06 5.6968615936e+06 1.40 54.1 512 516 343 20 5.6180147901e+06 5.6968615936e+06 1.40 57.1 832 836 625 25 5.6180147901e+06 5.6968615936e+06 1.40 65.8 1408 1412 1201 34 5.6180147901e+06 5.6968615936e+06 1.40 75.2 Objective of best integer solution : 5.618014790067e+06 Best objective bound : 5.696861593585e+06 Construct solution objective : Not employed User objective cut value : Not employed Number of cuts generated : 0 Number of branches : 1408 Number of relaxations solved : 1412 Number of interior point iterations: 47112 Number of simplex iterations : 0 Time spend presolving the root : 0.82 Time spend optimizing the root : 1.21 Mixed integer optimizer terminated. Time: 80.00 Optimizer terminated. Time: 80.33 Integer solution solution summary Problem status : PRIMAL_FEASIBLE Solution status : PRIMAL_FEASIBLE Primal. obj: 5.6180147901e+06 nrm: 3e+06 Viol. con: 2e+00 var: 4e-12 cones: 0e+00 itg: 0e+00
Let us inspect the solution.
M.acceptedSolutionStatus(AccSolutionStatus.Feasible)
zVal = z.level()
tradedIndxs = np.where(np.abs(zVal) == 1.0)[0]
print(tradedIndxs)
print("number of activated trades in z = {}".format(len(tradedIndxs)))
[ 73 96 107 123 154 223 261 312 342 352 449 573 576 617 632 660 669 692 782 787 880 888 1048 1059 1100 1173 1205 1207 1267 1295 1400 1459 1523 1552 1629 1708 1723 1740 1743 1813 1864 1923 1930 1963 1978 1991 1997 2102 2111 2127 2133 2151 2222 2360 2417 2441 2451 2530 2552 2567 2615 2628 2645 2656 2692 2695 2704 2737 2745 2793 2803 2810 2814 2817 2983 2986 2997 3037 3064 3069 3114 3166 3210 3286 3330 3465 3477 3526 3556 3637 3675 3696 3738 3740 3745 3768 3769 3858 3907 3931] number of activated trades in z = 100
X = M.getVariable("x").level()
bigTrades = np.where(np.abs(X) != 0.0)[0] # "Activated" positions as measured by x_i != 0
print(bigTrades)
print("number of nonzero trades = {}".format(len(bigTrades)))
[ 4 73 96 107 123 126 154 179 223 261 312 342 352 357 383 449 452 573 576 617 632 660 669 692 696 767 782 787 880 888 977 983 1048 1059 1100 1134 1173 1205 1207 1267 1295 1400 1459 1523 1552 1569 1600 1629 1693 1708 1723 1740 1743 1813 1864 1868 1923 1927 1930 1963 1978 1991 1997 2102 2111 2127 2133 2151 2222 2360 2382 2402 2413 2417 2441 2446 2451 2521 2530 2552 2567 2615 2628 2645 2656 2692 2695 2704 2737 2745 2793 2803 2810 2814 2817 2983 2986 2997 2999 3037 3064 3069 3114 3166 3210 3285 3286 3330 3465 3474 3477 3492 3498 3505 3522 3526 3556 3637 3675 3696 3738 3740 3745 3768 3769 3858 3872 3907 3931 3969] number of nonzero trades = 130
X[4]
-4.547473508864641e-13
What to consider with the mixed-integer optimizer:
M.setSolverParam("mioTolRelGap", 0.005) # 0.5 %
k.setValue(150)
M.solve()
Problem Name : Objective sense : max Type : CONIC (conic optimization problem) Constraints : 12015 Cones : 1 Scalar variables : 11953 Matrix variables : 0 Integer variables : 3969 Optimizer started. Mixed integer optimizer started. Threads used: 64 Presolve started. Presolve terminated. Time = 0.82 Presolved problem: 7982 variables, 4155 constraints, 202526 non-zeros Presolved problem: 0 general integer, 3969 binary, 4013 continuous Clique table size: 0 BRANCHES RELAXS ACT_NDS DEPTH BEST_INT_OBJ BEST_RELAX_OBJ REL_GAP(%) TIME 0 1 1 0 NA 5.6968636869e+06 NA 1.6 0 1 1 0 5.6181637134e+06 5.6968636869e+06 1.40 1.7 0 1 1 0 5.6968637315e+06 5.6968637315e+06 0.00e+00 40.5 An optimal solution satisfying the relative gap tolerance of 5.00e-01(%) has been located. The relative gap is 0.00e+00(%). An optimal solution satisfying the absolute gap tolerance of 0.00e+00 has been located. The absolute gap is 0.00e+00. Objective of best integer solution : 5.696863731453e+06 Best objective bound : 5.696863731453e+06 Construct solution objective : 5.618163713377e+06 User objective cut value : Not employed Number of cuts generated : 0 Number of branches : 0 Number of relaxations solved : 1 Number of interior point iterations: 28 Number of simplex iterations : 0 Time spend presolving the root : 0.82 Time spend optimizing the root : 0.76 Mixed integer optimizer terminated. Time: 40.48 Optimizer terminated. Time: 40.50 Integer solution solution summary Problem status : PRIMAL_FEASIBLE Solution status : INTEGER_OPTIMAL Primal. obj: 5.6968637315e+06 nrm: 3e+06 Viol. con: 3e-05 var: 2e-04 cones: 1e-05 itg: 0e+00