AMERICAN LANGUAGE HUB_Level2_Student'sBook_Answerkey.pdf
Seminar on Optimization Techniques: LP, QP, SOCP, SDP, LMI, BMI
1. Seminar:
LP, QP, SOCP, SDP, LMI, BMI and other UFOs
(not-Dr.-yet) Mikhail V. Konnik
School of Electrical Engineering and Computer Science
The University of Newcastle
Australia
December 13, 2013
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
1 Austra
2. Disclaimer
the Author is not an expert (yet) in LMI/BMI/SDP (but he is doing his
best to become one), and therefore:
the whole presentation is just an overview of LMI/BMI from the point of
view of the Author's (severely incomplete) understanding of optimisation;
the Author is not ready to solve complicated BMI/UFO problems that
the people in the audience surely have :-)
the problem of solving even LMI (let alone BMI) is much more complicated
than many people in the audience think;
software packages for LMI/BMI are not as mature as for e.g. QP;
the Author is known for his anity to reinvent the wheel to better
understand how stu works.
The wheel that is being inventing by the Author is still in the blueprint
stage, is not round, and cannot be used for anything remotely serious.
the Author is not liable for any loss, damage, illness, injury, headache,
crash, collapse of the Universe or anything else caused by this presentation.
proceed at your own risk!
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
2 Austra
3. Introduction to Linear Matrix Inequalities
Part I ::
A short introduction to Linear Matrix Inequalities
(LMI) and Semidenite Programming (SDP)
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
3 Austra
4. Introduction to Linear Matrix Inequalities
LMI and Control
Linear Matrix Inequalities and Control
The story of LMI begins in about 1890, when Lyapunov published his seminal
work, where he showed that the dierential equation
d
dx
x (t )
= Ax (t )
is stable (i.e., all trajectories converge to zero) if and only if there exists a
positive-denite matrix P such that[1]
A
The requirement P
T P + PA
0
0 is what we now call a Lyapunov inequality on P .
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
4 Austra
5. Introduction to Linear Matrix Inequalities
LMI and Control
Linear Matrix Inequalities and Control
The Lyapunov inequality A
T P + PA
0 is not the only example of how one
can convert control problems into LMI: Riccati inequality can be converted
into an LMI, too:
A
T P + PA + PBR −1 B T P + Q
0,
is equivalent to the following LMI (via Schur complement):
−AT P − PA − Q
B
The requirement P
TP
PB
R
0
(1)
0 is what we now call a Lyapunov inequality on P .
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
5 Austra
6. Introduction to Linear Matrix Inequalities
What is an LMI?
What is an LMI?
A (strict) linear matrix inequality (LMI) is a convex
constraint in the form:
N
F (x )
= F0 + x1 F1 + · · · + xN FN = F0 +
i =1
xi Fi
0,
(2)
where:
x
= [x1 , . . . , xN ]
is a vector of
unknown scalars (the decision
or optimisation variables) x1 , x2 and so on;
F0 , F1 , F2 , . . . , FN are n
The sign F
×n
symmetric matrices, they are given.
0 is a generalised inequality meaning F is a negative denite
matrix (i.e., the largest eigenvalue of F (x ) is negative), F
denite, and F
0 is positive
0 is positive semi-denite.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
6 Austra
7. Introduction to Linear Matrix Inequalities
What is an LMI?
What is an LMI? - A set of CONVEX constraints!
Note that the LMI:
F (x )
= F0 + x1 F1 + . . .
N
+xN FN = F0 +
i =1
0,
xi Fi
is just a convex constraint
on x , and therefore:
Its solution set, called
the
feasible set,
convex subset of
is a
RN
Finding a solution x of
LMI, if any, is a
convex
Figure 1: Supporting hyperplanes for LMI.
problem.
Convexity has an important consequence: even though LMI has no analytical
solution in general, it can be solved numerically.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
7 Austra
8. Introduction to Linear Matrix Inequalities
Solving Linear Matrix Inequalities in a brute-force way
Solving LMI in a brute-force way
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
8 Austra
9. Introduction to Linear Matrix Inequalities
Solving Linear Matrix Inequalities in a brute-force way
Solving LMI in a brute-force way
The LMI species a convex
constraint
on
x,
and
to
solve an LMI means that
we can nd x that satises
the LMI.
Brute-force: we can try the
variables in
x1 x2
X =
x2 x3
compute
Z
= AT X + XA
Figure 2:
The corresponding set from the LMI
F (x ) := x1 F1 + x2 F2
I.
check eigenvalues
E
= eig (Z )
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
9 Austra
10. Semidenite Programming
We can do better: solve LMI via SDP
A problem which subsumes linear, quadratic,
geometric and second-order cone programming
is called a
semidenite program (SDP):
TX
minimize
C
subject to
F0
X
- - this is optimality metric!
N
Ax
+
i =1
xi Fi
0,
- - this is LMI!
=b
where the matrices F0 , F1 , . . . Fn
∈ Sk ,
and A
∈ Rp×n .
That is: LMI cuts a set of solutions via convex set of constraints, and SDP
allows to pick up an optimal solution using some optimality metric.
To solve LMI ⇔ solve a Semidenite Programming (SDP) problem.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
10 Austra
11. Semidenite Programming
What is Semidenite?
The set of real symmetric n × n matrices
is denoted
Sn .
A matrix Z
semidenite
Rn .
∈ Sn
if
is called
T
x Zx ≥
positive
0 for all x
∈
On the other hand, a matrix Z is
called
positive denite
for all nonzero x
∈ Rn .
if x
T Zx
0
The set of positive semidenite matrices
is denoted
Sn .
+
In other words, the eigenvalues
λ1 . . . λn
of a positive
semidenite matrix Z+ are nonnegative (i.e., can be zeros or less than
machine epsilon
The set
Sn
+
).
is a convex cone.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
11 Austra
12. Semidenite Programming
What is Semidenite? (continued)
A semidenite program (SDP) is a generalization of a linear program (LP),
where the inequality constraints are replaced by matrix inequalities corresponding
to the cone of positive semidenite matrices.
An SDP in the pure primal form is dened as:
minimise
trace(CX )
subject to
trace(Ai X ) = bi ∀i = 1, . . . , m
X
X
(3)
0.
∈ Sn is the decision variable (can be a vector or a matrix), b ∈ Rm
+
n
C , A1 . . . Am ∈ S+ are given symmetric matrices. There are m ane
where X
and
constraints among the entries of the
positive semidenite matrix X .
feasible sets of SDPs
can have curved faces, together with sharp corners where faces intersect.
Unlike LP, where the feasible sets are polyhedra, the
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
12 Austra
13. Semidenite Programming
Scope of SDP
TX
minimise
C
subject to
F0
X
SDP:
N
minimise
SOCP:
c
x
+
QP:
x
subject to
LP:
2
x
Ax
0
- - - LMI
+ bi
2
ciT x + di
=g
T Hx + c T x + d
b
Tx + d
minimise
c
subject to
Ax
x
xi Fi
Tx
Fx
1
i =1
Ai x
subject to
minimise
Scope of Semidenite Programming
b
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
13 Austra
14. Semidenite Programming
SDP: One Ring to Rule them All
Semidenite Programming SDP: One Ring to Rule them All
Using the power of the Sauron's One Ring SDP, re-cast a QP problem into
an SDP problem. Use Cholesky factor
x
subject to
Ax
minimise
X
subject to
C
TX
F (x )
0
:= blockdiag (FQ (x ), Ai x − bi ),
Ai is the ith row of A, and we use Schur complement r
FQ (x )
=
H.
≥b
Equivalent SDP:
where F (x )
of the Hessian matrix
T Q T Qx + c T x + r
minimise
x
H = QT Q
r
− cT x
Qx
x
T QT
I
≥ x T Q T Qx + c T x ,
0.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
14 Austra
15. Summary for SDP and LMI
Summary for SDP and LMI
1
A (strict) linear matrix inequality (LMI) is a convex
form:
N
F0
+
i =1
Finding a solution x of LMI, if any, is a
3
To solve LMI
⇔
convex problem.
solve a Semidenite Programming (SDP):
TX
minimize
C
subject to
x1 F1
x
0.
xi Fi
2
Ax
- - this is an optimality metric!
+ · · · + xn Fn + F0
0
- - this is LMI!
=b
where the matrices F0 , F1 , . . . Fn
4
constraint in the
∈ Sk ,
and A
∈ Rp×n .
LMI cuts a set of solutions via convex set of constraints, and SDP allows
to pick up an optimal solution using some optimality metric.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
15 Austra
16. Solving convex SDPs and LMIs
Part II ::
Solving Semidenite Problems (SDP)
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
16 Austra
17. Solving convex SDPs and LMIs
Analytical techniques and numerical algorithms
Solving convex SDPs and LMIs : analytic techniques and
numerical algorithms
Even small Semidenite problems (and LMIs) are too dicult to solve
analytically - use numerical methods!
1
Primal-dual Interior point methods
2
Logarithmic barrier methods
3
Augmented Lagrangian
There are some analytical methods that can help/simplify/relax the problem:
1
Schur complement - convert an LMI into a matrix
2
S-Procedure - a Lagrange relaxation technique for problems with quadratic
constraints (solves a system of quadratic inequalities via LMI).
3
The Elimination Lemma - eliminates some variables in LMI.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
17 Austra
18. Solving convex SDPs and LMIs
Schur complement
Analytical techniques: Schur complement
Schur complement is an indispensable tool for transforming non-linear
constraints into convex LMI. A symmetric matrix X can be decomposed:
X
The matrix S
C
− B T A− 1 B
In other words, for all X
∈
=
A
B
T
B
is called the
n
S , Y
(4)
C
Schur complement of A in X .
∈ Rm×n
, Z
∈
S
m
, the following
statements are equivalent:
0; X
a) Z
b)
− Y T Z −1 Y
X
T
Y
Y
Z
0.
0.
a) Z
b) Z
0; X
0 ;
− Y T Z −1 Y
X
Y
Y
T
Z
0.
0.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
18 Austra
19. Solving convex SDPs and LMIs
S-procedure
Analytical techniques: S-procedure for Quadratic Forms and
Strict Inequalities
In some problems, we nd that some quadratic function must be negative
whenever some other quadratic functions are all negative.
With the S-procedure, we can replace this problem by one inequality
to be satised by introducing some positive scalars to be determined.
Motivation:
The fundamental question of the theory of the S-lemma is the
following: ([2], page: 371).
When is a quadratic inequality a consequence of other quadratic
inequalities?
In short terms, the
S-procedure is a Lagrange relaxation method; it tries
to solve a system of quadratic inequalities via LMI.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
19 Austra
20. Solving convex SDPs and LMIs
S-procedure
Analytical techniques: S-procedure for Quadratic Forms and
Strict Inequalities
. . . Tp ∈ Rn×n be
condition on T0 . . . Tp :
Let T0
ξ T T0 ξ 0
symmetric matrices.
for all
It is obvious that if there exists
ξ=0
We consider the following
such that
τ1 ≥ 0, . . . τp ≥ 0
ξ T Ti ξ ≥ 0
(5)
such that:
p
T0
−
i =1
τi Ti
0,
then (5) holds.
For further details and bibliography comments, the reader is referred to [3].
*Please do NOT embarrass the Author with questions :-)
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
20 Austra
21. Solving convex SDPs and LMIs
S-procedure
Analytical techniques: S-procedure, continued - an
illustrative example
The following constraint on the variable P : ([3], page: 36)
∀ξ = 0
ξ
π
and
T
π
satisfying
T P + PA
T
B P
A
π T π ≤ ξ T C T C ξ,
ξ
π
PB
0
(6)
0
Applying the S-procedure, (6) is equivalent to:
A
T P + PA + τ C T C
T
B P
Thus the problem of nding P
0
PB
−τ I
0
such that (6) holds can be expressed as
an LMI in P and the scalar variable
τ.
([3], page: 36)
*Please do NOT embarrass the Author with questions here ;-)
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
21 Austra
22. Solving convex SDPs and LMIs
The Elimination Lemma
Analytical techniques: S-procedure, continued - an
illustrative example
The Elimination Lemma [3] allows to eliminate some variables appearing in
LMI leading to inequalities without the eliminated variable. Consider:
G (z )
+ U (z )XV (z )T + V (z )X T U (z )T 0,
(7)
ˆ
ˆ
Suppose that for every z , U (z ) and V (z ) are orthogonal complements of
U (z ) and V (z ) respectively. Then (7) holds for some X and z
=
z0 if and
only if the inequalities
T G (z )U (z ) 0,
ˆ
ˆ
U (z )
hold with z
T G (z )V (z ) 0,
ˆ
ˆ
V (z )
(8)
= z0 .
In other words, feasibility of the matrix inequality (7) with variables X and
z is equivalent to the feasibility of (8) with variable z .
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
22 Austra
23. Solving convex SDPs and LMIs
Numerical algorithms for convex SDP and LMI
Numerical algorithms for convex SDP and LMI
Schur complement, S-procedure, Elimination Lemma and other analytical
tricks will
NOT solve LMIs - use numerical methods!
1
Primal-dual Interior point methods
2
Logarithmic barrier methods
3
Augmented Lagrangian
LMIs are usually solved as SDP problems; therefore it is benecial to study
how the SDP solvers work.
*The Author promises NOT to embarrass the Audience with details here ;-)
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
23 Austra
24. Solving convex SDPs and LMIs
Canonical and matrix forms of LMIs
Numerical algorithms for convex SDP and LMI: Canonical
and matrix forms of LMIs
Generally, we do not encounter the LMI in the canonical form in control
theory but rather in the form of matrix variables. Thus, before the solution
of LMI commences via SDP, the LMI must be pre-processed, or
parsed.
For example, the Lyapunov's inequality:
A
T P + PA
0 P
= PT
0
(9)
can be written in the canonical form:
m
F (x )
where F0
=0
and Fi
= F0 +
i =1
= −AT Bi − Bi A,
xi Fi
0,
and where Bi , i
(10)
= 1, . . . , n(n + 1)/2
are matrix bases for symmetric matrices of size n.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
24 Austra
25. Solving convex SDPs and LMIs
Canonical and matrix forms of LMIs
Canonical form of LMI
Consider the Lyapunov inequality in
where A
−1
0
=
2
−2
and X
matrix form:
=
x1
x2
x2
x3
A
T X + XA
0
. The decision variables are
scalars x1 , x2 , and x3 of the matrix X .
Convert to the
A
canonical form that is F (x ) = F0 + x1 F1 + · · · + xN FN
T X + XA → · · · →
−2x1
2x1 − 3x2
− 3x2
4x2 − 4x3
2x1
:
0
Now we extract the coecients for each variable x1 , x2 and x3 :
−2x1
2x1 − 3x2
− 3x2
4x2 − 4x3
2x1
→ x1 ·
−2
2
2
0
matrix
F1
+x2 ·
0
−3
−3
4
matrix
Therefore, we converted the problem into LMI x1 F1
F2
0
0
0
+x3 ·
−4
matrix
+ x2 F2 + x3 F3
F3
0.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
25 Austra
26. Solving convex SDPs and LMIs
Software toolboxes for numerical solution of SDP and LMI
Software toolboxes for numerical solution of SDP and LMI
Solvers:
Parsers:
1
2
YALMIP - yet another
LMI parser
CVX - supports two
solvers:
SeDuMi
1
2
CVX
are
3
parsers,
SDP3 and PENSDP.
infeasible
-
infeasible
path-
SeDuMi
-
Self-Dualself-dual
embedding technique, Interior
is, they don't solve the actual
other solvers such as SeDuMi,
SDPT3
Minimization,
that
problem; instead, they rely on
Mehrotra-type
following algorithm
Please note that both YALMIP
and
-
primal-dual interior-point
and
SDPT3
SDPA
predictor-corrector
Point
4
LMI Control Toolbox
Lab,
Nemirovskii's
- LMI
projective
algorithm
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
26 Austra
27. Solving convex SDPs and LMIs
Software toolboxes for numerical solution of SDP and LMI
Author's success story: Kalman gains via LMI
˜
Denote the state estimation error as x
e
ˆ
= z − z.
=
x
ˆ
−x
and the estimation error
The overall system's dynamics:
˜
x (k
+ 1) = (A − KCy )˜(k ) +
x
e (k )
Bw
− KDyw
w (k )
(11)
˜
= Cz x (k ) + Dzw w (k )
Write the SDP for the state estimator synthesis:
min
Y ,X ,P
trace (X )
subject to
P
(PA − YCy )T
(PB − YDyw )T
X
T
Cz
P
0
Cz
P
and Y
PA
− YCy
PB
− YDyw
P
0
0
Im
0
0
= PK ,
X
= P −1
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
27 Austra
28. Solving convex SDPs and LMIs
Software toolboxes for numerical solution of SDP and LMI
Author's success story: Kalman gains via LMI
An LMI for Kalman gains for a small 2
×2
system was solved using the
SeDuMi solver and the YALMIP parser interface:
X = sdpvar(2,2); Dyw = [0 0 1 0; 0 0 0 1];
Y = sdpvar(2,2); Cz = [1 0; 0 1];
P = sdpvar(2,2); Dzw = 0; Cy = C_k;
Bw = [0 0
0 0; ...
0 0.01 0 0];
Fset=set(P0) + set( [P, P*A_k-Y*Cy, P*Bw - Y*Dyw;...
(P*A_k-Y*Cy)', P, zeros(2,4);...
(P*Bw - Y*Dyw)', zeros(4,2), eye(4)]0 ) +...
set([X,Cz; Cz', P]0);
sol = solvesdp(Fset, trace(X));
The matrix of gains found via LMI is, of course, the same (up to numerical
errors) as the one found via Riccati equations.
K = new_X*new_Y
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
28 Austra
29. Summary for methods of solving SDPs
Summary for methods of solving SDPs
1
There are analytical tools that help to simplify an SDP/LMI problem:
Schur complement, S-Procedure, Elimination Lemma.
2
Even small Semidenite problems (and LMIs) are too dicult to solve
analytically - use numerical methods!
3
Good news:
convex SDPs (LMIs) are solvable within reasonable time
using numerical methods (Interior point, Logarithmic barrier, Augmented
Lagrangian).
4
convert an LMI from the matrix form into semidenite (canonical)
parsers (YALMIP, CVX).
To solve an LMI/SDP, we need solvers (SDPA, SDPT3, SeDuMi, LMI
To
form, we need
5
Control Toolbox).
6
Author's success story: using YALMIP and SeDuMi, he solved an LMI
for Kalman gains for a small 2
×2
system.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
29 Austra
30. Solving non-convex SDPs and BMIs
Part III ::
Solving non-convex Semidenite Problems and
Bilinear Matrix Inequalities (BMIs)
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
30 Austra
31. Introduction to Bilinear Matrix Inequalities
Bilinear Matrix Inequalities
While a Linear matrix inequality (LMI) denotes a constraint of the form:
n
F (x )
= F0 +
i =1
where Fi are xed symmetric matrices and x
a
0,
xi Fi
∈
(12)
n
is the decision variable,
R
Bilinear Matrix Inequality constraints are in the form:
n
F (x )
= F0 +
i =1
n
xi Fi
n
+
j =1 j =1
Fi ,j xi xj
0,
(13)
are denoted BMIs (bilinear matrix inequalities).
Bilinear Matrix Inequalities are in the realm of
non-convex
Semidenite
Programming.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
31 Austra
32. Introduction to Bilinear Matrix Inequalities
Bilinear Matrix Inequalities: a Challenge
non-convex and
NP-hard in general[4], hence intractable in theory.
BMIs correspond to non-convex and (possibly) non-smooth Semidenite
Optimisation problems with BMIs are known to be
Programming - this is global optimisation (i.e., exponential time, local
solutions...);
no reliable solver exists for non-convex SDP problems:
branch-and-
bound, genetic algorithms, spectral bundle method, barrier methods no guarantee to nd a (global) solution within reasonable time.
numerical algorithms are inhumanly dicult to devise: only few exist.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
32 Austra
33. Introduction to Bilinear Matrix Inequalities
Bilinear Matrix Inequalities: an Illustration
Below is an illustration of the
non-convex
objective
is
function,
max
3
an
example
of
a
BMI
problem.
x2
subject to
which
quadratic problem with linear
x
2
2
− 2x2 − x1 − x2 ≥ 0
− x1 − x2 − x1 x2 ≥ 0
1
+ x2 x1 ≥ 0
Figure 3:
Illustration of a BMI: a non-
convex feasible set delimited by circular
and hyperbolic arcs (adapted from[5]).
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
33 Austra
34. Introduction to Bilinear Matrix Inequalities
Numerical algorithms for non-convex SDPs and BMIs
Numerical algorithms for non-convex SDPs and BMIs
Don't even try to solve BMIs analytically - it is dicult even for numerical
methods of global optimisation (non-convex SDPs):
1
Genetic algorithms:
Read-coded Genetic Algorithms, Extrapolation-
directed Crossover (EDX), Minimal Generation Gap (MGG);
2
Branch-and-Bound, Branch-and-Cut methods (have troubles with solving
medium/large problems due to loose lower-bound approximations);
3
Coordinate-descent method (BMI problem is solved independently for
each coordinate at each step using a LMI optimisation)
4
spectral Bundle method;
5
attempts to use Augmented Lagrangian / Barrier methods.
*The Author promises NOT to embarrass the Audience with details here ;-)
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
34 Austra
35. Software toolboxes for numerical solution of non-convex
Introduction to Bilinear Matrix Inequalities SDPs and BMIs
Software toolboxes for numerical solution of non-convex
SDPs and BMIs
Solvers:
1
Parsers:
1
YALMIP LMI parser
PENBMI
-
penalty/barrier
function
yet
another
(Augmented
Lagrangian?)
2
HIFOO
-
hybrid
(quasi-Newton
bundling
and
method
updating,
gradient
sampling)
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
35 Austra
36. Summary for methods of solving non-convex SDPs and
BMIs
Summary for methods of solving non-convex
SDPs and BMIs
1
Bilinear Matrix Inequalities are in the realm of
non-convex Semidenite
Programming = Global Optimisation.
2
BMIs are known to be
non-convex
and
NP-hard in general,
hence
intractable in theory.
3
no reliable solver exists for non-convex SDP problems:
branch-and-
bound, simulated annealing, genetic algorithms, spectral bundle method,
barrier methods - no guarantee to nd a (global) solution within reasonable
time.
4
numerical algorithms are inhumanly dicult to devise, only few exist:
PENBMI and HIFOO (current on 2013).
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
36 Austra
37. Conclusion and Summary
Part IV ::
Conclusion and Summary
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
37 Austra
38. Conclusion: Linear Matrix Inequalities / convex SDPs
Conclusion: Bilinear Matrix Inequalities /
non-convex SDPs
1
a
Linear Matrix Inequality is a convex constraint in the form:
N
F0
2
+
i =1
0.
xi Fi
In mathematical programming terminology, to solve LMI means to
solve a Semidenite Programming (SDP) problem:
TX
minimize
C
subject to
x1 F1
x
Ax
- - this is an optimality metric!
+ · · · + xn Fn + F0
0
- - this is LMI!
=b
where the matrices F0 , F1 , . . . Fn
∈ Sk ,
and A
∈ Rp×n .
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
38 Austra
39. Conclusion: Linear Matrix Inequalities / convex SDPs
Conclusion: Bilinear Matrix Inequalities /
non-convex SDPs, continued
1
There are analytical tools that help to simplify an SDP/LMI problem:
Schur complement (most useful), S-Procedure, Elimination Lemma.
2
convex SDPs (LMIs) can be solved within reasonable time using welldeveloped numerical methods (Primal-dual Interior point, Logarithmic
barrier, Augmented Lagrangian).
3
To convert an LMI from the matrix form into semidenite (canonical)
form, we need
4
parsers (YALMIP, CVX).
To solve an LMI/SDP, we need solvers (SDPA, SDPT3, SeDuMi, LMI
Control Toolbox).
5
Author's success story: using YALMIP and SeDuMi, he solved an LMI
for Kalman gains for a small 2
×2
system.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
39 Austra
40. Conclusion: Bilinear Matrix Inequalities / non-convex SDPs
Conclusion: Bilinear Matrix Inequalities /
non-convex SDPs
1
a
Bilinear Matrix Inequality constraints are in the form:
n
F (x )
2
3
= F0 +
i =1
n
xi Fi
n
+
j =1 j =1
Fi ,j xi xj
0,
non-convex Semidenite Programming.
BMIs = non-convex SDPs, which are known to be NP-hard in general,
BMIs are in the realm of
hence intractable in theory.
4
no reliable solver exists for non-convex SDPs: branch-and-bound, genetic
algorithms, barrier methods - no guarantee to nd a (global) solution
within reasonable time.
5
numerical algorithms are dicult to devise, only few toolboxes exist.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
40 Austra
41. Conclusion: Bilinear Matrix Inequalities / non-convex SDPs
The End
- [Frodo:] I will take the Ring to Mordor!
[pause]
- [Frodo:] Though... I do not know the way.
Frodo Baggins
from The Lord of the Rings: The Fellowship of the Ring
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
41 Austra
42. Conclusion: Bilinear Matrix Inequalities / non-convex SDPs
Convex set of LMI constraints
Convex set of LMI constraints
Let us denote by
X
the set of points x
∈
Rm
that satisfy:
X :=
m
x
The set
: F0 +
X
i =1
xi Fi
0
.
is convex
since we have
F (x )
if
∀ :
0 if and only
z
T
z F (x )z ≥
∈ Rn :
0.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
42 Austra
43. Conclusion: Bilinear Matrix Inequalities / non-convex SDPs
Convex set of LMI constraints
Stephen Boyd, V Balakrishnan, E Feron, and Laurent El Ghaoui.
History of linear matrix inequalities in control theory.
In American Control Conference, 1994, volume 1, pages 3134. IEEE, 1994.
Imre Pólik and Tamás Terlaky.
A survey of the s-lemma.
SIAM review, 49(3):371418, 2007.
Stephen Boyd, Laurent El Ghaoui, Eric Feron, and Venkataramanan Balakrishnan.
Linear matrix inequalities in system and control theory, volume 15.
SIAM, 1994.
Onur Toker and Hitay Ozbay.
On the np-hardness of solving bilinear matrix inequalities and simultaneous stabilization with static output
feedback.
In American Control Conference, 1995. Proceedings of the, volume 4, pages 25252526. IEEE, 1995.
Didier Henrion.
Course on lmi: What is an lmi?
Technical report, www.laas.fr/~henrion, October 2006.
(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
42 Austra