SlideShare ist ein Scribd-Unternehmen logo
1 von 43
Downloaden Sie, um offline zu lesen
Seminar:
LP, QP, SOCP, SDP, LMI, BMI and other UFOs
(not-Dr.-yet) Mikhail V. Konnik
School of Electrical Engineering and Computer Science
The University of Newcastle
Australia

December 13, 2013

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
1 Austra
Disclaimer
the Author is not an expert (yet) in LMI/BMI/SDP (but he is doing his
best to become one), and therefore:
the whole presentation is just an overview of LMI/BMI from the point of
view of the Author's (severely incomplete) understanding of optimisation;
the Author is not ready to solve complicated BMI/UFO problems that
the people in the audience surely have :-)

the problem of solving even LMI (let alone BMI) is much more complicated
than many people in the audience think;
software packages for LMI/BMI are not as mature as for e.g. QP;
the Author is known for his anity to reinvent the wheel to better
understand how stu works.
The wheel that is being inventing by the Author is still in the blueprint
stage, is not round, and cannot be used for anything remotely serious.

the Author is not liable for any loss, damage, illness, injury, headache,
crash, collapse of the Universe or anything else caused by this presentation.
proceed at your own risk!

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
2 Austra
Introduction to Linear Matrix Inequalities

Part I ::
A short introduction to Linear Matrix Inequalities
(LMI) and Semidenite Programming (SDP)

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
3 Austra
Introduction to Linear Matrix Inequalities

LMI and Control

Linear Matrix Inequalities and Control

The story of LMI begins in about 1890, when Lyapunov published his seminal
work, where he showed that the dierential equation
d
dx

x (t )

= Ax (t )

is stable (i.e., all trajectories converge to zero) if and only if there exists a
positive-denite matrix P such that[1]
A
The requirement P

T P + PA

0

0 is what we now call a Lyapunov inequality on P .

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
4 Austra
Introduction to Linear Matrix Inequalities

LMI and Control

Linear Matrix Inequalities and Control
The Lyapunov inequality A

T P + PA 

0 is not the only example of how one

can convert control problems into LMI: Riccati inequality can be converted
into an LMI, too:
A

T P + PA + PBR −1 B T P + Q

0,

is equivalent to the following LMI (via Schur complement):

−AT P − PA − Q
B

The requirement P

TP

PB
R

0

(1)

0 is what we now call a Lyapunov inequality on P .

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
5 Austra
Introduction to Linear Matrix Inequalities

What is an LMI?

What is an LMI?
A (strict) linear matrix inequality (LMI) is a convex

constraint in the form:
N

F (x )

= F0 + x1 F1 + · · · + xN FN = F0 +

i =1

xi Fi

0,

(2)

where:
x

= [x1 , . . . , xN ]

is a vector of

unknown scalars (the decision

or optimisation variables) x1 , x2 and so on;
F0 , F1 , F2 , . . . , FN are n
The sign F

×n

symmetric matrices, they are given.

0 is a generalised inequality meaning F is a negative denite

matrix (i.e., the largest eigenvalue of F (x ) is negative), F
denite, and F

0 is positive

0 is positive semi-denite.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
6 Austra
Introduction to Linear Matrix Inequalities

What is an LMI?

What is an LMI? - A set of CONVEX constraints!
Note that the LMI:
F (x )

= F0 + x1 F1 + . . .
N

+xN FN = F0 +

i =1

0,

xi Fi

is just a convex constraint
on x , and therefore:
Its solution set, called
the

feasible set,

convex subset of

is a

RN

Finding a solution x of
LMI, if any, is a

convex

Figure 1: Supporting hyperplanes for LMI.

problem.
Convexity has an important consequence: even though LMI has no analytical
solution in general, it can be solved numerically.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
7 Austra
Introduction to Linear Matrix Inequalities

Solving Linear Matrix Inequalities in a brute-force way

Solving LMI in a brute-force way

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
8 Austra
Introduction to Linear Matrix Inequalities

Solving Linear Matrix Inequalities in a brute-force way

Solving LMI in a brute-force way
The LMI species a convex

constraint

on

x,

and

to

solve an LMI means that
we can nd x that satises
the LMI.
Brute-force: we can try the
variables in
x1 x2
X =
x2 x3
compute
Z

= AT X + XA

Figure 2:

The corresponding set from the LMI

F (x ) := x1 F1 + x2 F2

I.

check eigenvalues
E

= eig (Z )

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013
December
9 Austra
Semidenite Programming

We can do better: solve LMI via SDP
A problem which subsumes linear, quadratic,
geometric and second-order cone programming
is called a

semidenite program (SDP):
TX

minimize

C

subject to

F0

X

- - this is optimality metric!

N

Ax

+
i =1

xi Fi

0,

- - this is LMI!

=b

where the matrices F0 , F1 , . . . Fn

∈ Sk ,

and A

∈ Rp×n .

That is: LMI cuts a set of solutions via convex set of constraints, and SDP
allows to pick up an optimal solution using some optimality metric.

To solve LMI ⇔ solve a Semidenite Programming (SDP) problem.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
10 Austra
Semidenite Programming

What is Semidenite?
The set of real symmetric n × n matrices
is denoted

Sn .

A matrix Z

semidenite
Rn .

∈ Sn
if

is called

T
x Zx ≥

positive

0 for all x

∈

On the other hand, a matrix Z is

called

positive denite

for all nonzero x

∈ Rn .

if x

T Zx 

0

The set of positive semidenite matrices
is denoted

Sn .
+

In other words, the eigenvalues

λ1 . . . λn

of a positive

semidenite matrix Z+ are nonnegative (i.e., can be zeros or less than
machine epsilon
The set

Sn
+

).

is a convex cone.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
11 Austra
Semidenite Programming

What is Semidenite? (continued)
A semidenite program (SDP) is a generalization of a linear program (LP),

where the inequality constraints are replaced by matrix inequalities corresponding
to the cone of positive semidenite matrices.
An SDP in the pure primal form is dened as:
minimise

trace(CX )

subject to

trace(Ai X ) = bi ∀i = 1, . . . , m

X

X

(3)

0.

∈ Sn is the decision variable (can be a vector or a matrix), b ∈ Rm
+
n
C , A1 . . . Am ∈ S+ are given symmetric matrices. There are m ane

where X
and

constraints among the entries of the

positive semidenite matrix X .

feasible sets of SDPs
can have curved faces, together with sharp corners where faces intersect.
Unlike LP, where the feasible sets are polyhedra, the

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
12 Austra
Semidenite Programming

Scope of SDP

TX

minimise

C

subject to

F0

X

SDP:

N

minimise

SOCP:

c

x

+

QP:

x

subject to

LP:

2

x

Ax

0

- - - LMI

+ bi

2

 ciT x + di

=g

T Hx + c T x + d

b

Tx + d

minimise

c

subject to

Ax

x

xi Fi

Tx

Fx
1

i =1

Ai x

subject to

minimise

Scope of Semidenite Programming

b

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
13 Austra
Semidenite Programming

SDP: One Ring to Rule them All

Semidenite Programming SDP: One Ring to Rule them All
Using the power of the Sauron's One Ring SDP, re-cast a QP problem into
an SDP problem. Use Cholesky factor
x

subject to

Ax

minimise

X

subject to

C

TX

F (x )

0

:= blockdiag (FQ (x ), Ai x − bi ),

Ai is the ith row of A, and we use Schur complement r
FQ (x )

=

H.

≥b

Equivalent SDP:

where F (x )

of the Hessian matrix

T Q T Qx + c T x + r

minimise

x

H = QT Q

r

− cT x
Qx

x

T QT
I

≥ x T Q T Qx + c T x ,

0.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
14 Austra
Summary for SDP and LMI

Summary for SDP and LMI
1

A (strict) linear matrix inequality (LMI) is a convex
form:

N
F0

+
i =1

Finding a solution x of LMI, if any, is a

3

To solve LMI

⇔

convex problem.

solve a Semidenite Programming (SDP):

TX

minimize

C

subject to

x1 F1

x

0.

xi Fi

2

Ax

- - this is an optimality metric!

+ · · · + xn Fn + F0

0

- - this is LMI!

=b

where the matrices F0 , F1 , . . . Fn
4

constraint in the

∈ Sk ,

and A

∈ Rp×n .

LMI cuts a set of solutions via convex set of constraints, and SDP allows
to pick up an optimal solution using some optimality metric.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
15 Austra
Solving convex SDPs and LMIs

Part II ::
Solving Semidenite Problems (SDP)

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
16 Austra
Solving convex SDPs and LMIs

Analytical techniques and numerical algorithms

Solving convex SDPs and LMIs : analytic techniques and
numerical algorithms
Even small Semidenite problems (and LMIs) are too dicult to solve
analytically - use numerical methods!
1

Primal-dual Interior point methods

2

Logarithmic barrier methods

3

Augmented Lagrangian

There are some analytical methods that can help/simplify/relax the problem:
1

Schur complement - convert an LMI into a matrix

2

S-Procedure - a Lagrange relaxation technique for problems with quadratic
constraints (solves a system of quadratic inequalities via LMI).

3

The Elimination Lemma - eliminates some variables in LMI.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
17 Austra
Solving convex SDPs and LMIs

Schur complement

Analytical techniques: Schur complement
Schur complement is an indispensable tool for transforming non-linear
constraints into convex LMI. A symmetric matrix X can be decomposed:
X

The matrix S

C

− B T A− 1 B

In other words, for all X

∈

=

A
B

T

B

is called the

n

S , Y

(4)

C

Schur complement of A in X .

∈ Rm×n

, Z

∈

S

m

, the following

statements are equivalent:
0; X

a) Z
b)

− Y T Z −1 Y

X

T
Y

Y

Z

0.

0.

a) Z
b) Z

0; X
0 ;

− Y T Z −1 Y
X
Y

Y

T

Z

0.
0.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
18 Austra
Solving convex SDPs and LMIs

S-procedure

Analytical techniques: S-procedure for Quadratic Forms and
Strict Inequalities
In some problems, we nd that some quadratic function must be negative
whenever some other quadratic functions are all negative.

With the S-procedure, we can replace this problem by one inequality
to be satised by introducing some positive scalars to be determined.
Motivation:

The fundamental question of the theory of the S-lemma is the

following: ([2], page: 371).
When is a quadratic inequality a consequence of other quadratic
inequalities?

In short terms, the

S-procedure is a Lagrange relaxation method; it tries

to solve a system of quadratic inequalities via LMI.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
19 Austra
Solving convex SDPs and LMIs

S-procedure

Analytical techniques: S-procedure for Quadratic Forms and
Strict Inequalities
. . . Tp ∈ Rn×n be
condition on T0 . . . Tp :

Let T0

ξ T T0 ξ  0

symmetric matrices.

for all

It is obvious that if there exists

ξ=0

We consider the following

such that

τ1 ≥ 0, . . . τp ≥ 0

ξ T Ti ξ ≥ 0

(5)

such that:

p
T0

−
i =1

τi Ti

0,

then (5) holds.
For further details and bibliography comments, the reader is referred to [3].
*Please do NOT embarrass the Author with questions :-)

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
20 Austra
Solving convex SDPs and LMIs

S-procedure

Analytical techniques: S-procedure, continued - an
illustrative example
The following constraint on the variable P : ([3], page: 36)

∀ξ = 0
ξ
π

and

T

π

satisfying

T P + PA
T
B P

A

π T π ≤ ξ T C T C ξ,
ξ
π

PB
0

(6)

0

Applying the S-procedure, (6) is equivalent to:
A

T P + PA + τ C T C
T
B P

Thus the problem of nding P

0

PB

−τ I

0

such that (6) holds can be expressed as

an LMI in P and the scalar variable

τ.

([3], page: 36)

*Please do NOT embarrass the Author with questions here ;-)

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
21 Austra
Solving convex SDPs and LMIs

The Elimination Lemma

Analytical techniques: S-procedure, continued - an
illustrative example
The Elimination Lemma [3] allows to eliminate some variables appearing in
LMI leading to inequalities without the eliminated variable. Consider:
G (z )

+ U (z )XV (z )T + V (z )X T U (z )T  0,

(7)

ˆ
ˆ
Suppose that for every z , U (z ) and V (z ) are orthogonal complements of
U (z ) and V (z ) respectively. Then (7) holds for some X and z

=

z0 if and

only if the inequalities

T G (z )U (z )  0,
ˆ

ˆ
U (z )
hold with z

T G (z )V (z )  0,
ˆ

ˆ
V (z )

(8)

= z0 .

In other words, feasibility of the matrix inequality (7) with variables X and
z is equivalent to the feasibility of (8) with variable z .

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
22 Austra
Solving convex SDPs and LMIs

Numerical algorithms for convex SDP and LMI

Numerical algorithms for convex SDP and LMI

Schur complement, S-procedure, Elimination Lemma and other analytical
tricks will

NOT solve LMIs - use numerical methods!

1

Primal-dual Interior point methods

2

Logarithmic barrier methods

3

Augmented Lagrangian

LMIs are usually solved as SDP problems; therefore it is benecial to study
how the SDP solvers work.
*The Author promises NOT to embarrass the Audience with details here ;-)

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
23 Austra
Solving convex SDPs and LMIs

Canonical and matrix forms of LMIs

Numerical algorithms for convex SDP and LMI: Canonical
and matrix forms of LMIs
Generally, we do not encounter the LMI in the canonical form in control
theory but rather in the form of matrix variables. Thus, before the solution
of LMI commences via SDP, the LMI must be pre-processed, or

parsed.

For example, the Lyapunov's inequality:
A

T P + PA

0 P

= PT

0

(9)

can be written in the canonical form:

m
F (x )

where F0

=0

and Fi

= F0 +

i =1

= −AT Bi − Bi A,

xi Fi

0,

and where Bi , i

(10)

= 1, . . . , n(n + 1)/2

are matrix bases for symmetric matrices of size n.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
24 Austra
Solving convex SDPs and LMIs

Canonical and matrix forms of LMIs

Canonical form of LMI
Consider the Lyapunov inequality in
where A

−1
0

=

2

−2

and X

matrix form:

=

x1

x2

x2

x3

A

T X + XA

0

. The decision variables are

scalars x1 , x2 , and x3 of the matrix X .
Convert to the

A

canonical form that is F (x ) = F0 + x1 F1 + · · · + xN FN

T X + XA → · · · →

−2x1
2x1 − 3x2

− 3x2
4x2 − 4x3
2x1

:

0

Now we extract the coecients for each variable x1 , x2 and x3 :

−2x1
2x1 − 3x2

− 3x2
4x2 − 4x3

2x1

→ x1 ·

−2

2

2

0

matrix

F1

+x2 ·

0

−3

−3

4

matrix

Therefore, we converted the problem into LMI x1 F1

F2

0

0

0

+x3 ·

−4

matrix

+ x2 F2 + x3 F3

F3

0.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
25 Austra
Solving convex SDPs and LMIs

Software toolboxes for numerical solution of SDP and LMI

Software toolboxes for numerical solution of SDP and LMI
Solvers:

Parsers:
1

2

YALMIP - yet another
LMI parser
CVX - supports two
solvers:

SeDuMi

1

2

CVX

are

3

parsers,

SDP3 and PENSDP.

infeasible

-

infeasible

path-

SeDuMi

-

Self-Dualself-dual

embedding technique, Interior

is, they don't solve the actual
other solvers such as SeDuMi,

SDPT3

Minimization,

that

problem; instead, they rely on

Mehrotra-type

following algorithm

Please note that both YALMIP
and

-

primal-dual interior-point

and

SDPT3

SDPA

predictor-corrector

Point
4

LMI Control Toolbox
Lab,

Nemirovskii's

- LMI

projective

algorithm

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
26 Austra
Solving convex SDPs and LMIs

Software toolboxes for numerical solution of SDP and LMI

Author's success story: Kalman gains via LMI
˜
Denote the state estimation error as x
e

ˆ
= z − z.

=

x

ˆ
−x

and the estimation error

The overall system's dynamics:

˜
x (k

+ 1) = (A − KCy )˜(k ) +
x
e (k )

Bw

− KDyw

w (k )

(11)

˜
= Cz x (k ) + Dzw w (k )

Write the SDP for the state estimator synthesis:
min

Y ,X ,P

trace (X )


subject to

P

 (PA − YCy )T
(PB − YDyw )T
X

T
Cz
P

0

Cz
P
and Y

PA

− YCy

PB

− YDyw

P

0

0



Im



0

0

= PK ,

X

= P −1

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
27 Austra
Solving convex SDPs and LMIs

Software toolboxes for numerical solution of SDP and LMI

Author's success story: Kalman gains via LMI
An LMI for Kalman gains for a small 2

×2

system was solved using the

SeDuMi solver and the YALMIP parser interface:

X = sdpvar(2,2); Dyw = [0 0 1 0; 0 0 0 1];
Y = sdpvar(2,2); Cz = [1 0; 0 1];
P = sdpvar(2,2); Dzw = 0; Cy = C_k;
Bw = [0 0
0 0; ...
0 0.01 0 0];
Fset=set(P0) + set( [P, P*A_k-Y*Cy, P*Bw - Y*Dyw;...
(P*A_k-Y*Cy)', P, zeros(2,4);...
(P*Bw - Y*Dyw)', zeros(4,2), eye(4)]0 ) +...
set([X,Cz; Cz', P]0);
sol = solvesdp(Fset, trace(X));
The matrix of gains found via LMI is, of course, the same (up to numerical
errors) as the one found via Riccati equations.

K = new_X*new_Y

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
28 Austra
Summary for methods of solving SDPs

Summary for methods of solving SDPs
1

There are analytical tools that help to simplify an SDP/LMI problem:
Schur complement, S-Procedure, Elimination Lemma.

2

Even small Semidenite problems (and LMIs) are too dicult to solve
analytically - use numerical methods!

3

Good news:

convex SDPs (LMIs) are solvable within reasonable time

using numerical methods (Interior point, Logarithmic barrier, Augmented
Lagrangian).
4

convert an LMI from the matrix form into semidenite (canonical)
parsers (YALMIP, CVX).
To solve an LMI/SDP, we need solvers (SDPA, SDPT3, SeDuMi, LMI
To

form, we need

5

Control Toolbox).
6

Author's success story: using YALMIP and SeDuMi, he solved an LMI
for Kalman gains for a small 2

×2

system.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
29 Austra
Solving non-convex SDPs and BMIs

Part III ::
Solving non-convex Semidenite Problems and
Bilinear Matrix Inequalities (BMIs)

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
30 Austra
Introduction to Bilinear Matrix Inequalities

Bilinear Matrix Inequalities
While a Linear matrix inequality (LMI) denotes a constraint of the form:

n
F (x )

= F0 +

i =1

where Fi are xed symmetric matrices and x
a

0,

xi Fi

∈

(12)

n

is the decision variable,

R

Bilinear Matrix Inequality constraints are in the form:
n
F (x )

= F0 +

i =1

n
xi Fi

n

+
j =1 j =1

Fi ,j xi xj

0,

(13)

are denoted BMIs (bilinear matrix inequalities).
Bilinear Matrix Inequalities are in the realm of

non-convex

Semidenite

Programming.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
31 Austra
Introduction to Bilinear Matrix Inequalities

Bilinear Matrix Inequalities: a Challenge

non-convex and
NP-hard in general[4], hence intractable in theory.
BMIs correspond to non-convex and (possibly) non-smooth Semidenite
Optimisation problems with BMIs are known to be

Programming - this is global optimisation (i.e., exponential time, local
solutions...);
no reliable solver exists for non-convex SDP problems:

branch-and-

bound, genetic algorithms, spectral bundle method, barrier methods no guarantee to nd a (global) solution within reasonable time.
numerical algorithms are inhumanly dicult to devise: only few exist.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
32 Austra
Introduction to Bilinear Matrix Inequalities

Bilinear Matrix Inequalities: an Illustration
Below is an illustration of the

non-convex

objective

is

function,

max

3

an

example

of

a

BMI

problem.

x2

subject to

which

quadratic problem with linear

x

2
2
− 2x2 − x1 − x2 ≥ 0

− x1 − x2 − x1 x2 ≥ 0
1

+ x2 x1 ≥ 0

Figure 3:

Illustration of a BMI: a non-

convex feasible set delimited by circular
and hyperbolic arcs (adapted from[5]).

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
33 Austra
Introduction to Bilinear Matrix Inequalities

Numerical algorithms for non-convex SDPs and BMIs

Numerical algorithms for non-convex SDPs and BMIs
Don't even try to solve BMIs analytically - it is dicult even for numerical
methods of global optimisation (non-convex SDPs):
1

Genetic algorithms:

Read-coded Genetic Algorithms, Extrapolation-

directed Crossover (EDX), Minimal Generation Gap (MGG);
2

Branch-and-Bound, Branch-and-Cut methods (have troubles with solving
medium/large problems due to loose lower-bound approximations);

3

Coordinate-descent method (BMI problem is solved independently for
each coordinate at each step using a LMI optimisation)

4

spectral Bundle method;

5

attempts to use Augmented Lagrangian / Barrier methods.

*The Author promises NOT to embarrass the Audience with details here ;-)

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
34 Austra
Software toolboxes for numerical solution of non-convex
Introduction to Bilinear Matrix Inequalities SDPs and BMIs

Software toolboxes for numerical solution of non-convex
SDPs and BMIs

Solvers:
1

Parsers:
1

YALMIP LMI parser

PENBMI

-

penalty/barrier

function
yet

another

(Augmented

Lagrangian?)
2

HIFOO

-

hybrid

(quasi-Newton
bundling

and

method
updating,
gradient

sampling)

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
35 Austra
Summary for methods of solving non-convex SDPs and
BMIs

Summary for methods of solving non-convex
SDPs and BMIs
1

Bilinear Matrix Inequalities are in the realm of

non-convex Semidenite

Programming = Global Optimisation.
2

BMIs are known to be

non-convex

and

NP-hard in general,

hence

intractable in theory.
3

no reliable solver exists for non-convex SDP problems:

branch-and-

bound, simulated annealing, genetic algorithms, spectral bundle method,
barrier methods - no guarantee to nd a (global) solution within reasonable
time.
4

numerical algorithms are inhumanly dicult to devise, only few exist:
PENBMI and HIFOO (current on 2013).

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
36 Austra
Conclusion and Summary

Part IV ::
Conclusion and Summary

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
37 Austra
Conclusion: Linear Matrix Inequalities / convex SDPs

Conclusion: Bilinear Matrix Inequalities /
non-convex SDPs
1

a

Linear Matrix Inequality is a convex constraint in the form:
N
F0

2

+
i =1

0.

xi Fi

In mathematical programming terminology, to solve LMI means to
solve a Semidenite Programming (SDP) problem:

TX

minimize

C

subject to

x1 F1

x

Ax

- - this is an optimality metric!

+ · · · + xn Fn + F0

0

- - this is LMI!

=b

where the matrices F0 , F1 , . . . Fn

∈ Sk ,

and A

∈ Rp×n .

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
38 Austra
Conclusion: Linear Matrix Inequalities / convex SDPs

Conclusion: Bilinear Matrix Inequalities /
non-convex SDPs, continued
1

There are analytical tools that help to simplify an SDP/LMI problem:
Schur complement (most useful), S-Procedure, Elimination Lemma.

2

convex SDPs (LMIs) can be solved within reasonable time using welldeveloped numerical methods (Primal-dual Interior point, Logarithmic
barrier, Augmented Lagrangian).

3

To convert an LMI from the matrix form into semidenite (canonical)
form, we need

4

parsers (YALMIP, CVX).

To solve an LMI/SDP, we need solvers (SDPA, SDPT3, SeDuMi, LMI
Control Toolbox).

5

Author's success story: using YALMIP and SeDuMi, he solved an LMI
for Kalman gains for a small 2

×2

system.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
39 Austra
Conclusion: Bilinear Matrix Inequalities / non-convex SDPs

Conclusion: Bilinear Matrix Inequalities /
non-convex SDPs
1

a

Bilinear Matrix Inequality constraints are in the form:
n
F (x )

2
3

= F0 +

i =1

n
xi Fi

n

+
j =1 j =1

Fi ,j xi xj

0,

non-convex Semidenite Programming.
BMIs = non-convex SDPs, which are known to be NP-hard in general,
BMIs are in the realm of

hence intractable in theory.
4

no reliable solver exists for non-convex SDPs: branch-and-bound, genetic
algorithms, barrier methods - no guarantee to nd a (global) solution
within reasonable time.

5

numerical algorithms are dicult to devise, only few toolboxes exist.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
40 Austra
Conclusion: Bilinear Matrix Inequalities / non-convex SDPs

The End

- [Frodo:] I will take the Ring to Mordor!
[pause]
- [Frodo:] Though... I do not know the way.

Frodo Baggins
from The Lord of the Rings: The Fellowship of the Ring

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
41 Austra
Conclusion: Bilinear Matrix Inequalities / non-convex SDPs

Convex set of LMI constraints

Convex set of LMI constraints
Let us denote by

X

the set of points x

∈

Rm

that satisfy:

X :=

m
x

The set

: F0 +

X

i =1

xi Fi

0

.

is convex

since we have
F (x )
if

∀ :

0 if and only
z

T
z F (x )z ≥

∈ Rn :
0.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
42 Austra
Conclusion: Bilinear Matrix Inequalities / non-convex SDPs

Convex set of LMI constraints

Stephen Boyd, V Balakrishnan, E Feron, and Laurent El Ghaoui.
History of linear matrix inequalities in control theory.
In American Control Conference, 1994, volume 1, pages 3134. IEEE, 1994.
Imre Pólik and Tamás Terlaky.
A survey of the s-lemma.
SIAM review, 49(3):371418, 2007.
Stephen Boyd, Laurent El Ghaoui, Eric Feron, and Venkataramanan Balakrishnan.
Linear matrix inequalities in system and control theory, volume 15.
SIAM, 1994.
Onur Toker and Hitay Ozbay.
On the np-hardness of solving bilinear matrix inequalities and simultaneous stabilization with static output
feedback.
In American Control Conference, 1995. Proceedings of the, volume 4, pages 25252526. IEEE, 1995.
Didier Henrion.
Course on lmi: What is an lmi?
Technical report, www.laas.fr/~henrion, October 2006.

(not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42
Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013
December 13,
42 Austra

Weitere ähnliche Inhalte

Was ist angesagt?

About the 2-Banach Spaces
About the 2-Banach Spaces About the 2-Banach Spaces
About the 2-Banach Spaces IJMER
 
A Numerical Analytic Continuation and Its Application to Fourier Transform
A Numerical Analytic Continuation and Its Application to Fourier TransformA Numerical Analytic Continuation and Its Application to Fourier Transform
A Numerical Analytic Continuation and Its Application to Fourier TransformHidenoriOgata
 
Bron Kerbosch Algorithm - Presentation by Jun Zhai, Tianhang Qiang and Yizhen...
Bron Kerbosch Algorithm - Presentation by Jun Zhai, Tianhang Qiang and Yizhen...Bron Kerbosch Algorithm - Presentation by Jun Zhai, Tianhang Qiang and Yizhen...
Bron Kerbosch Algorithm - Presentation by Jun Zhai, Tianhang Qiang and Yizhen...Amrinder Arora
 
Csr2011 june17 14_00_bulatov
Csr2011 june17 14_00_bulatovCsr2011 june17 14_00_bulatov
Csr2011 june17 14_00_bulatovCSR2011
 
Csr2011 june17 14_00_bulatov
Csr2011 june17 14_00_bulatovCsr2011 june17 14_00_bulatov
Csr2011 june17 14_00_bulatovCSR2011
 
20 k-means, k-center, k-meoids and variations
20 k-means, k-center, k-meoids and variations20 k-means, k-center, k-meoids and variations
20 k-means, k-center, k-meoids and variationsAndres Mendez-Vazquez
 
Algorithm chapter 10
Algorithm chapter 10Algorithm chapter 10
Algorithm chapter 10chidabdu
 
Introduction to Max-SAT and Max-SAT Evaluation
Introduction to Max-SAT and Max-SAT EvaluationIntroduction to Max-SAT and Max-SAT Evaluation
Introduction to Max-SAT and Max-SAT EvaluationMasahiro Sakai
 
Meta-learning and the ELBO
Meta-learning and the ELBOMeta-learning and the ELBO
Meta-learning and the ELBOYoonho Lee
 
Writing a SAT solver as a hobby project
Writing a SAT solver as a hobby projectWriting a SAT solver as a hobby project
Writing a SAT solver as a hobby projectMasahiro Sakai
 
Radial Basis Function Interpolation
Radial Basis Function InterpolationRadial Basis Function Interpolation
Radial Basis Function InterpolationJesse Bettencourt
 
Contra qpi continuous functions in ideal bitopological spaces
Contra qpi  continuous functions in ideal bitopological spacesContra qpi  continuous functions in ideal bitopological spaces
Contra qpi continuous functions in ideal bitopological spacesAlexander Decker
 
(DL hacks輪読) How to Train Deep Variational Autoencoders and Probabilistic Lad...
(DL hacks輪読) How to Train Deep Variational Autoencoders and Probabilistic Lad...(DL hacks輪読) How to Train Deep Variational Autoencoders and Probabilistic Lad...
(DL hacks輪読) How to Train Deep Variational Autoencoders and Probabilistic Lad...Masahiro Suzuki
 
Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed l_1/l_2 Regular...
Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed l_1/l_2 Regular...Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed l_1/l_2 Regular...
Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed l_1/l_2 Regular...Laurent Duval
 
Cryptography Baby Step Giant Step
Cryptography Baby Step Giant StepCryptography Baby Step Giant Step
Cryptography Baby Step Giant StepSAUVIK BISWAS
 

Was ist angesagt? (20)

About the 2-Banach Spaces
About the 2-Banach Spaces About the 2-Banach Spaces
About the 2-Banach Spaces
 
A Numerical Analytic Continuation and Its Application to Fourier Transform
A Numerical Analytic Continuation and Its Application to Fourier TransformA Numerical Analytic Continuation and Its Application to Fourier Transform
A Numerical Analytic Continuation and Its Application to Fourier Transform
 
Mohammad Sabawi NTCCIT-2018 Presentation
Mohammad Sabawi NTCCIT-2018 PresentationMohammad Sabawi NTCCIT-2018 Presentation
Mohammad Sabawi NTCCIT-2018 Presentation
 
Bron Kerbosch Algorithm - Presentation by Jun Zhai, Tianhang Qiang and Yizhen...
Bron Kerbosch Algorithm - Presentation by Jun Zhai, Tianhang Qiang and Yizhen...Bron Kerbosch Algorithm - Presentation by Jun Zhai, Tianhang Qiang and Yizhen...
Bron Kerbosch Algorithm - Presentation by Jun Zhai, Tianhang Qiang and Yizhen...
 
Csr2011 june17 14_00_bulatov
Csr2011 june17 14_00_bulatovCsr2011 june17 14_00_bulatov
Csr2011 june17 14_00_bulatov
 
Csr2011 june17 14_00_bulatov
Csr2011 june17 14_00_bulatovCsr2011 june17 14_00_bulatov
Csr2011 june17 14_00_bulatov
 
20 k-means, k-center, k-meoids and variations
20 k-means, k-center, k-meoids and variations20 k-means, k-center, k-meoids and variations
20 k-means, k-center, k-meoids and variations
 
Back tracking
Back trackingBack tracking
Back tracking
 
Algorithm chapter 10
Algorithm chapter 10Algorithm chapter 10
Algorithm chapter 10
 
Introduction to Max-SAT and Max-SAT Evaluation
Introduction to Max-SAT and Max-SAT EvaluationIntroduction to Max-SAT and Max-SAT Evaluation
Introduction to Max-SAT and Max-SAT Evaluation
 
Meta-learning and the ELBO
Meta-learning and the ELBOMeta-learning and the ELBO
Meta-learning and the ELBO
 
Writing a SAT solver as a hobby project
Writing a SAT solver as a hobby projectWriting a SAT solver as a hobby project
Writing a SAT solver as a hobby project
 
Radial Basis Function Interpolation
Radial Basis Function InterpolationRadial Basis Function Interpolation
Radial Basis Function Interpolation
 
Iclr2016 vaeまとめ
Iclr2016 vaeまとめIclr2016 vaeまとめ
Iclr2016 vaeまとめ
 
Wg qcolorable
Wg qcolorableWg qcolorable
Wg qcolorable
 
Np completeness
Np completenessNp completeness
Np completeness
 
Contra qpi continuous functions in ideal bitopological spaces
Contra qpi  continuous functions in ideal bitopological spacesContra qpi  continuous functions in ideal bitopological spaces
Contra qpi continuous functions in ideal bitopological spaces
 
(DL hacks輪読) How to Train Deep Variational Autoencoders and Probabilistic Lad...
(DL hacks輪読) How to Train Deep Variational Autoencoders and Probabilistic Lad...(DL hacks輪読) How to Train Deep Variational Autoencoders and Probabilistic Lad...
(DL hacks輪読) How to Train Deep Variational Autoencoders and Probabilistic Lad...
 
Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed l_1/l_2 Regular...
Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed l_1/l_2 Regular...Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed l_1/l_2 Regular...
Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed l_1/l_2 Regular...
 
Cryptography Baby Step Giant Step
Cryptography Baby Step Giant StepCryptography Baby Step Giant Step
Cryptography Baby Step Giant Step
 

Ähnlich wie Seminar on Optimization Techniques: LP, QP, SOCP, SDP, LMI, BMI

Matrix Completion Presentation
Matrix Completion PresentationMatrix Completion Presentation
Matrix Completion PresentationMichael Hankin
 
Introduction to Full Counting Statistics
Introduction to Full Counting StatisticsIntroduction to Full Counting Statistics
Introduction to Full Counting StatisticsKrzysztof Pomorski
 
Amirim Project - Threshold Functions in Random Simplicial Complexes - Avichai...
Amirim Project - Threshold Functions in Random Simplicial Complexes - Avichai...Amirim Project - Threshold Functions in Random Simplicial Complexes - Avichai...
Amirim Project - Threshold Functions in Random Simplicial Complexes - Avichai...Avichai Cohen
 
Lec04 min cost linear problems
Lec04 min cost linear problemsLec04 min cost linear problems
Lec04 min cost linear problemsDanny Luk
 
A study of the worst case ratio of a simple algorithm for simple assembly lin...
A study of the worst case ratio of a simple algorithm for simple assembly lin...A study of the worst case ratio of a simple algorithm for simple assembly lin...
A study of the worst case ratio of a simple algorithm for simple assembly lin...narmo
 
Wavelets and Other Adaptive Methods
Wavelets and Other Adaptive MethodsWavelets and Other Adaptive Methods
Wavelets and Other Adaptive MethodsKamrul Hasan
 
slides_cedric_weber_1.pdf
slides_cedric_weber_1.pdfslides_cedric_weber_1.pdf
slides_cedric_weber_1.pdfsasdude1
 
Topographic graph clustering with kernel and dissimilarity methods
Topographic graph clustering with kernel and dissimilarity methodsTopographic graph clustering with kernel and dissimilarity methods
Topographic graph clustering with kernel and dissimilarity methodstuxette
 
20070823
2007082320070823
20070823neostar
 
Graph Spectra through Network Complexity Measures: Information Content of Eig...
Graph Spectra through Network Complexity Measures: Information Content of Eig...Graph Spectra through Network Complexity Measures: Information Content of Eig...
Graph Spectra through Network Complexity Measures: Information Content of Eig...Hector Zenil
 
Properties of field induced Josephson junction(s)
Properties of field induced Josephson junction(s)Properties of field induced Josephson junction(s)
Properties of field induced Josephson junction(s)Krzysztof Pomorski
 
Generating Chebychev Chaotic Sequence
Generating Chebychev Chaotic SequenceGenerating Chebychev Chaotic Sequence
Generating Chebychev Chaotic SequenceCheng-An Yang
 
An Introduction to Spectral Graph Theory
An Introduction to Spectral Graph TheoryAn Introduction to Spectral Graph Theory
An Introduction to Spectral Graph Theoryjoisino
 
11.final paper -0047www.iiste.org call-for_paper-58
11.final paper -0047www.iiste.org call-for_paper-5811.final paper -0047www.iiste.org call-for_paper-58
11.final paper -0047www.iiste.org call-for_paper-58Alexander Decker
 
Computational methods and vibrational properties applied to materials modeling
Computational methods and vibrational properties applied to materials modelingComputational methods and vibrational properties applied to materials modeling
Computational methods and vibrational properties applied to materials modelingcippo1987Ita
 
StatPhysPerspectives_AMALEA_Cetraro_AnnaCarbone.pdf
StatPhysPerspectives_AMALEA_Cetraro_AnnaCarbone.pdfStatPhysPerspectives_AMALEA_Cetraro_AnnaCarbone.pdf
StatPhysPerspectives_AMALEA_Cetraro_AnnaCarbone.pdfAnna Carbone
 
Continuum Modeling and Control of Large Nonuniform Networks
Continuum Modeling and Control of Large Nonuniform NetworksContinuum Modeling and Control of Large Nonuniform Networks
Continuum Modeling and Control of Large Nonuniform NetworksYang Zhang
 
conference_poster_4
conference_poster_4conference_poster_4
conference_poster_4Jiayi Jiang
 
Master Thesis on the Mathematial Analysis of Neural Networks
Master Thesis on the Mathematial Analysis of Neural NetworksMaster Thesis on the Mathematial Analysis of Neural Networks
Master Thesis on the Mathematial Analysis of Neural NetworksAlina Leidinger
 

Ähnlich wie Seminar on Optimization Techniques: LP, QP, SOCP, SDP, LMI, BMI (20)

Matrix Completion Presentation
Matrix Completion PresentationMatrix Completion Presentation
Matrix Completion Presentation
 
Introduction to Full Counting Statistics
Introduction to Full Counting StatisticsIntroduction to Full Counting Statistics
Introduction to Full Counting Statistics
 
Amirim Project - Threshold Functions in Random Simplicial Complexes - Avichai...
Amirim Project - Threshold Functions in Random Simplicial Complexes - Avichai...Amirim Project - Threshold Functions in Random Simplicial Complexes - Avichai...
Amirim Project - Threshold Functions in Random Simplicial Complexes - Avichai...
 
Lec04 min cost linear problems
Lec04 min cost linear problemsLec04 min cost linear problems
Lec04 min cost linear problems
 
QMC Program: Trends and Advances in Monte Carlo Sampling Algorithms Workshop,...
QMC Program: Trends and Advances in Monte Carlo Sampling Algorithms Workshop,...QMC Program: Trends and Advances in Monte Carlo Sampling Algorithms Workshop,...
QMC Program: Trends and Advances in Monte Carlo Sampling Algorithms Workshop,...
 
A study of the worst case ratio of a simple algorithm for simple assembly lin...
A study of the worst case ratio of a simple algorithm for simple assembly lin...A study of the worst case ratio of a simple algorithm for simple assembly lin...
A study of the worst case ratio of a simple algorithm for simple assembly lin...
 
Wavelets and Other Adaptive Methods
Wavelets and Other Adaptive MethodsWavelets and Other Adaptive Methods
Wavelets and Other Adaptive Methods
 
slides_cedric_weber_1.pdf
slides_cedric_weber_1.pdfslides_cedric_weber_1.pdf
slides_cedric_weber_1.pdf
 
Topographic graph clustering with kernel and dissimilarity methods
Topographic graph clustering with kernel and dissimilarity methodsTopographic graph clustering with kernel and dissimilarity methods
Topographic graph clustering with kernel and dissimilarity methods
 
20070823
2007082320070823
20070823
 
Graph Spectra through Network Complexity Measures: Information Content of Eig...
Graph Spectra through Network Complexity Measures: Information Content of Eig...Graph Spectra through Network Complexity Measures: Information Content of Eig...
Graph Spectra through Network Complexity Measures: Information Content of Eig...
 
Properties of field induced Josephson junction(s)
Properties of field induced Josephson junction(s)Properties of field induced Josephson junction(s)
Properties of field induced Josephson junction(s)
 
Generating Chebychev Chaotic Sequence
Generating Chebychev Chaotic SequenceGenerating Chebychev Chaotic Sequence
Generating Chebychev Chaotic Sequence
 
An Introduction to Spectral Graph Theory
An Introduction to Spectral Graph TheoryAn Introduction to Spectral Graph Theory
An Introduction to Spectral Graph Theory
 
11.final paper -0047www.iiste.org call-for_paper-58
11.final paper -0047www.iiste.org call-for_paper-5811.final paper -0047www.iiste.org call-for_paper-58
11.final paper -0047www.iiste.org call-for_paper-58
 
Computational methods and vibrational properties applied to materials modeling
Computational methods and vibrational properties applied to materials modelingComputational methods and vibrational properties applied to materials modeling
Computational methods and vibrational properties applied to materials modeling
 
StatPhysPerspectives_AMALEA_Cetraro_AnnaCarbone.pdf
StatPhysPerspectives_AMALEA_Cetraro_AnnaCarbone.pdfStatPhysPerspectives_AMALEA_Cetraro_AnnaCarbone.pdf
StatPhysPerspectives_AMALEA_Cetraro_AnnaCarbone.pdf
 
Continuum Modeling and Control of Large Nonuniform Networks
Continuum Modeling and Control of Large Nonuniform NetworksContinuum Modeling and Control of Large Nonuniform Networks
Continuum Modeling and Control of Large Nonuniform Networks
 
conference_poster_4
conference_poster_4conference_poster_4
conference_poster_4
 
Master Thesis on the Mathematial Analysis of Neural Networks
Master Thesis on the Mathematial Analysis of Neural NetworksMaster Thesis on the Mathematial Analysis of Neural Networks
Master Thesis on the Mathematial Analysis of Neural Networks
 

Kürzlich hochgeladen

Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17Celine George
 
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdfInclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdfTechSoup
 
Judging the Relevance and worth of ideas part 2.pptx
Judging the Relevance  and worth of ideas part 2.pptxJudging the Relevance  and worth of ideas part 2.pptx
Judging the Relevance and worth of ideas part 2.pptxSherlyMaeNeri
 
DATA STRUCTURE AND ALGORITHM for beginners
DATA STRUCTURE AND ALGORITHM for beginnersDATA STRUCTURE AND ALGORITHM for beginners
DATA STRUCTURE AND ALGORITHM for beginnersSabitha Banu
 
ANG SEKTOR NG agrikultura.pptx QUARTER 4
ANG SEKTOR NG agrikultura.pptx QUARTER 4ANG SEKTOR NG agrikultura.pptx QUARTER 4
ANG SEKTOR NG agrikultura.pptx QUARTER 4MiaBumagat1
 
Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17Celine George
 
Choosing the Right CBSE School A Comprehensive Guide for Parents
Choosing the Right CBSE School A Comprehensive Guide for ParentsChoosing the Right CBSE School A Comprehensive Guide for Parents
Choosing the Right CBSE School A Comprehensive Guide for Parentsnavabharathschool99
 
How to do quick user assign in kanban in Odoo 17 ERP
How to do quick user assign in kanban in Odoo 17 ERPHow to do quick user assign in kanban in Odoo 17 ERP
How to do quick user assign in kanban in Odoo 17 ERPCeline George
 
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptxMULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptxAnupkumar Sharma
 
ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...
ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...
ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...JhezDiaz1
 
Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)
Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)
Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)lakshayb543
 
ISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITY
ISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITYISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITY
ISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITYKayeClaireEstoconing
 
Science 7 Quarter 4 Module 2: Natural Resources.pptx
Science 7 Quarter 4 Module 2: Natural Resources.pptxScience 7 Quarter 4 Module 2: Natural Resources.pptx
Science 7 Quarter 4 Module 2: Natural Resources.pptxMaryGraceBautista27
 
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptxINTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptxHumphrey A Beña
 
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATIONTHEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATIONHumphrey A Beña
 
AMERICAN LANGUAGE HUB_Level2_Student'sBook_Answerkey.pdf
AMERICAN LANGUAGE HUB_Level2_Student'sBook_Answerkey.pdfAMERICAN LANGUAGE HUB_Level2_Student'sBook_Answerkey.pdf
AMERICAN LANGUAGE HUB_Level2_Student'sBook_Answerkey.pdfphamnguyenenglishnb
 

Kürzlich hochgeladen (20)

Model Call Girl in Tilak Nagar Delhi reach out to us at 🔝9953056974🔝
Model Call Girl in Tilak Nagar Delhi reach out to us at 🔝9953056974🔝Model Call Girl in Tilak Nagar Delhi reach out to us at 🔝9953056974🔝
Model Call Girl in Tilak Nagar Delhi reach out to us at 🔝9953056974🔝
 
Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 3 STEPS Using Odoo 17
 
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdfInclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
 
Judging the Relevance and worth of ideas part 2.pptx
Judging the Relevance  and worth of ideas part 2.pptxJudging the Relevance  and worth of ideas part 2.pptx
Judging the Relevance and worth of ideas part 2.pptx
 
DATA STRUCTURE AND ALGORITHM for beginners
DATA STRUCTURE AND ALGORITHM for beginnersDATA STRUCTURE AND ALGORITHM for beginners
DATA STRUCTURE AND ALGORITHM for beginners
 
ANG SEKTOR NG agrikultura.pptx QUARTER 4
ANG SEKTOR NG agrikultura.pptx QUARTER 4ANG SEKTOR NG agrikultura.pptx QUARTER 4
ANG SEKTOR NG agrikultura.pptx QUARTER 4
 
YOUVE GOT EMAIL_FINALS_EL_DORADO_2024.pptx
YOUVE GOT EMAIL_FINALS_EL_DORADO_2024.pptxYOUVE GOT EMAIL_FINALS_EL_DORADO_2024.pptx
YOUVE GOT EMAIL_FINALS_EL_DORADO_2024.pptx
 
Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17
 
Choosing the Right CBSE School A Comprehensive Guide for Parents
Choosing the Right CBSE School A Comprehensive Guide for ParentsChoosing the Right CBSE School A Comprehensive Guide for Parents
Choosing the Right CBSE School A Comprehensive Guide for Parents
 
TataKelola dan KamSiber Kecerdasan Buatan v022.pdf
TataKelola dan KamSiber Kecerdasan Buatan v022.pdfTataKelola dan KamSiber Kecerdasan Buatan v022.pdf
TataKelola dan KamSiber Kecerdasan Buatan v022.pdf
 
YOUVE_GOT_EMAIL_PRELIMS_EL_DORADO_2024.pptx
YOUVE_GOT_EMAIL_PRELIMS_EL_DORADO_2024.pptxYOUVE_GOT_EMAIL_PRELIMS_EL_DORADO_2024.pptx
YOUVE_GOT_EMAIL_PRELIMS_EL_DORADO_2024.pptx
 
How to do quick user assign in kanban in Odoo 17 ERP
How to do quick user assign in kanban in Odoo 17 ERPHow to do quick user assign in kanban in Odoo 17 ERP
How to do quick user assign in kanban in Odoo 17 ERP
 
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptxMULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
 
ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...
ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...
ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...
 
Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)
Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)
Visit to a blind student's school🧑‍🦯🧑‍🦯(community medicine)
 
ISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITY
ISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITYISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITY
ISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITY
 
Science 7 Quarter 4 Module 2: Natural Resources.pptx
Science 7 Quarter 4 Module 2: Natural Resources.pptxScience 7 Quarter 4 Module 2: Natural Resources.pptx
Science 7 Quarter 4 Module 2: Natural Resources.pptx
 
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptxINTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
INTRODUCTION TO CATHOLIC CHRISTOLOGY.pptx
 
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATIONTHEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
 
AMERICAN LANGUAGE HUB_Level2_Student'sBook_Answerkey.pdf
AMERICAN LANGUAGE HUB_Level2_Student'sBook_Answerkey.pdfAMERICAN LANGUAGE HUB_Level2_Student'sBook_Answerkey.pdf
AMERICAN LANGUAGE HUB_Level2_Student'sBook_Answerkey.pdf
 

Seminar on Optimization Techniques: LP, QP, SOCP, SDP, LMI, BMI

  • 1. Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer Science The University of Newcastle Australia December 13, 2013 (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013 December 1 Austra
  • 2. Disclaimer the Author is not an expert (yet) in LMI/BMI/SDP (but he is doing his best to become one), and therefore: the whole presentation is just an overview of LMI/BMI from the point of view of the Author's (severely incomplete) understanding of optimisation; the Author is not ready to solve complicated BMI/UFO problems that the people in the audience surely have :-) the problem of solving even LMI (let alone BMI) is much more complicated than many people in the audience think; software packages for LMI/BMI are not as mature as for e.g. QP; the Author is known for his anity to reinvent the wheel to better understand how stu works. The wheel that is being inventing by the Author is still in the blueprint stage, is not round, and cannot be used for anything remotely serious. the Author is not liable for any loss, damage, illness, injury, headache, crash, collapse of the Universe or anything else caused by this presentation. proceed at your own risk! (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013 December 2 Austra
  • 3. Introduction to Linear Matrix Inequalities Part I :: A short introduction to Linear Matrix Inequalities (LMI) and Semidenite Programming (SDP) (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013 December 3 Austra
  • 4. Introduction to Linear Matrix Inequalities LMI and Control Linear Matrix Inequalities and Control The story of LMI begins in about 1890, when Lyapunov published his seminal work, where he showed that the dierential equation d dx x (t ) = Ax (t ) is stable (i.e., all trajectories converge to zero) if and only if there exists a positive-denite matrix P such that[1] A The requirement P T P + PA 0 0 is what we now call a Lyapunov inequality on P . (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013 December 4 Austra
  • 5. Introduction to Linear Matrix Inequalities LMI and Control Linear Matrix Inequalities and Control The Lyapunov inequality A T P + PA 0 is not the only example of how one can convert control problems into LMI: Riccati inequality can be converted into an LMI, too: A T P + PA + PBR −1 B T P + Q 0, is equivalent to the following LMI (via Schur complement): −AT P − PA − Q B The requirement P TP PB R 0 (1) 0 is what we now call a Lyapunov inequality on P . (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013 December 5 Austra
  • 6. Introduction to Linear Matrix Inequalities What is an LMI? What is an LMI? A (strict) linear matrix inequality (LMI) is a convex constraint in the form: N F (x ) = F0 + x1 F1 + · · · + xN FN = F0 + i =1 xi Fi 0, (2) where: x = [x1 , . . . , xN ] is a vector of unknown scalars (the decision or optimisation variables) x1 , x2 and so on; F0 , F1 , F2 , . . . , FN are n The sign F ×n symmetric matrices, they are given. 0 is a generalised inequality meaning F is a negative denite matrix (i.e., the largest eigenvalue of F (x ) is negative), F denite, and F 0 is positive 0 is positive semi-denite. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013 December 6 Austra
  • 7. Introduction to Linear Matrix Inequalities What is an LMI? What is an LMI? - A set of CONVEX constraints! Note that the LMI: F (x ) = F0 + x1 F1 + . . . N +xN FN = F0 + i =1 0, xi Fi is just a convex constraint on x , and therefore: Its solution set, called the feasible set, convex subset of is a RN Finding a solution x of LMI, if any, is a convex Figure 1: Supporting hyperplanes for LMI. problem. Convexity has an important consequence: even though LMI has no analytical solution in general, it can be solved numerically. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013 December 7 Austra
  • 8. Introduction to Linear Matrix Inequalities Solving Linear Matrix Inequalities in a brute-force way Solving LMI in a brute-force way (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013 December 8 Austra
  • 9. Introduction to Linear Matrix Inequalities Solving Linear Matrix Inequalities in a brute-force way Solving LMI in a brute-force way The LMI species a convex constraint on x, and to solve an LMI means that we can nd x that satises the LMI. Brute-force: we can try the variables in x1 x2 X = x2 x3 compute Z = AT X + XA Figure 2: The corresponding set from the LMI F (x ) := x1 F1 + x2 F2 I. check eigenvalues E = eig (Z ) (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 13, 2013 December 9 Austra
  • 10. Semidenite Programming We can do better: solve LMI via SDP A problem which subsumes linear, quadratic, geometric and second-order cone programming is called a semidenite program (SDP): TX minimize C subject to F0 X - - this is optimality metric! N Ax + i =1 xi Fi 0, - - this is LMI! =b where the matrices F0 , F1 , . . . Fn ∈ Sk , and A ∈ Rp×n . That is: LMI cuts a set of solutions via convex set of constraints, and SDP allows to pick up an optimal solution using some optimality metric. To solve LMI ⇔ solve a Semidenite Programming (SDP) problem. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 10 Austra
  • 11. Semidenite Programming What is Semidenite? The set of real symmetric n × n matrices is denoted Sn . A matrix Z semidenite Rn . ∈ Sn if is called T x Zx ≥ positive 0 for all x ∈ On the other hand, a matrix Z is called positive denite for all nonzero x ∈ Rn . if x T Zx 0 The set of positive semidenite matrices is denoted Sn . + In other words, the eigenvalues λ1 . . . λn of a positive semidenite matrix Z+ are nonnegative (i.e., can be zeros or less than machine epsilon The set Sn + ). is a convex cone. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 11 Austra
  • 12. Semidenite Programming What is Semidenite? (continued) A semidenite program (SDP) is a generalization of a linear program (LP), where the inequality constraints are replaced by matrix inequalities corresponding to the cone of positive semidenite matrices. An SDP in the pure primal form is dened as: minimise trace(CX ) subject to trace(Ai X ) = bi ∀i = 1, . . . , m X X (3) 0. ∈ Sn is the decision variable (can be a vector or a matrix), b ∈ Rm + n C , A1 . . . Am ∈ S+ are given symmetric matrices. There are m ane where X and constraints among the entries of the positive semidenite matrix X . feasible sets of SDPs can have curved faces, together with sharp corners where faces intersect. Unlike LP, where the feasible sets are polyhedra, the (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 12 Austra
  • 13. Semidenite Programming Scope of SDP TX minimise C subject to F0 X SDP: N minimise SOCP: c x + QP: x subject to LP: 2 x Ax 0 - - - LMI + bi 2 ciT x + di =g T Hx + c T x + d b Tx + d minimise c subject to Ax x xi Fi Tx Fx 1 i =1 Ai x subject to minimise Scope of Semidenite Programming b (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 13 Austra
  • 14. Semidenite Programming SDP: One Ring to Rule them All Semidenite Programming SDP: One Ring to Rule them All Using the power of the Sauron's One Ring SDP, re-cast a QP problem into an SDP problem. Use Cholesky factor x subject to Ax minimise X subject to C TX F (x ) 0 := blockdiag (FQ (x ), Ai x − bi ), Ai is the ith row of A, and we use Schur complement r FQ (x ) = H. ≥b Equivalent SDP: where F (x ) of the Hessian matrix T Q T Qx + c T x + r minimise x H = QT Q r − cT x Qx x T QT I ≥ x T Q T Qx + c T x , 0. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 14 Austra
  • 15. Summary for SDP and LMI Summary for SDP and LMI 1 A (strict) linear matrix inequality (LMI) is a convex form: N F0 + i =1 Finding a solution x of LMI, if any, is a 3 To solve LMI ⇔ convex problem. solve a Semidenite Programming (SDP): TX minimize C subject to x1 F1 x 0. xi Fi 2 Ax - - this is an optimality metric! + · · · + xn Fn + F0 0 - - this is LMI! =b where the matrices F0 , F1 , . . . Fn 4 constraint in the ∈ Sk , and A ∈ Rp×n . LMI cuts a set of solutions via convex set of constraints, and SDP allows to pick up an optimal solution using some optimality metric. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 15 Austra
  • 16. Solving convex SDPs and LMIs Part II :: Solving Semidenite Problems (SDP) (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 16 Austra
  • 17. Solving convex SDPs and LMIs Analytical techniques and numerical algorithms Solving convex SDPs and LMIs : analytic techniques and numerical algorithms Even small Semidenite problems (and LMIs) are too dicult to solve analytically - use numerical methods! 1 Primal-dual Interior point methods 2 Logarithmic barrier methods 3 Augmented Lagrangian There are some analytical methods that can help/simplify/relax the problem: 1 Schur complement - convert an LMI into a matrix 2 S-Procedure - a Lagrange relaxation technique for problems with quadratic constraints (solves a system of quadratic inequalities via LMI). 3 The Elimination Lemma - eliminates some variables in LMI. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 17 Austra
  • 18. Solving convex SDPs and LMIs Schur complement Analytical techniques: Schur complement Schur complement is an indispensable tool for transforming non-linear constraints into convex LMI. A symmetric matrix X can be decomposed: X The matrix S C − B T A− 1 B In other words, for all X ∈ = A B T B is called the n S , Y (4) C Schur complement of A in X . ∈ Rm×n , Z ∈ S m , the following statements are equivalent: 0; X a) Z b) − Y T Z −1 Y X T Y Y Z 0. 0. a) Z b) Z 0; X 0 ; − Y T Z −1 Y X Y Y T Z 0. 0. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 18 Austra
  • 19. Solving convex SDPs and LMIs S-procedure Analytical techniques: S-procedure for Quadratic Forms and Strict Inequalities In some problems, we nd that some quadratic function must be negative whenever some other quadratic functions are all negative. With the S-procedure, we can replace this problem by one inequality to be satised by introducing some positive scalars to be determined. Motivation: The fundamental question of the theory of the S-lemma is the following: ([2], page: 371). When is a quadratic inequality a consequence of other quadratic inequalities? In short terms, the S-procedure is a Lagrange relaxation method; it tries to solve a system of quadratic inequalities via LMI. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 19 Austra
  • 20. Solving convex SDPs and LMIs S-procedure Analytical techniques: S-procedure for Quadratic Forms and Strict Inequalities . . . Tp ∈ Rn×n be condition on T0 . . . Tp : Let T0 ξ T T0 ξ 0 symmetric matrices. for all It is obvious that if there exists ξ=0 We consider the following such that τ1 ≥ 0, . . . τp ≥ 0 ξ T Ti ξ ≥ 0 (5) such that: p T0 − i =1 τi Ti 0, then (5) holds. For further details and bibliography comments, the reader is referred to [3]. *Please do NOT embarrass the Author with questions :-) (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 20 Austra
  • 21. Solving convex SDPs and LMIs S-procedure Analytical techniques: S-procedure, continued - an illustrative example The following constraint on the variable P : ([3], page: 36) ∀ξ = 0 ξ π and T π satisfying T P + PA T B P A π T π ≤ ξ T C T C ξ, ξ π PB 0 (6) 0 Applying the S-procedure, (6) is equivalent to: A T P + PA + τ C T C T B P Thus the problem of nding P 0 PB −τ I 0 such that (6) holds can be expressed as an LMI in P and the scalar variable τ. ([3], page: 36) *Please do NOT embarrass the Author with questions here ;-) (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 21 Austra
  • 22. Solving convex SDPs and LMIs The Elimination Lemma Analytical techniques: S-procedure, continued - an illustrative example The Elimination Lemma [3] allows to eliminate some variables appearing in LMI leading to inequalities without the eliminated variable. Consider: G (z ) + U (z )XV (z )T + V (z )X T U (z )T 0, (7) ˆ ˆ Suppose that for every z , U (z ) and V (z ) are orthogonal complements of U (z ) and V (z ) respectively. Then (7) holds for some X and z = z0 if and only if the inequalities T G (z )U (z ) 0, ˆ ˆ U (z ) hold with z T G (z )V (z ) 0, ˆ ˆ V (z ) (8) = z0 . In other words, feasibility of the matrix inequality (7) with variables X and z is equivalent to the feasibility of (8) with variable z . (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 22 Austra
  • 23. Solving convex SDPs and LMIs Numerical algorithms for convex SDP and LMI Numerical algorithms for convex SDP and LMI Schur complement, S-procedure, Elimination Lemma and other analytical tricks will NOT solve LMIs - use numerical methods! 1 Primal-dual Interior point methods 2 Logarithmic barrier methods 3 Augmented Lagrangian LMIs are usually solved as SDP problems; therefore it is benecial to study how the SDP solvers work. *The Author promises NOT to embarrass the Audience with details here ;-) (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 23 Austra
  • 24. Solving convex SDPs and LMIs Canonical and matrix forms of LMIs Numerical algorithms for convex SDP and LMI: Canonical and matrix forms of LMIs Generally, we do not encounter the LMI in the canonical form in control theory but rather in the form of matrix variables. Thus, before the solution of LMI commences via SDP, the LMI must be pre-processed, or parsed. For example, the Lyapunov's inequality: A T P + PA 0 P = PT 0 (9) can be written in the canonical form: m F (x ) where F0 =0 and Fi = F0 + i =1 = −AT Bi − Bi A, xi Fi 0, and where Bi , i (10) = 1, . . . , n(n + 1)/2 are matrix bases for symmetric matrices of size n. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 24 Austra
  • 25. Solving convex SDPs and LMIs Canonical and matrix forms of LMIs Canonical form of LMI Consider the Lyapunov inequality in where A −1 0 = 2 −2 and X matrix form: = x1 x2 x2 x3 A T X + XA 0 . The decision variables are scalars x1 , x2 , and x3 of the matrix X . Convert to the A canonical form that is F (x ) = F0 + x1 F1 + · · · + xN FN T X + XA → · · · → −2x1 2x1 − 3x2 − 3x2 4x2 − 4x3 2x1 : 0 Now we extract the coecients for each variable x1 , x2 and x3 : −2x1 2x1 − 3x2 − 3x2 4x2 − 4x3 2x1 → x1 · −2 2 2 0 matrix F1 +x2 · 0 −3 −3 4 matrix Therefore, we converted the problem into LMI x1 F1 F2 0 0 0 +x3 · −4 matrix + x2 F2 + x3 F3 F3 0. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 25 Austra
  • 26. Solving convex SDPs and LMIs Software toolboxes for numerical solution of SDP and LMI Software toolboxes for numerical solution of SDP and LMI Solvers: Parsers: 1 2 YALMIP - yet another LMI parser CVX - supports two solvers: SeDuMi 1 2 CVX are 3 parsers, SDP3 and PENSDP. infeasible - infeasible path- SeDuMi - Self-Dualself-dual embedding technique, Interior is, they don't solve the actual other solvers such as SeDuMi, SDPT3 Minimization, that problem; instead, they rely on Mehrotra-type following algorithm Please note that both YALMIP and - primal-dual interior-point and SDPT3 SDPA predictor-corrector Point 4 LMI Control Toolbox Lab, Nemirovskii's - LMI projective algorithm (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 26 Austra
  • 27. Solving convex SDPs and LMIs Software toolboxes for numerical solution of SDP and LMI Author's success story: Kalman gains via LMI ˜ Denote the state estimation error as x e ˆ = z − z. = x ˆ −x and the estimation error The overall system's dynamics: ˜ x (k + 1) = (A − KCy )˜(k ) + x e (k ) Bw − KDyw w (k ) (11) ˜ = Cz x (k ) + Dzw w (k ) Write the SDP for the state estimator synthesis: min Y ,X ,P trace (X )  subject to P  (PA − YCy )T (PB − YDyw )T X T Cz P 0 Cz P and Y PA − YCy PB − YDyw P 0 0  Im  0 0 = PK , X = P −1 (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 27 Austra
  • 28. Solving convex SDPs and LMIs Software toolboxes for numerical solution of SDP and LMI Author's success story: Kalman gains via LMI An LMI for Kalman gains for a small 2 ×2 system was solved using the SeDuMi solver and the YALMIP parser interface: X = sdpvar(2,2); Dyw = [0 0 1 0; 0 0 0 1]; Y = sdpvar(2,2); Cz = [1 0; 0 1]; P = sdpvar(2,2); Dzw = 0; Cy = C_k; Bw = [0 0 0 0; ... 0 0.01 0 0]; Fset=set(P0) + set( [P, P*A_k-Y*Cy, P*Bw - Y*Dyw;... (P*A_k-Y*Cy)', P, zeros(2,4);... (P*Bw - Y*Dyw)', zeros(4,2), eye(4)]0 ) +... set([X,Cz; Cz', P]0); sol = solvesdp(Fset, trace(X)); The matrix of gains found via LMI is, of course, the same (up to numerical errors) as the one found via Riccati equations. K = new_X*new_Y (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 28 Austra
  • 29. Summary for methods of solving SDPs Summary for methods of solving SDPs 1 There are analytical tools that help to simplify an SDP/LMI problem: Schur complement, S-Procedure, Elimination Lemma. 2 Even small Semidenite problems (and LMIs) are too dicult to solve analytically - use numerical methods! 3 Good news: convex SDPs (LMIs) are solvable within reasonable time using numerical methods (Interior point, Logarithmic barrier, Augmented Lagrangian). 4 convert an LMI from the matrix form into semidenite (canonical) parsers (YALMIP, CVX). To solve an LMI/SDP, we need solvers (SDPA, SDPT3, SeDuMi, LMI To form, we need 5 Control Toolbox). 6 Author's success story: using YALMIP and SeDuMi, he solved an LMI for Kalman gains for a small 2 ×2 system. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 29 Austra
  • 30. Solving non-convex SDPs and BMIs Part III :: Solving non-convex Semidenite Problems and Bilinear Matrix Inequalities (BMIs) (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 30 Austra
  • 31. Introduction to Bilinear Matrix Inequalities Bilinear Matrix Inequalities While a Linear matrix inequality (LMI) denotes a constraint of the form: n F (x ) = F0 + i =1 where Fi are xed symmetric matrices and x a 0, xi Fi ∈ (12) n is the decision variable, R Bilinear Matrix Inequality constraints are in the form: n F (x ) = F0 + i =1 n xi Fi n + j =1 j =1 Fi ,j xi xj 0, (13) are denoted BMIs (bilinear matrix inequalities). Bilinear Matrix Inequalities are in the realm of non-convex Semidenite Programming. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 31 Austra
  • 32. Introduction to Bilinear Matrix Inequalities Bilinear Matrix Inequalities: a Challenge non-convex and NP-hard in general[4], hence intractable in theory. BMIs correspond to non-convex and (possibly) non-smooth Semidenite Optimisation problems with BMIs are known to be Programming - this is global optimisation (i.e., exponential time, local solutions...); no reliable solver exists for non-convex SDP problems: branch-and- bound, genetic algorithms, spectral bundle method, barrier methods no guarantee to nd a (global) solution within reasonable time. numerical algorithms are inhumanly dicult to devise: only few exist. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 32 Austra
  • 33. Introduction to Bilinear Matrix Inequalities Bilinear Matrix Inequalities: an Illustration Below is an illustration of the non-convex objective is function, max 3 an example of a BMI problem. x2 subject to which quadratic problem with linear x 2 2 − 2x2 − x1 − x2 ≥ 0 − x1 − x2 − x1 x2 ≥ 0 1 + x2 x1 ≥ 0 Figure 3: Illustration of a BMI: a non- convex feasible set delimited by circular and hyperbolic arcs (adapted from[5]). (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 33 Austra
  • 34. Introduction to Bilinear Matrix Inequalities Numerical algorithms for non-convex SDPs and BMIs Numerical algorithms for non-convex SDPs and BMIs Don't even try to solve BMIs analytically - it is dicult even for numerical methods of global optimisation (non-convex SDPs): 1 Genetic algorithms: Read-coded Genetic Algorithms, Extrapolation- directed Crossover (EDX), Minimal Generation Gap (MGG); 2 Branch-and-Bound, Branch-and-Cut methods (have troubles with solving medium/large problems due to loose lower-bound approximations); 3 Coordinate-descent method (BMI problem is solved independently for each coordinate at each step using a LMI optimisation) 4 spectral Bundle method; 5 attempts to use Augmented Lagrangian / Barrier methods. *The Author promises NOT to embarrass the Audience with details here ;-) (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 34 Austra
  • 35. Software toolboxes for numerical solution of non-convex Introduction to Bilinear Matrix Inequalities SDPs and BMIs Software toolboxes for numerical solution of non-convex SDPs and BMIs Solvers: 1 Parsers: 1 YALMIP LMI parser PENBMI - penalty/barrier function yet another (Augmented Lagrangian?) 2 HIFOO - hybrid (quasi-Newton bundling and method updating, gradient sampling) (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 35 Austra
  • 36. Summary for methods of solving non-convex SDPs and BMIs Summary for methods of solving non-convex SDPs and BMIs 1 Bilinear Matrix Inequalities are in the realm of non-convex Semidenite Programming = Global Optimisation. 2 BMIs are known to be non-convex and NP-hard in general, hence intractable in theory. 3 no reliable solver exists for non-convex SDP problems: branch-and- bound, simulated annealing, genetic algorithms, spectral bundle method, barrier methods - no guarantee to nd a (global) solution within reasonable time. 4 numerical algorithms are inhumanly dicult to devise, only few exist: PENBMI and HIFOO (current on 2013). (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 36 Austra
  • 37. Conclusion and Summary Part IV :: Conclusion and Summary (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 37 Austra
  • 38. Conclusion: Linear Matrix Inequalities / convex SDPs Conclusion: Bilinear Matrix Inequalities / non-convex SDPs 1 a Linear Matrix Inequality is a convex constraint in the form: N F0 2 + i =1 0. xi Fi In mathematical programming terminology, to solve LMI means to solve a Semidenite Programming (SDP) problem: TX minimize C subject to x1 F1 x Ax - - this is an optimality metric! + · · · + xn Fn + F0 0 - - this is LMI! =b where the matrices F0 , F1 , . . . Fn ∈ Sk , and A ∈ Rp×n . (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 38 Austra
  • 39. Conclusion: Linear Matrix Inequalities / convex SDPs Conclusion: Bilinear Matrix Inequalities / non-convex SDPs, continued 1 There are analytical tools that help to simplify an SDP/LMI problem: Schur complement (most useful), S-Procedure, Elimination Lemma. 2 convex SDPs (LMIs) can be solved within reasonable time using welldeveloped numerical methods (Primal-dual Interior point, Logarithmic barrier, Augmented Lagrangian). 3 To convert an LMI from the matrix form into semidenite (canonical) form, we need 4 parsers (YALMIP, CVX). To solve an LMI/SDP, we need solvers (SDPA, SDPT3, SeDuMi, LMI Control Toolbox). 5 Author's success story: using YALMIP and SeDuMi, he solved an LMI for Kalman gains for a small 2 ×2 system. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 39 Austra
  • 40. Conclusion: Bilinear Matrix Inequalities / non-convex SDPs Conclusion: Bilinear Matrix Inequalities / non-convex SDPs 1 a Bilinear Matrix Inequality constraints are in the form: n F (x ) 2 3 = F0 + i =1 n xi Fi n + j =1 j =1 Fi ,j xi xj 0, non-convex Semidenite Programming. BMIs = non-convex SDPs, which are known to be NP-hard in general, BMIs are in the realm of hence intractable in theory. 4 no reliable solver exists for non-convex SDPs: branch-and-bound, genetic algorithms, barrier methods - no guarantee to nd a (global) solution within reasonable time. 5 numerical algorithms are dicult to devise, only few toolboxes exist. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 40 Austra
  • 41. Conclusion: Bilinear Matrix Inequalities / non-convex SDPs The End - [Frodo:] I will take the Ring to Mordor! [pause] - [Frodo:] Though... I do not know the way. Frodo Baggins from The Lord of the Rings: The Fellowship of the Ring (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 41 Austra
  • 42. Conclusion: Bilinear Matrix Inequalities / non-convex SDPs Convex set of LMI constraints Convex set of LMI constraints Let us denote by X the set of points x ∈ Rm that satisfy: X := m x The set : F0 + X i =1 xi Fi 0 . is convex since we have F (x ) if ∀ : 0 if and only z T z F (x )z ≥ ∈ Rn : 0. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 42 Austra
  • 43. Conclusion: Bilinear Matrix Inequalities / non-convex SDPs Convex set of LMI constraints Stephen Boyd, V Balakrishnan, E Feron, and Laurent El Ghaoui. History of linear matrix inequalities in control theory. In American Control Conference, 1994, volume 1, pages 3134. IEEE, 1994. Imre Pólik and Tamás Terlaky. A survey of the s-lemma. SIAM review, 49(3):371418, 2007. Stephen Boyd, Laurent El Ghaoui, Eric Feron, and Venkataramanan Balakrishnan. Linear matrix inequalities in system and control theory, volume 15. SIAM, 1994. Onur Toker and Hitay Ozbay. On the np-hardness of solving bilinear matrix inequalities and simultaneous stabilization with static output feedback. In American Control Conference, 1995. Proceedings of the, volume 4, pages 25252526. IEEE, 1995. Didier Henrion. Course on lmi: What is an lmi? Technical report, www.laas.fr/~henrion, October 2006. (not-Dr.-yet) Mikhail V. Konnik School of Electrical Engineering and Computer ScienceThe University of Newcastle / 42 Seminar: LP, QP, SOCP, SDP, LMI, BMI and other UFOs 2013 December 13, 42 Austra