The document discusses Karmarkar's algorithm, an interior point method for solving linear programming problems. It introduces key concepts of Karmarkar's algorithm such as projecting a vector onto the feasible region, Karmarkar's centering transformation, and Karmarkar's potential function. The original algorithm assumes the linear program is in canonical form and generates a sequence of interior points with decreasing objective function values using a projective transformation to move points to the center of the feasible region.
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Karmarkar's Interior Point Method
1. Karmarkar’s
Algorithm
AK Dhamija
Introduction Karmarkar’s Algorithm
Complexity
LP Problem An Interior Point Method of Linear Programming Problem
Klee-Minty
Example
Comparison
Original
Algorithm AK Dhamija
Steps
Iterations
Transformation
DIPR, DRDO
Affine Variant
Three Concepts
Example
Concepts 1 & 2 November 20, 2009
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 1/44
2. Overview
Karmarkar’s
Algorithm 1 Introduction
Complexity
AK Dhamija LP Problem
Klee-Minty Example
Introduction Comparison
Complexity
LP Problem 2 Original Algorithm
Klee-Minty Steps
Example
Iterations
Comparison
Transformation
Original
Algorithm 3 Affine Variant
Steps Three Concepts
Iterations Example
Transformation Concepts 1 & 2
Affine Variant & 3: Centering
Iterations
Three Concepts
Rescaling
Example
Concepts 1 & 2
Final Solution
& 3: Centering Another Example
Iterations
Rescaling 4 Further Issues
Final Solution
Another
Example 5 References
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 2/44
3. Algorithms
Karmarkar’s
Algorithm
AK Dhamija
Introduction Algorithms Complexity
Complexity
If n is the size of the problem (n may be number of variables in LP Problem)
LP Problem
Klee-Minty • Time Complexity - Time taken by algorithm
Example
Comparison • Space Complexity - Memory needed by algorithm
√
Original
• Increasing order of complexity : lg n, n, n, n lg n, n2 , n3 , 2n , n!
Algorithm • O(n2 ) = c1 n2 , O(2n ) = c2 2n
Steps
Iterations • Compare O(n2 ) = 100n2 , O(2n ) = 2n
Transformation • 22500 vs 32768 for n = 15
Affine Variant • 1000000 vs 1267650600228229401496703205376 for n = 100
Three Concepts
Example
• O(n2 ) = c1 n2 is polynomial time (aka efficient) algorithm and O(2n ) = c2 2n is exponential
time (aka inefficient) algorithm
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 3/44
4. LP Problem
Karmarkar’s
Algorithm
AK Dhamija
Problem Instance
Introduction • Set of n real variables e.g. x,y etc
Complexity
LP Problem
• Set of restrictions in the form of linear inequalities e.g. x + y ≤ 5, y ≥ 0.2 etc
Klee-Minty • Goal e.g. Maximize 4x + y
Example
Comparison
• Solution: (x, y) = (5, 0)
Original
Algorithm
Feasible Region
Steps
Iterations
Transformation
• Feasible region is convex
Affine Variant • Each constraint generates at most one edge in
Three Concepts the feasible region
Example
Concepts 1 & 2 • As you move the goal line, G, to increase it’s
& 3: Centering value, the point that will maximize it is one of
Iterations the ’corners’.
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 4/44
5. LP Algorithms
Karmarkar’s Naive Algorithm
Algorithm
• Find all intersection points, check these points to satisfy all constraints i.e feasibility, and find out
AK Dhamija the point of optimum goal value
• O(mn ) for n variables and m constraints - exponential in n
Introduction
Complexity
LP Problem Simplex (George B. Dantzig, 1947)
Klee-Minty
Example • Start from a feasible corner, and Move to the one that gives the highest value to the goal function
Comparison
• STOPPING condition : none of the neighboring points gives a higher value than the current one
Original
Algorithm
• Average time is linear; worst case time is exponential (covers all feasible corners)
Steps
• Remarkably successful in practice
Iterations • Integer Programming is NP-complete
Transformation
Affine Variant
Ellipsoidal Method (Leonid Khachiyan, 1979)
Three Concepts
Example
Concepts 1 & 2
• Runtime: polynomial
& 3: Centering • Theoretical value; not useful in practice
Iterations
Rescaling
Final Solution Interior point method (Narendra Karmarkar, 1984)
Another
Example • Runtime: polynomial
Further Issues • Practical value, Recently these have become competitive in practice with simplex.
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 5/44
6. An example of Exponential Time
Karmarkar’s
Algorithm
Klee-Minty Example
AK Dhamija
Introduction
Complexity
LP Problem
Klee-Minty
Example
Comparison
Original
Algorithm
Steps
Iterations
Transformation
Affine Variant
Developments
Three Concepts
Example
• The LP has n variables, n constraints and 2n extreme
Concepts 1 & 2
& 3: Centering
points
Iterations
Rescaling • The elementary simplex method, starting at x = 0, goes
Final Solution
Another
Example
through each of the extreme points before reaching the
Further Issues optimum solution at (0, 0, ..., 5n )
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 6/44
7. Klee-Minty Example
Karmarkar’s
Algorithm
Here is the pivot sequence for n = 3, which goes through all 8
AK Dhamija
extreme points, starting at the origin. Let s be the slack
Introduction variables.
Complexity
LP Problem
Klee-Minty
Example
Comparison
Original
Algorithm
Steps
Iterations
Transformation
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example Exponential Time Algorithm in Worst case
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 7/44
8. General Introduction
Karmarkar’s
Algorithm
AK Dhamija
Interior-point methods for linear programming
Introduction Components
Complexity
LP Problem • A breakthrough development in linear programming during
Klee-Minty
Example the 1980s
Comparison
Original • Started in 1984 by a young mathematician at AT&T Bell
Algorithm
Steps Labs, N. Karmarkar
Iterations
Transformation • Polynomial-time algorithm which can solve huge linear
Affine Variant
Three Concepts
programming problems beyond the reach of the simplex
Example
Concepts 1 & 2
method.
& 3: Centering
Iterations • Karmarkar’s method stimulated development in both
Rescaling
Final Solution interior-point and simplex methods.
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 8/44
9. Difference between Interior point methods and the
simplex method
Karmarkar’s
Algorithm
AK Dhamija The nature of trial solutions and Complexity
Introduction
Complexity
LP Problem Simplex Method
Klee-Minty
Example • CPF (Corner Point Feasible) solutions
Comparison
• Worst Case:No of iterations can increase exponentially in the number of variables n : O(2n )
Original • Practically remarkably successful except some huge problems (eg Airlines Scheduling etc)
Algorithm
Steps
Iterations
Transformation
Affine Variant Interior Point Methods
Three Concepts
Example • Interior points (points inside the boundary of the feasible region)
Concepts 1 & 2
& 3: Centering
• Polynomial time
Iterations • Advantageous in solving huge problems, but cumbersome for not-so-large problems
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 9/44
10. Karmarkar’s Original Algorithm
Karmarkar’s
Algorithm Karmarkar assumes that the LP is given in Canonical form of
AK Dhamija the problem
Introduction • Min Z = CX
Complexity
LP Problem • such that AX = 0, 1X = 1, X ≥ 0
Klee-Minty
Example
Comparison
Original Assumptions
Algorithm
Steps
1 1 1
• X0 = ( n , n , ...., n ) is a feasible solution
Iterations
Transformation
• Minimum(Z) = 0
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
To apply the algorithm to LP problem in standard form, a
Iterations
Rescaling
transformation is needed
Final Solution
Another
• Min Z = CX
Example
Further Issues • such that AX ≤ b, X ≥ 0
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 10/44
11. Karmarkar’s Original Algorithm
Karmarkar’s
Algorithm
Examples : Standard Vs Canonical Form
AK Dhamija Standard Form
Introduction Min Z = y1 + y2 (Z = CY)
Complexity
LP Problem
such that
Klee-Minty
Example
Comparison • y1 + 2y2 ≤ 2 (AY ≤ b)
Original
Algorithm
• y1 , y2 ≥ 0
Steps
Iterations
Transformation Canonical Form
Affine Variant
Three Concepts Min Z = 5x1 + 5x2 (Z = CX)
Example
Concepts 1 & 2 such that
& 3: Centering
Iterations
Rescaling • 3x1 + 8x2 + 3x3 - 2x4 = 0 (AX = 0)
Final Solution
Another
Example
• x1 + x2 + x3 + x4 = 1 (1X = 1)
Further Issues • xj ≥ 0, j = 1, 2, 3, 4
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 11/44
12. Karmarkar’s Original Algorithm
Karmarkar’s
Algorithm
AK Dhamija
The Principle Idea
Introduction
Complexity
Create a sequence of points x(0) , x(1) , x(2) , ..., x(k) having decreasing values of the objective function.
LP Problem
Klee-Minty In the kth step, the point x(k) is brought into the center of the simplexa by projective transformation.
Example
Comparison a
n-dimensional unit simplex S is the set of points(x1 , x2 , ...xn )T satisfying
Original
Algorithm
x1 + x2 + ... + xn = 1 and xj ≥ 0, j = 1, 2, ...n
Steps
Iterations
Transformation Three key Concepts
Affine Variant • Projection of a vector onto the set of X satisfying AX = 0
Three Concepts
Example • Karmarkar’s Centering Transformation
Concepts 1 & 2 • Karmarkar’s Potential Function
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 12/44
13. Three Concepts
Karmarkar’s Projection
Algorithm
• we want to move from a feasible point X0 to another feasible point X1 , that for some fixed vector
AK Dhamija v, will have a larger value of vX
• if we choose to move in direction d = (d1 , d2 , ...dn ) that solves the optimization problem
Introduction Max vd such that
Complexity Ad = 0, d1 + d2 + ...dn = 0 (so that Ad remains feasible)
LP Problem and ||d|| = 1
Klee-Minty then we will be moving in a feasible direction that maximizes the increase in vX per unit length
Example moved.
Comparison
• The direction d that solves this optimization problem is given by the projection of v onto X
Original satisfying AX = 0 and x1 + x2 + ... + xn = 0 and is given by [I − BT (BBT )−1 B]v,
Algorithm A
Steps where B =
1
Iterations
Transformation
Affine Variant
Three Concepts • Geometrically, if we can write v = p + w, where p
Example satisfies Ap = 0 and w is perpendicular to all
Concepts 1 & 2 vectors X satisfying AX = 0, then p is the
& 3: Centering projection of v onto the set of X satisfying AX = 0.
Iterations
Rescaling
Final Solution
• For Example, v = (−2, −1, 7) is projected onto a
Another set of 3-d vectors satisfying x3 = 0(ie x1 − x2
Example plane), then v = (−2, −1, 0) + (0, 0, 7). So here,
p = (−2, −1, 0) is the vector in the set of X
Further Issues satisfying AX = 0 that is closest to v.
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 13/44
14. Three Concepts
Karmarkar’s
Algorithm
Karmarkar’s Centering Transformation
AK Dhamija
• If xk is a point in S, then f ([x1 , x2 , ...xn ]|xk ) transforms a point [x1 , x2 , ...xn ]T in S into a
point [y1 , y2 , ...yn ] in S, where
Introduction xj
Complexity xk
j
LP Problem yj = r=n xr
r=1 xk
Klee-Minty r
Example
Comparison • Consider the LP
Min z = x1 + 3x2 - 3x3 such that
Original x2 - x3 = 0
Algorithm x1 + x2 + x3 = 1
Steps xi ≥ 0
Iterations This LP has a feasible solution [ 1 , 1 , 1 ]T and the optimal value of z is 0.
3 3 3
Transformation
The feasible point [ 1 , 3 , 8 ]T yields the following transformation
4 8
3
Affine Variant 1 3 3
f ([x1 , x2 , x3 ]|[ 4 , 8 , 8 ]) =
Three Concepts 8x2 8x3
Example 4x1 3 3
[x1 , x2 , x3 ]|[ 8x2 8x3 , 8x2 8x3 , 8x2 8x3 ]
Concepts 1 & 2 4x1 + + 4x1 + + 4x1 + +
3 3 3 3 3 3
& 3: Centering 1 , 1 , 1 ]|[ 1 , 3 , 3 ]) = [ 12 , 8 , 8 ]
Iterations
For example, f ([ 3 3 3 4 8 8 28 28 28
Rescaling
Final Solution
• We now refer to the variables x1 , x2 , ..., xn as being the original space and the variables
y1 , y2 , ..., yn as being the transformed space and the unit simplex involving variables
Another
Example y1 , y2 , ..., yn will be called transformed unit simplex.
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 14/44
15. Three Concepts
Karmarkar’s
Algorithm
AK Dhamija
Properties of Centering Transformation
Introduction • f (xk |xk ) = [ n , n , ..., n ]T
1 1 1
Complexity f (.|xk ) maps xk into the center of the transformed unit simplex.
LP Problem
Klee-Minty • f (x|xk ) ∈ S and For x = x , f (x|xk ) = f (x |xk )
Example Any point in S is transformed into a point in the transformed unit simplex, and no two points in S
Comparison can yield the same point (i.e. f is one-one mapping)
Original • For any point [y1 , y2 , ...yn ]T in S, there is a unique point [x1 , x2 , ...xn ]T in S satisfying
Algorithm
f ([x1 , x2 , ...xn ]T |xk ) = [y1 , y2 , ...yn ]T
Steps
Iterations
The point [x1 , x2 , ...xn ]T is given by
Transformation xk yj
j
r=n xk y
Affine Variant r=1 r r
Three Concepts we can write f −1 ([y1 , y2 , ...yn ]T |xk )
= [x1 , x2 , ...xn ]T
Example The above two equations imply that f is a one-one onto mapping from s to S.
Concepts 1 & 2
& 3: Centering
• A point x) in S will satisfy Ax = 0 if A[diag(xk )]f (x|xk ) = 0
Iterations
Feasible points in the original problem correspond to points y in the transformed unit simplex that
Rescaling satisfy A[diag(xk )]y = 0
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 15/44
16. Karmarkar’s Original Algorithm
Karmarkar’s Step k = 0
Algorithm
• Start with the solution point Y0 = X0 = ( n , n , ...., n )T
1 1 1
AK Dhamija
• Compute step length parameters r = √ 1
n(n−1)
and α =
(n−1)
3n
Introduction
Complexity
LP Problem Step k
Klee-Minty
Example Define
Comparison
• Dk = diag{Xk } = diag{xk1 , xk2 , ..., xkn }
Original
Algorithm • P=
ADk
1
Steps
Iterations Compute
Transformation
• cp = [I − PT (PPT )−1 P](CDk )T
Affine Variant
Three Concepts
• Ynew = Y0 - αr ||cp ||
p
c
Example
Concepts 1 & 2 • Xk+1 = 1D Y
D Y
k new
k new
& 3: Centering
Iterations • Z = CXk+1
Rescaling
Final Solution
• k=k+1
Another
Example
• Repeat iteration k until Z becomes less than prescribed tolerance
Dk −1 X
Further Issues Centering : X is brought to center by Y = −1
1D X
k
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 16/44
17. Karmarkar’s Original Algorithm
Karmarkar’s
Algorithm
AK Dhamija
Introduction
Complexity Certain Remarks
LP Problem
Klee-Minty
Example
• We move from the center of the transformed unit simplex in a direction opposite to the projection of
Comparison CDk )T onto the transformation of the feasible region( the set of y satisfying Adiag{Xk }y =
0). As per our explanation of projection, this ensures that we maintain feasibility(in the transformed
Original space) and move in a direction that maximizes the rate of decrease of CDk )T
Algorithm
Steps • By moving a distance αr from the center of the transformed unit simplex, we ensure that yk+1
Iterations will remain in the interior of the transformed simplex
Transformation
• When we use the inverse of Karmarkar’s centering transformation to transform yk+1 back into
Affine Variant xk+1 , the definition of projection imply that xk+1 will be feasible for the original LP
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 17/44
18. Karmarkar’s Original Algorithm
Karmarkar’s
Algorithm
AK Dhamija
Potential Function : Why do we project CDk )T rather than C)T onto the transformed space?
Introduction
Complexity • Because we are projecting CDk T rather than CT , we can not be sure that each iteration will
LP Problem decrease Z. In fact it is possible for CXk+1 > CXk to occur.
Klee-Minty
Example • Karmarkar’s Potential Function f (X) is defined as f (X) =
Comparison j=n T
sumj=1 ln( CX
x
), X = [x1 , x2 , ..., xn ]T
j
Original
Algorithm • Karmarkar showed that if we project CDk )T (not C)T ) onto the feasible region in the
Steps transformed space, then for some δ > 0, it will be true that for k = 0, 1, 2, ... f (Xk )
Iterations - f (Xk+1 ) ≥ δ.
Transformation
• So, each iteration of karmarkar’s algorithm decreases the potential function by an amount bounded
Affine Variant away from 0.
Three Concepts
Example • Karmarkar showed that if the potential function evaluated at xk is small enough, the Z = CXk
Concepts 1 & 2 will be near 0. Because f (xk ) is decreased by at least δ per iteration, it follows that by choosing k
& 3: Centering sufficiently large, we can ensure that the Z-value for Xk is less than
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 18/44
19. Karmarkar’s Original Algorithm - Two Iterations
Karmarkar’s
Algorithm
AK Dhamija Problem
Min Z = 2x2 - x3
Introduction such that
Complexity
LP Problem • x1 - 2x2 + x3 = 0
Klee-Minty
Example • x1 + x2 + x3 = 1
Comparison • xj ≥ 0, j = 1, 2, 3
Original
Solution : z = 0 at (0, 0.33334, 0.66667)
Algorithm
Steps
Iterations
Transformation Preliminary Step
Affine Variant • k=0
Three Concepts
Example
• X0 = ( n , n , ...., n )T = ( 1 , 1 , 1 )T
1 1 1
3 3 3
Concepts 1 & 2
& 3: Centering
• r= √ 1
n(n−1)
= √ 1
3(3−1)
= √1
6
Iterations
Rescaling • α=
(n−1)
3n
(3−1) 2
= (3)(3) = 9
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 19/44
21. Karmarkar’s Original Algorithm - Two Iterations
Karmarkar’s Iteration 1
Algorithm
• D1 = diag{X1 } = diag{0.2692, 0.3333, 0.3974}
AK Dhamija
• AD1 = (1, −2, 1)diag{X1 } = diag{0.2692, 0.3333, 0.3974} =
(0.2692, −0.6666, 0.3974)
Introduction −0.6666
Complexity • P=
AD1
1
=
0.2692
1 1
0.3974
1
LP Problem
Klee-Minty • CD1 = (0, 2, −1)diag{0.2692, 0.3333, 0.3974} = (0, 0.6666, −0.3974)
Example
Comparison • PPT =
0.675
0
0
3
; (PPT )−1 =
1.482
0
0
0.333
Original
Algorithm 0.441 0.067 0.492
Steps • PT (PPT )−1 P = 0.067 0.992 −0.059
Iterations 0.492 −0.059 0.567
Transformation
• cp = [I − PT (PPT )−1 P](CD1 )T = (0.151, −0.018, −0.132)T
Affine Variant
Three Concepts
• ||cp || = (0.151)2 + (−0.018)2 + (−0.132)2 = 0.2014
Example ( 2 )( √ )
1
Concepts 1 & 2 • c
Ynew = Y0 - αr ||cp || = ( 1 , 3 , 1 )T -
p 3
1
3
9
0.2014
6
(0.151, −0.018, −0.132)T =
& 3: Centering
Iterations (0.2653, 0.3414, 0.3928)T
Rescaling
Final Solution • D1 Ynew = diag{0.2692, 0.3333, 0.3974}(0.2653, 0.3414, 0.3928)T =
Another (0.0714, 0.1138, 0.1561)T ; 1D1 Ynew = 0.3413
Example
Further Issues
• D Y
1 new = (0.2092, 0.3334, 0.4574)T
X2 = 1D Y
1 new
References • Z = CX1 = (0, 2, −1)(0.2092, 0.3334, 0.4574)T = 0.2094; k = 1 + 1 = 2
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 21/44
22. Transforming a LP Problem into Karmarkar’s
Special Form
Karmarkar’s Steps for Conversion of the LP problem : Min Z = CX such that AX ≥ b, X ≥ 0
Algorithm
AK Dhamija
1 Write the dual of given primal problem
Min Z = bW
such that AT W ≤ CT , W ≥ 0
Introduction
Complexity
2 1 Introduce slack & surplus variables to primal and dual problems
LP Problem 2 Combine these two problems.
Klee-Minty
Example 3 1 Introduce a bounding constraint xi + wi ≤ K), where K should be sufficiently large
Comparison to include all feasible solutions of original problem.
Original 2 Introduce a slack variable in the bounding constraint and obtain xi + wi + s = K
Algorithm
Steps
4 1 Introduce a dummy variable d (subject to condition d = 1)to homogenize the constraints.
Iterations 2 Replace the equations
Transformation xi + wi + s = K and d = 1
with the following equations
Affine Variant
xi + wi + s − Kd = 0 and xi + wi + s + d = K + 1
Three Concepts
Example 5 Introduce the following transformation so as to obtain one on the RHS of the last equation
Concepts 1 & 2 xj = (K + 1)yj , j = 1, 2, ...m + n
& 3: Centering wj = (K + 1)ym+n+j , j = 1, 2, ...m + n
Iterations
s = (K + 1)y2m+2n+1
Rescaling
d = (K + 1)y2m+2n+2
Final Solution
Another
Example 6 Introduce an artificial variable y2m+2n+3 (to be minimized) in all the equations such that the sum
of the coefficients in each homogenous equation is zero and coefficient of the artificial variable in the
Further Issues last equation is one.
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 22/44
23. Example of transforming a LP Problem into
Karmarkar’s Special Form
Karmarkar’s Steps for Conversion of the LP problem : Max Z = 2x1 + x2 such that x1 + x2 ≤ 5, x1 − x2 ≤ 3,
Algorithm x1 , x2 are nonnegative
AK Dhamija
1 Write the dual of given primal problem
Min Z = 5w1 + 3w2
Introduction such that
Complexity w1 + w2 ≥ 2
LP Problem w1 − w2 ≥ 1
Klee-Minty w1 , w2 ≥ 0
Example
Comparison 2 Introduction of slack & surplus variables and combination of primal and dual problems
x1 + x2 + x3 = 5
Original x1 − x2 + x4 = 3
Algorithm w1 + w2 − w3 = 2
Steps w1 − w2 − w4 = 1
Iterations 2x1 + x2 = 5w1 + 3w2
Transformation
Affine Variant
3 Addition of boundary constraint with slack variable s
4 4
i=1 xi + i=1 wi + s = K
Three Concepts
Example 4 Homogenized equivalent system with dummy variable d
Concepts 1 & 2
x1 + x2 + x3 − 5d = 0
& 3: Centering
x1 − x2 + x4 − 3d = 0
Iterations
w1 + w2 − w3 − 2d = 0
Rescaling
Final Solution
w1 − w2 − w4 − d = 0
Another 2x1 + x2 − 5w1 − 3w2 = 0
4 4
Example i=1 xi + i=1 wi + s − Kd = 0
4 4
Further Issues i=1 xi + i=1 wi + s + d = (K + 1)
with all variables non-negative
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 23/44
24. Example of transforming a LP Problem into
Karmarkar’s Special Form
Karmarkar’s Steps for Conversion of the LP problem : Max Z = 2x1 + x2 such that x1 + x2 ≤ 5, x1 − x2 ≤ 3,
Algorithm x1 , x2 are nonnegative
AK Dhamija
5 Introduction of transformations
xj = (K + 1)yj , j = 1, 2, ...4
Introduction wj = (K + 1)y4+j , j = 1, 2, ...4
Complexity s = (K + 1)y9
LP Problem d = (K + 1)y10
Klee-Minty The system of equations thus obtained is as follows
Example y1 + y2 + y3 − 5y10 = 0
Comparison y1 − y2 + y4 − 3y10 = 0
y5 + y6 − y7 − 2y10 = 0
Original
y5 − y6 − y8 − y10 = 0
Algorithm
2y1 + y2 − 5y5 − 3y6 = 0
Steps 9
Iterations i=1 yi − Ky10 = 0
10
Transformation i=1 yi = 1
with all variables non-negative
Affine Variant
Three Concepts 6 Introduce an artificial variable y11
Example Minimize y11
Concepts 1 & 2 subject to
& 3: Centering y1 + y2 + y3 − 5y10 + 2y11 = 0
Iterations y1 − y2 + y4 − 3y10 + 2y11 = 0
Rescaling y5 + y6 − y7 − 2y10 + y11 = 0
Final Solution y5 − y6 − y8 − y10 + 2y11 = 0
Another 2y1 + y2 − 5y5 − 3y6 + 5y11 = 0
Example 9
i=1 yi − Ky10 + (K − 9)y11 = 0
Further Issues 11
i=1 yi = 1
with all variables non-negative
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 24/44
25. The Affine Variant of Algorithm
Karmarkar’s
Algorithm
AK Dhamija
Introduction Three Concepts
Complexity
LP Problem • Concept 1: Shoot through the interior of the feasible
Klee-Minty
Example
Comparison
region towards an optimal solution
Original • Concept 2: Move in the direction that improves the
Algorithm
Steps objective function value at the fastest feasible rate.
Iterations
Transformation
• Concept 3: Transform the feasible region to place the
Affine Variant
Three Concepts current trail solution near its center, thereby enabling the
Example
Concepts 1 & 2 fastest feasible rate for Concept 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 25/44
26. An Example
Karmarkar’s The Problem
Algorithm
Max Z = x1 + 2x2
AK Dhamija such that
• x1 + x2 ≤ 8
Introduction
Complexity
• xj ≥ 0, j = 1, 2
LP Problem
The optimal Solution is (x1 , x2 ) = (0, 8) with Z = 16
Klee-Minty
Example
Comparison
Original
Algorithm
Steps
Iterations
Transformation
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 26/44
27. An Example
Karmarkar’s Concept 1: Shoot through the interior of the feasible region toward an optimal solution
Algorithm
• The algorithm begins with an initial solution that lies in the interior of the feasible region
AK Dhamija
• Arbitrarily choose (x1 , x2 ) = (2, 2) to be the initial solution
Introduction
Complexity Concept 2: Move in a direction that improves the objective function value at the fastest feasible rate
LP Problem
Klee-Minty • The direction is perpendiculars to (and toward) the objective function line.
Example
Comparison
• (3, 4) = (2, 2) + (1, 2), where the vector (1, 2) is the gradient( aka Coefficients) of the Objective
Function
Original
Algorithm
Steps
Iterations
Transformation
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 27/44
28. An Example
Karmarkar’s The algorithm actually operates on the augmented form
Algorithm
Max Z = CX such that
AK Dhamija
• AX = b
• X≥0
Introduction
Complexity
LP Problem The Transformation
Klee-Minty
Example ⇒
Max Z = x1 + 2x2 = Max Z = x1 + 2x2 such that
Comparison
• x1 + x2 ≤ 8 = x1 + x2 + x3 = 8, where x3 is the slack
⇒
Original
Algorithm • xj ≥ 0, j = 1, 2 = xj ≥ 0, j = 1, 2, 3
⇒
Steps The optimal Solution is (x1 , x2 ) = (0, 8) with Z = 16
Iterations
Transformation
Affine Variant The Matrices
Three Concepts
x1 0
Example
C= 1 2 0 X = x2 A = 1 1 1 b= 8 0= 0
Concepts 1 & 2
x3 0
& 3: Centering
Iterations
Rescaling
Final Solution The Solution
Another
Example • ⇒ ⇒
Initial Solution (2, 2) = (2, 2, 4), Optimum (0, 8) = (0, 8, 0)
Further Issues • ⇒
Gradient of Objective Function (1, 2) = (1, 2, 0)
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 28/44
29. An Interior-point Algorithm
Gradient of Objective function : C = 1 2 0
Karmarkar’s
Algorithm
Using Projected Gradient to implement Concept 1 & 2
AK Dhamija • ⇒
Adding the gradient to the initial leads to (3, 4, 4) = (2, 2, 4) + (1, 2, 0) = Infeasible
• To remain feasible, the algorithm projects the point (3, 4, 4) down onto the feasible tetrahedron
Introduction
Complexity
• The next trial solution moves in the direction of projected gradient i.e. the gradient projected onto
the feasible region
LP Problem
Klee-Minty
Example
Comparison
Original
Algorithm
Steps
Iterations
Transformation
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 29/44
30. An Interior-point Algorithm
Karmarkar’s Using Projected Gradient to implement Concept 1 & 2
Algorithm
2 −1 −3 1
3 3
AK Dhamija • Projection Matrix P = I − AT (AAT )−1 A = − 1
3
2
3
1
−3
−13
−13
2
3
Introduction
0
Complexity
LP Problem
• T
Projected Gradient cP = PC = 1
Klee-Minty −1
Example
Comparison 2 2 0
• Move (2, 2, 4) towards cP to X = 2 + 4αcP = 2 + 4α 1
Original 4 4 −1
Algorithm
Steps
• α determines how far we move. large α ⇒ too close to the boundary and α ⇒ more iterations we
Iterations
have chosen α = 0.5, so the new trial solution move to: X = (2, 3, 3)
Transformation
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 30/44
31. An Interior-point Algorithm
Karmarkar’s
Algorithm
Concept 3: Transform the feasible region to place the current trail
AK Dhamija
solution near its center, thereby enabling a large improvement when
Introduction concept 2 is implemented
Complexity
LP Problem
Klee-Minty
Centering scheme for implementing Concept 3
Example
Comparison
• Why : The centering scheme keeps turning the direction of the
Original
Algorithm projected gradient to point more nearly toward an optimal
Steps solution as the algorithm converges toward this solution
Iterations
Transformation
• How : Simply changing the scale for each of the variable so that
Affine Variant
Three Concepts the trail solution becomes equidistant from the constraint
Example
Concepts 1 & 2 boundaries in the new coordinate system
& 3: Centering
Iterations ˜
• x is brought to the center X = (1, 1, 1) in the new coordinate
Rescaling
Final Solution system by X˜ = D−1 X, where D = diag{X}
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 31/44
32. An Interior-point Algorithm
Karmarkar’s Centering scheme for implementing Concept 3
Algorithm
1 0 0
1 0 0
x1 2 1
AK Dhamija 2 2
X = D−1 X = 0
˜ 1 0 x2 = 0 1 0 2 = 1
2 2
0 0 1 x3 0 0 1 4 1
4 4
Introduction
Complexity
LP Problem The Problem in the new coordinate system becomes
Klee-Minty
Example Max Z = 2˜1 + 4˜2
x x
Comparison such that
Original
Algorithm
• 2˜1 + 2˜2 + 4˜3 = 8
x x x
Steps • xj ≥ 0, j = 1, 2, 3
˜
Iterations
Transformation
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 32/44
34. Complete Illustration of the Algorithm
Karmarkar’s Iteration 1
Algorithm
• Identify v (how far to move) as the absolute value of the negative component of cP having the
AK Dhamija largest value, so that v = | − 2| = 2
1
Introduction • In this coordinate system the algorithm moves from current solution to x = 1 + α cP =
˜ v
Complexity 1
LP Problem 5
1 1 4
Klee-Minty 1 + 0.5 3 = 7
Example 2 4
1 −2 1
Comparison 2
Original
Algorithm
Steps
Iterations
Transformation
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 34/44
35. Complete Illustration of the Algorithm
Karmarkar’s
Algorithm Iteration 1
AK Dhamija x1 2 0 0
• ˜
In the original coordinate system the solution is X = x2 DX = 0 2 0
x3 0 0 4
Introduction
5 5
4 2
Complexity 7
4 = 7 2
LP Problem 1
Klee-Minty 2
2
Example
Comparison
Original
Algorithm
Steps
Iterations
Transformation
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
• This completes the iteration 1. The new solution will be used to start the next iteration
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 35/44
36. Complete Illustration of the Algorithm
Karmarkar’s
Algorithm Iteration 2
AK Dhamija • Current trial Solution (x1 , x2 , x3 ) = ( 5 , 7 , 2)
2 2
5
2
0 0 1
Introduction • D = diag{x1 , x2 , x3 } = 0 7
2
0 X=D ˜ −1
X= 1
Complexity 0 0 2 1
LP Problem
13 − 187 −9 2 − 11
Klee-Minty 18 12
Example • P = − 18
7 41
90
− 14 cP = 133
45
60
Comparison 2
−9 − 14
45
37
45
− 41
15
Original
Algorithm
• The current solution becomes (0.83, 1.4, 0.5) which corresponds (2.08, 4.92, 1.0) in the original
coordinate system
Steps
Iterations
Transformation
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 36/44
37. Complete Illustration of the Algorithm
Karmarkar’s
Algorithm Iteration 3
AK Dhamija • Current trial Solution (x1 , x2 , x3 ) = (2.08, 4.92, 1.0)
2.08 0 0
Introduction • D = diag{x1 , x2 , x3 } = 0 4.92 0
Complexity 0 0 1.0
LP Problem
−1.63
Klee-Minty
Example • cP = 1.05
Comparison −1.79
Original • The current solution becomes (0.54, 1.3, 0.5) which corresponds (1.13, 6.37, 0.5) in the original
Algorithm coordinate system
Steps
Iterations
Transformation
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 37/44
38. Complete Illustration of the Algorithm
Karmarkar’s Effect of rescaling of each iteration
Algorithm
Sliding the optimal solution toward (1, 1, 1) while the other BF solutions tend to slide away
AK Dhamija
Introduction
Complexity
LP Problem
Klee-Minty
Example
Comparison
Original
Algorithm
Steps
Iterations
Transformation (1) (2)
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
(3) (4)
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 38/44
39. Complete Illustration of the Algorithm
Karmarkar’s
Algorithm
More iterations
AK Dhamija
Starting from the current trial solution x, following the steps, x is moving toward the optimum (0, 8).
When the trial solution is virtually unchanged from the proceeding one, then the algorithm has virtually
Introduction converged to an optimal solution. We stop.
Complexity
LP Problem
Klee-Minty
Example
Comparison
Original
Algorithm
Steps
Iterations
Transformation
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 39/44
40. Another Example
Karmarkar’s
Algorithm
The Problem
AK Dhamija Max Z = 5x1 + 4x2
such that
Introduction
Complexity
• 6x1 + 4x2 ≤ 24
LP Problem • x1 + 2x2 ≤ 6
Klee-Minty
Example
• −x1 + x2 ≤ 1
Comparison • x2 ≤ 2
Original • xj ≥ 0, j = 1, 2
Algorithm
Steps
Iterations
Augmented Form
Transformation
Affine Variant Max Z = 5x1 + 4x2
such that
Three Concepts
Example
Concepts 1 & 2
• 6x1 + 4x2 + x3 = 24
& 3: Centering • x1 + 2x2 + x4 = 6
Iterations
Rescaling
• −x1 + x2 + x5 = 1
Final Solution • x2 + x6 = 2
Another
Example • xj ≥ 0, j = 1, 2, 3, 4, 5, 6
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 40/44
41. Another Example
Karmarkar’s
The Matrices
Algorithm
6 4 1 0 0 0 24
AK Dhamija 1 2 0 1 0 0 6
A= −1 1 C=
b= 5 4 0 0 0 0
1 0 0 1 0
Introduction 0 1 0 0 0 1 2
Complexity
LP Problem
Klee-Minty
Example
Comparison
Original
Algorithm
Steps
Iterations
Transformation
Affine Variant
Three Concepts
Example
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 41/44
42. Another Example
Karmarkar’s
Algorithm
Starting from an initial solution X = (1, 1, 14, 3, 1, 1)
AK Dhamija
Introduction
The Steps
Complexity
LP Problem
D = diag{X}
Klee-Minty
Example ˜
A = AD
Comparison
Original ˜
c = DC
Algorithm
Steps ˜ ˜˜ ˜
P = I − AT (AAT )−1 A
Iterations
Transformation
cP = P˜
c
Affine Variant
Three Concepts
Example
1
˜ = 1 +
X α
v cP
Concepts 1 & 2
& 3: Centering
Iterations
Rescaling
1
Final Solution
Another X = DX˜
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 42/44
43. Further Issues
Karmarkar’s
Algorithm
AK Dhamija
Interior-point methods is designed for dealing with big problems. Although the claim that it’s much faster
than the simplex method is controversy, many tests on huge LP problems show its outperformance
Introduction
Complexity
LP Problem Future Research
Klee-Minty
Example • Infeasible interior points method - remove the assumption that there always exits a nonempty interior
Comparison • Methods applying to LP problems in standard form
Original • Methods dealing with finding initial solution, and estimating the optimal solution
Algorithm
Steps
• Methods working with primal-dual problems
Iterations • Studies about moving step-long/short steps
Transformation
• Studies about efficient implementation and complexity of various methods
Affine Variant
Three Concepts
Example Karmarkar’s paper not only started the development of interior point methods, but also encouraged rapid
Concepts 1 & 2
& 3: Centering improvement of simplex methods
Iterations
Rescaling
Final Solution
Another
Example
Further Issues
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 43/44
44. References
Karmarkar’s References
Algorithm
• N. Karmarkar, A New Polynomial - Time Algorithm for Linear Programming, Combinatorica 4 (4),
AK Dhamija 1984, p. 373-395
• Yanrong Hu & Ce Yu, Paper review of ENGG*6140 Optimization Techniques – Interior-Point
Introduction Methods for LP, 2003
Complexity • I. Lustig, R. Marsten, D. Shanno, Interior-point Methods for Linear Programming: Computational
LP Problem State of the Art, ORSA Journal on Computing, 6 (1), 1994, p. 1-14
Klee-Minty
Example • Hillier,Lieberman,Introduction to Operations Research (7th edition) 320-334
Comparison
• Taha, Operations Research: An Introduction (6th edition) 336-345
Original • D. Gay, A Variant of Karmarkar’s Linear Programming Algorithm for Problems in Standard form,
Algorithm Mathematical Programming 37 (1987) 81-90
Steps
Iterations
• E.R. Barnes, A Variation on Karmarkar’s Algorithm for Solving Linear Programming problems,
Transformation Mathematical Programming 36, 1986, p. 174-182
Affine Variant
• V. Klee and G.J. Minty, How Good is the Simplex Algorithm? In O. Shisha, editor, Inequalities, III,
pages 159-175. Academic Press, New York, NY, 1972.
Three Concepts
Example • Richard Bronson and Govindasamy Naadimuthu, Schaum’s Outlines, Operations Research, Second
Concepts 1 & 2 Edition, Mcgraw Hill.
& 3: Centering
Iterations
• Wayne L Winston, Operations Research, Applications and Algorithms, Fourth Edition, Thomson
Rescaling
Brooks/Cole, 2007
Final Solution
Another
Example Presentation available at
Further Issues www.geocities.com/a_k_dhamija.
References
AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 44/44