1) The document discusses sparse methods for machine learning, focusing on sparse linear estimation using the l1-norm and extensions to structured sparse methods on vectors and matrices.
2) It reviews theory and algorithms for l1-norm regularization, including optimality conditions, first-order methods like subgradient descent, and coordinate descent.
3) The document also discusses going beyond lasso to structured sparsity, non-linear kernels, and sparse methods for matrices applied to problems like multi-task learning.
NIPS2009: Sparse Methods for Machine Learning: Theory and Algorithms
1. Sparse methods for machine learning
Theory and algorithms
Francis Bach
Willow project, INRIA - Ecole Normale Sup´rieure
e
NIPS Tutorial - December 2009
Special thanks to R. Jenatton, J. Mairal, G. Obozinski
2. Supervised learning and regularization
⢠Data: xi â X , yi â Y, i = 1, . . . , n
⢠Minimize with respect to function f : X â Y:
n
Îť
â(yi, f (xi)) + f 2
i=1
2
Error on data + Regularization
Loss & function space ? Norm ?
⢠Two theoretical/algorithmic issues:
1. Loss
2. Function space / norm
3. Regularizations
⢠Main goal: avoid overďŹtting
⢠Two main lines of work:
1. Euclidean and Hilbertian norms (i.e., â2-norms)
â Possibility of non linear predictors
â Non parametric supervised learning and kernel methods
â Well developped theory and algorithms (see, e.g., Wahba, 1990;
Sch¨lkopf and Smola, 2001; Shawe-Taylor and Cristianini, 2004)
o
4. Regularizations
⢠Main goal: avoid overďŹtting
⢠Two main lines of work:
1. Euclidean and Hilbertian norms (i.e., â2-norms)
â Possibility of non linear predictors
â Non parametric supervised learning and kernel methods
â Well developped theory and algorithms (see, e.g., Wahba, 1990;
Sch¨lkopf and Smola, 2001; Shawe-Taylor and Cristianini, 2004)
o
2. Sparsity-inducing norms
â Usually restricted to linear predictors on vectors f (x) = wâ¤x
p
â Main example: â1-norm w 1 = i=1 |wi|
â Perform model selection as well as regularization
â Theory and algorithms âin the makingâ
5. â2 vs. â1 - Gaussian hare vs. Laplacian tortoise
⢠First-order methods (Fu, 1998; Wu and Lange, 2008)
⢠Homotopy methods (Markowitz, 1956; Efron et al., 2004)
6. Lasso - Two main recent theoretical results
1. Support recovery condition (Zhao and Yu, 2006; Wainwright, 2009;
Zou, 2006; Yuan and Lin, 2007): the Lasso is sign-consistent if and
only if
QJcJQâ1sign(wJ) â 1,
JJ
1 n
where Q = limnâ+â n i=1 xixâ¤
i â RpĂp
7. Lasso - Two main recent theoretical results
1. Support recovery condition (Zhao and Yu, 2006; Wainwright, 2009;
Zou, 2006; Yuan and Lin, 2007): the Lasso is sign-consistent if and
only if
QJcJQâ1sign(wJ) â 1,
JJ
1 n
where Q = limnâ+â n i=1 xixâ¤
i â RpĂp
2. Exponentially many irrelevant variables (Zhao and Yu, 2006;
Wainwright, 2009; Bickel et al., 2009; Lounici, 2008; Meinshausen
and Yu, 2008): under appropriate assumptions, consistency is possible
as long as
log p = O(n)
8. Going beyond the Lasso
⢠â1-norm for linear feature selection in high dimensions
â Lasso usually not applicable directly
⢠Non-linearities
⢠Dealing with exponentially many features
⢠Sparse learning on matrices
9. Going beyond the Lasso
Non-linearity - Multiple kernel learning
⢠Multiple kernel learning
p
â Learn sparse combination of matrices k(x, xâ˛) = j=1 Ρj kj (x, xâ˛)
â Mixing positive aspects of â1-norms and â2-norms
⢠Equivalent to group Lasso
â p multi-dimensional features ÎŚj (x), where
kj (x, xâ˛) = ÎŚj (x)â¤ÎŚj (xâ˛)
p â¤
â learn predictor j=1 wj ÎŚj (x)
p
â Penalization by j=1 wj 2
10. Going beyond the Lasso
Structured set of features
⢠Dealing with exponentially many features
â Can we design eďŹcient algorithms for the case log p â n?
â Use structure to reduce the number of allowed patterns of zeros
â Recursivity, hierarchies and factorization
⢠Prior information on sparsity patterns
â Grouped variables with overlapping groups
11. Going beyond the Lasso
Sparse methods on matrices
⢠Learning problems on matrices
â Multi-task learning
â Multi-category classiďŹcation
â Matrix completion
â Image denoising
â NMF, topic models, etc.
⢠Matrix factorization
â Two types of sparsity (low-rank or dictionary learning)
12. Sparse methods for machine learning
Outline
⢠Introduction - Overview
⢠Sparse linear estimation with the â1-norm
â Convex optimization and algorithms
â Theoretical results
⢠Structured sparse methods on vectors
â Groups of features / Multiple kernel learning
â Extensions (hierarchical or overlapping groups)
⢠Sparse methods on matrices
â Multi-task learning
â Matrix factorization (low-rank, sparse PCA, dictionary learning)
13. Why â1-norm constraints leads to sparsity?
⢠Example: minimize quadratic function Q(w) subject to w 1 T.
â coupled soft thresholding
⢠Geometric interpretation
â NB : penalizing is âequivalentâ to constraining
w2 w2
w1 w1
14. â1-norm regularization (linear setting)
⢠Data: covariates xi â Rp, responses yi â Y, i = 1, . . . , n
⢠Minimize with respect to loadings/weights w â Rp:
n
J(w) = â(yi, wâ¤xi) + Îť w 1
i=1
Error on data + Regularization
⢠Including a constant term b? Penalizing or constraining?
⢠square loss â basis pursuit in signal processing (Chen et al., 2001),
Lasso in statistics/machine learning (Tibshirani, 1996)
15. A review of nonsmooth convex
analysis and optimization
⢠Analysis: optimality conditions
⢠Optimization: algorithms
â First-order methods
⢠Books: Boyd and Vandenberghe (2004), Bonnans et al. (2003),
Bertsekas (1995), Borwein and Lewis (2000)
16. Optimality conditions for smooth optimization
Zero gradient
n
⤠Ν 2
⢠Example: â2-regularization: minp â(yi, w xi) + w 2
wâR
i=1
2
n
â Gradient âJ(w) = i=1 ââ˛(yi, wâ¤xi)xi + Îťw where ââ˛(yi, wâ¤xi)
is the partial derivative of the loss w.r.t the second variable
n
â If square loss, i=1 â(yi, wâ¤xi) = 1 y â Xw 2
2 2
â gradient = âX â¤(y â Xw) + Îťw
â normal equations â w = (X â¤X + ÎťI)â1X â¤y
17. Optimality conditions for smooth optimization
Zero gradient
n
⤠Ν 2
⢠Example: â2-regularization: minp â(yi, w xi) + w 2
wâR
i=1
2
n
â Gradient âJ(w) = i=1 ââ˛(yi, wâ¤xi)xi + Îťw where ââ˛(yi, wâ¤xi)
is the partial derivative of the loss w.r.t the second variable
n
â If square loss, i=1 â(yi, wâ¤xi) = 1 y â Xw 2
2 2
â gradient = âX â¤(y â Xw) + Îťw
â normal equations â w = (X â¤X + ÎťI)â1X â¤y
⢠â1-norm is non diďŹerentiable!
â cannot compute the gradient of the absolute value
â Directional derivatives (or subgradient)
18. Directional derivatives - convex functions on Rp
⢠Directional derivative in the direction â at w:
J(w + Îľâ) â J(w)
âJ(w, â) = lim
Îľâ0+ Îľ
⢠Always exist when J is convex and continuous
⢠Main idea: in non smooth situations, may need to look at all
directions â and not simply p independent ones
⢠Proposition: J is diďŹerentiable at w, if and only if â â âJ(w, â)
is linear. Then, âJ(w, â) = âJ(w)â¤â
19. Optimality conditions for convex functions
⢠Unconstrained minimization (function deďŹned on Rp):
â Proposition: w is optimal if and only if ââ â Rp, âJ(w, â) 0
â Go up locally in all directions
⢠Reduces to zero-gradient for smooth problems
⢠Constrained minimization (function deďŹned on a convex set K)
â restrict â to directions so that w + Îľâ â K for small Îľ
21. Optimality conditions for â1-norm regularization
⢠General loss: w optimal if and only if for all j â {1, . . . , p},
sign(wj ) = 0 â âL(w)j + Îť sign(wj ) = 0
sign(wj ) = 0 â |âL(w)j | Îť
⢠Square loss: w optimal if and only if for all j â {1, . . . , p},
â¤
sign(wj ) = 0 â âXj (y â Xw) + Îť sign(wj ) = 0
â¤
sign(wj ) = 0 â |Xj (y â Xw)| Îť
â For J â {1, . . . , p}, XJ â RnĂ|J| = X(:, J) denotes the columns
of X indexed by J, i.e., variables indexed by J
22. First order methods for convex optimization on Rp
Smooth optimization
⢠Gradient descent: wt+1 = wt â ÎątâJ(wt)
â with line search: search for a decent (not necessarily best) Îąt
â ďŹxed diminishing step size, e.g., Îąt = a(t + b)â1
⢠Convergence of f (wt) to f â = minwâRp f (w) (Nesterov, 2003)
â
â
â f convex and M -Lipschitz: f (wt)âf = O M/ t
â and, diďŹerentiable with L-Lipschitz gradient: f (wt)âf â = O L/t
Âľ
â and, f Âľ-strongly convex: f (wt) â f â = O L exp(â4t L )
Âľ
⢠L = condition number of the optimization problem
⢠Coordinate descent: similar properties
⢠NB: âoptimal schemeâ f (wt)âf â = O L min{exp(â4t Âľ/L), tâ2}
23. First-order methods for convex optimization on Rp
Non smooth optimization
⢠First-order methods for non diďŹerentiable objective
â Subgradient descent: wt+1 = wt â Îątgt, with gt â âJ(wt), i.e.,
â¤
such that ââ, gt â âJ(wt, â)
â with exact line search: not always convergent (see counter-
example)
â diminishing step size, e.g., Îąt = a(t + b)â1: convergent
â Coordinate descent: not always convergent (show counter-example)
M
⢠Convergence rates (f convex and M -Lipschitz): f (wt)âf â = O â
t
25. Counter-example (Bertsekas, 1995)
Steepest descent for nonsmooth objectives
â5(9x2 + 16x2)1/2 if x1 > |x2|
1 2
⢠q(x1, x2) =
â(9x1 + 16|x2|)1/2 if x1 |x2|
⢠Steepest descent starting from any x such that x1 > |x2| >
(9/16)2|x1|
5
0
â5
â5 0 5
26. Sparsity-inducing norms
Using the structure of the problem
⢠Problems of the form minp L(w) + Ν w or min L(w)
wâR w Âľ
â L smooth
â Orthogonal projections on the ball or the dual ball can be performed
in semi-closed form, e.g., â1-norm (Maculan and GALDINO
DE PAULA, 1989) or mixed â1-â2 (see, e.g., van den Berg et al.,
2009)
⢠May use similar techniques than smooth optimization
â Projected gradient descent
â Proximal methods (Beck and Teboulle, 2009)
â Dual ascent methods
⢠Similar convergence rates
28. Cheap (and not dirty) algorithms for all losses
⢠Coordinate descent (Fu, 1998; Wu and Lange, 2008; Friedman
et al., 2007)
â convergent here under reasonable assumptions! (Bertsekas, 1995)
â separability of optimality conditions
â equivalent to iterative thresholding
29. Cheap (and not dirty) algorithms for all losses
⢠Coordinate descent (Fu, 1998; Wu and Lange, 2008; Friedman
et al., 2007)
â convergent here under reasonable assumptions! (Bertsekas, 1995)
â separability of optimality conditions
â equivalent to iterative thresholding
⢠âΡ-trickâ (Micchelli and Pontil, 2006; Rakotomamonjy et al., 2008;
Jenatton et al., 2009b)
2
p p wj
â Notice that j=1 |wj |
= minΡ 0 1
2 j=1 + Ρj
Ρj
â Alternating minimization with respect to Ρ (closed-form) and w
(weighted squared â2-norm regularized problem)
30. Cheap (and not dirty) algorithms for all losses
⢠Coordinate descent (Fu, 1998; Wu and Lange, 2008; Friedman
et al., 2007)
â convergent here under reasonable assumptions! (Bertsekas, 1995)
â separability of optimality conditions
â equivalent to iterative thresholding
⢠âΡ-trickâ (Micchelli and Pontil, 2006; Rakotomamonjy et al., 2008;
Jenatton et al., 2009b)
2
p p wj
â Notice that j=1 |wj |
= minΡ 0 1
2 j=1 + Ρj
Ρj
â Alternating minimization with respect to Ρ (closed-form) and w
(weighted squared â2-norm regularized problem)
⢠Dedicated algorithms that use sparsity (active sets and homotopy
methods)
31. Special case of square loss
⢠Quadratic programming formulation: minimize
p
1 + â
yâXw 2+Îť (wj +wj ) such that w = w+âwâ, w+ 0, wâ 0
2 j=1
32. Special case of square loss
⢠Quadratic programming formulation: minimize
p
1 + â
yâXw 2+Îť (wj +wj ) such that w = w+âwâ, w+ 0, wâ 0
2 j=1
â generic toolboxes â very slow
⢠Main property: if the sign pattern s â {â1, 0, 1}p of the solution is
known, the solution can be obtained in closed form
â Lasso equivalent to minimizing 1 y â XJ wJ 2 + Îťsâ¤wJ w.r.t. wJ
2 J
where J = {j, sj = 0}.
â Closed form solution wJ = (XJ XJ )â1(XJ y â ÎťsJ )
⤠â¤
⢠Algorithm: âGuessâ s and check optimality conditions
33. Optimality conditions for the sign vector s (Lasso)
⢠For s â {â1, 0, 1}p sign vector, J = {j, sj = 0} the nonzero pattern
⢠potential closed form solution: wJ = (XJ XJ )â1(XJ y â ÎťsJ ) and
⤠â¤
wJ c = 0
⢠s is optimal if and only if
â active variables: sign(wJ ) = sJ
â¤
â inactive variables: XJ c (y â XJ wJ ) â Îť
⢠Active set algorithms (Lee et al., 2007; Roth and Fischer, 2008)
â Construct J iteratively by adding variables to the active set
â Only requires to invert small linear systems
34. Homotopy methods for the square loss (Markowitz,
1956; Osborne et al., 2000; Efron et al., 2004)
⢠Goal: Get all solutions for all possible values of the regularization
parameter Îť
⢠Same idea as before: if the sign vector is known,
wJ (Îť) = (XJ XJ )â1(XJ y â ÎťsJ )
â ⤠â¤
valid, as long as,
â
â sign condition: sign(wJ (Îť)) = sJ
⤠â
â subgradient condition: XJ c (XJ wJ (Îť) â y) â Îť
â this deďŹnes an interval on Îť: the path is thus piecewise aďŹne
⢠Simply need to ďŹnd break points and directions
36. Algorithms for â1-norms (square loss):
Gaussian hare vs. Laplacian tortoise
⢠Coordinate descent: O(pn) per iterations for â1 and â2
⢠âExactâ algorithms: O(kpn) for â1 vs. O(p2n) for â2
37. Additional methods - Softwares
⢠Many contributions in signal processing, optimization, machine
learning
â Proximal methods (Nesterov, 2007; Beck and Teboulle, 2009)
â Extensions to stochastic setting (Bottou and Bousquet, 2008)
⢠Extensions to other sparsity-inducing norms
⢠Softwares
â Many available codes
â SPAMS (SPArse Modeling Software) - note diďŹerence with
SpAM (Ravikumar et al., 2008)
http://www.di.ens.fr/willow/SPAMS/
38. Sparse methods for machine learning
Outline
⢠Introduction - Overview
⢠Sparse linear estimation with the â1-norm
â Convex optimization and algorithms
â Theoretical results
⢠Structured sparse methods on vectors
â Groups of features / Multiple kernel learning
â Extensions (hierarchical or overlapping groups)
⢠Sparse methods on matrices
â Multi-task learning
â Matrix factorization (low-rank, sparse PCA, dictionary learning)
39. Theoretical results - Square loss
⢠Main assumption: data generated from a certain sparse w
⢠Three main problems:
1. Regular consistency: convergence of estimator w to w, i.e.,
Ë
w â w tends to zero when n tends to â
Ë
2. Model selection consistency: convergence of the sparsity pattern
of w to the pattern w
Ë
3. EďŹciency: convergence of predictions with w to the predictions
Ë
1
with w, i.e., n X w â Xw 2 tends to zero
Ë 2
⢠Main results:
â Condition for model consistency (support recovery)
â High-dimensional inference
40. Model selection consistency (Lasso)
⢠Assume w sparse and denote J = {j, wj = 0} the nonzero pattern
⢠Support recovery condition (Zhao and Yu, 2006; Wainwright, 2009;
Zou, 2006; Yuan and Lin, 2007): the Lasso is sign-consistent if and
only if
QJcJQâ1sign(wJ) â 1
JJ
1 n
where Q = limnâ+â n i=1 xixâ¤
i â RpĂp (covariance matrix)
41. Model selection consistency (Lasso)
⢠Assume w sparse and denote J = {j, wj = 0} the nonzero pattern
⢠Support recovery condition (Zhao and Yu, 2006; Wainwright, 2009;
Zou, 2006; Yuan and Lin, 2007): the Lasso is sign-consistent if and
only if
QJcJQâ1sign(wJ) â 1
JJ
1 n
where Q = limnâ+â n i=1 xixâ¤
i â RpĂp (covariance matrix)
⢠Condition depends on w and J (may be relaxed)
â may be relaxed by maximizing out sign(w) or J
⢠Valid in low and high-dimensional settings
⢠Requires lower-bound on magnitude of nonzero wj
42. Model selection consistency (Lasso)
⢠Assume w sparse and denote J = {j, wj = 0} the nonzero pattern
⢠Support recovery condition (Zhao and Yu, 2006; Wainwright, 2009;
Zou, 2006; Yuan and Lin, 2007): the Lasso is sign-consistent if and
only if
QJcJQâ1sign(wJ) â 1
JJ
1 n
where Q = limnâ+â n i=1 xixâ¤
i â RpĂp (covariance matrix)
⢠The Lasso is usually not model-consistent
â Selects more variables than necessary (see, e.g., Lv and Fan, 2009)
â Fixing the Lasso: adaptive Lasso (Zou, 2006), relaxed
Lasso (Meinshausen, 2008), thresholding (Lounici, 2008),
Bolasso (Bach, 2008a), stability selection (Meinshausen and
B¨hlmann, 2008), Wasserman and Roeder (2009)
u
43. Adaptive Lasso and concave penalization
⢠Adaptive Lasso (Zou, 2006; Huang et al., 2008)
p
|wj |
â Weighted â1-norm: minp L(w) + Îť
wâR
j=1
|wj |Îą
Ë
â w estimator obtained from â2 or â1 regularization
Ë
⢠Reformulation in terms of concave penalization
p
minp L(w) + g(|wj |)
wâR
j=1
â Example: g(|wj |) = |wj |1/2 or log |wj |. Closer to the â0 penalty
â Concave-convex procedure: replace g(|wj |) by aďŹne upper bound
â Better sparsity-inducing properties (Fan and Li, 2001; Zou and Li,
2008; Zhang, 2008b)
44. Bolasso (Bach, 2008a)
â
⢠Property: for a speciďŹc choice of regularization parameter Îť â n:
â all variables in J are always selected with high probability
â all other ones selected with probability in (0, 1)
⢠Use the bootstrap to simulate several replications
â Intersecting supports of variables
â Final estimation of w on the entire dataset
Bootstrap 1 J1
Bootstrap 2 J2
Bootstrap 3 J3
Bootstrap 4 J4
Bootstrap 5 J5
Intersection
45. Model selection consistency of the Lasso/Bolasso
⢠probabilities of selection of each variable vs. regularization param. ¾
variable index
variable index
5 5
LASSO 10 10
15 15
0 5 10 15 0 5 10 15
âlog(Âľ) âlog(Âľ)
variable index
variable index
5 5
BOLASSO 10 10
15 15
0 5 10 15 0 5 10 15
âlog(Âľ) âlog(Âľ)
Support recovery condition satisďŹed not satisďŹed
46. High-dimensional inference
Going beyond exact support recovery
⢠Theoretical results usually assume that non-zero wj are large enough,
log p
i.e., |wj | Ď n
⢠May include too many variables but still predict well
⢠Oracle inequalities
â Predict as well as the estimator obtained with the knowledge of J
â Assume i.i.d. Gaussian noise with variance Ď 2
â We have:
1 Ď 2|J|
E X woracle â Xw 2 =
Ë 2
n n
47. High-dimensional inference
Variable selection without computational limits
⢠Approaches based on penalized criteria (close to BIC)
p
min min yâ XJ wJ 2
2
2
+ CĎ |J| 1 + log
Jâ{1,...,p} wJ âR|J| |J|
⢠Oracle inequality if data generated by w with k non-zeros (Massart,
2003; Bunea et al., 2007):
1 2 kĎ 2 p
X w â Xw
Ë 2 C 1 + log
n n k
⢠Gaussian noise - No assumptions regarding correlations
k log p
⢠Scaling between dimensions: n small
⢠Optimal in the minimax sense
48. High-dimensional inference
Variable selection with orthogonal design
1
⢠Orthogonal design: assume that n X â¤X = I
1
⢠Lasso is equivalent to soft-thresholding n X â¤Y â Rp
1 ⤠1 ⤠Ν
â Solution: wj = soft-thresholding of
Ë n Xj y = wj + n Xj Îľ at
n
(|t|âa)+ sign(t)
1 2
min w â wt + a|w|
âa wâR 2
a t Solution w = (|t| â a)+ sign(t)
49. High-dimensional inference
Variable selection with orthogonal design
1
⢠Orthogonal design: assume that n X â¤X = I
1
⢠Lasso is equivalent to soft-thresholding n X â¤Y â Rp
1 ⤠1 ⤠Ν
â Solution: wj = soft-thresholding of
Ë n Xj y = wj + n Xj Îľ at
â n
â Take Îť = AĎ n log p
⢠Where does the log p = O(n) come from?
â
â Expectation of the maximum of p Gaussian variables â log p
â Union-bound:
P(âj â Jc, |Xj Îľ|
â¤
Îť) jâJc
â¤
P(|Xj Îľ| Îť)
2 2 2
â Îť 2 â A log p 1â A
|Jc|e 2nĎ pe 2 =p 2
50. High-dimensional inference (Lasso)
⢠Main result: we only need k log p = O(n)
â if w is suďŹciently sparse
â and input variables are not too correlated
1
⢠Precise conditions on covariance matrix Q = n X â¤X.
â Mutual incoherence (Lounici, 2008)
â Restricted eigenvalue conditions (Bickel et al., 2009)
â Sparse eigenvalues (Meinshausen and Yu, 2008)
â Null space property (Donoho and Tanner, 2005)
⢠Links with signal processing and compressed sensing (Cand`s and
e
Wakin, 2008)
⢠Assume that Q has unit diagonal
51. Mutual incoherence (uniform low correlations)
⢠Theorem (Lounici, 2008):
â yi = wâ¤xi + Îľi, Îľ i.i.d. normal with mean zero and variance Ď 2
⤠1
â Q = X X/n with unit diagonal and cross-terms less than
â 14k
2
â if w 0 k, and A > 8, then, with Îť = AĎ n log p
1/2
log p 1âA2 /8
P wâw
Ë â 5AĎ 1âp
n
log p
⢠Model consistency by thresholding if min |wj | > CĎ
j,wj =0 n
⢠Mutual incoherence condition depends strongly on k
⢠Improved result by averaging over sparsity patterns (Cand`s and Plan,
e
2009b)
52. Restricted eigenvalue conditions
⢠Theorem (Bickel et al., 2009):
2 ââ¤Qâ
â assume Îş(k) = min min 2 >0
|J| k â, âJ c 1 âJ 1 âJ 2
â
â assume Îť = AĎ n log p and A2 > 8
1âA2/8
â then, with probability 1 â p , we have
16A log p
estimation error wâw
Ë 1 2(k)
Ďk
Îş n
1 2 16A2 Ď 2k
prediction error X w â Xw
Ë 2 2(k) n
log p
n Îş
⢠Condition imposes a potentially hidden scaling between (n, p, k)
⢠Condition always satisďŹed for Q = I
53. Checking suďŹcient conditions
⢠Most of the conditions are not computable in polynomial time
⢠Random matrices
â Sample X â RnĂp from the Gaussian ensemble
â Conditions satisďŹed with high probability for certain (n, p, k)
â Example from Wainwright (2009): n Ck log p
⢠Checking with convex optimization
â Relax conditions to convex optimization problems (dâAspremont
et al., 2008; Juditsky and Nemirovski, 2008; dâAspremont and
El Ghaoui, 2008)
â Example: sparse eigenvalues min|J| k Îťmin(QJJ )
â Open problem: veriďŹable assumptions still lead to weaker results
54. Sparse methods
Common extensions
⢠Removing bias of the estimator
â Keep the active set, and perform unregularized restricted
estimation (Cand`s and Tao, 2007)
e
â Better theoretical bounds
â Potential problems of robustness
⢠Elastic net (Zou and Hastie, 2005)
â Replace Îť w 1 by Îť w 1 + Îľ w 2 2
â Make the optimization strongly convex with unique solution
â Better behavior with heavily correlated variables
55. Relevance of theoretical results
⢠Most results only for the square loss
â Extend to other losses (Van De Geer, 2008; Bach, 2009b)
⢠Most results only for â1-regularization
â May be extended to other norms (see, e.g., Huang and Zhang,
2009; Bach, 2008b)
⢠Condition on correlations
â very restrictive, far from results for BIC penalty
⢠Non sparse generating vector
â little work on robustness to lack of sparsity
⢠Estimation of regularization parameter
â No satisfactory solution â open problem
56. Alternative sparse methods
Greedy methods
⢠Forward selection
⢠Forward-backward selection
⢠Non-convex method
â Harder to analyze
â Simpler to implement
â Problems of stability
⢠Positive theoretical results (Zhang, 2009, 2008a)
â Similar suďŹcient conditions than for the Lasso
57. Alternative sparse methods
Bayesian methods
n
⢠Lasso: minimize i=1 (yi â wâ¤xi)2 + Îť w 1
â Equivalent to MAP estimation with Gaussian likelihood and
p
factorized Laplace prior p(w) â j=1 eâÎť|wj | (Seeger, 2008)
â However, posterior puts zero weight on exact zeros
⢠Heavy-tailed distributions as a proxy to sparsity
â Student distributions (Caron and Doucet, 2008)
â Generalized hyperbolic priors (Archambeau and Bach, 2008)
â Instance of automatic relevance determination (Neal, 1996)
⢠Mixtures of âDiracsâ and another absolutely continuous distributions,
e.g., âspike and slabâ (Ishwaran and Rao, 2005)
⢠Less theory than frequentist methods
58. Comparing Lasso and other strategies for linear
regression
⢠Compared methods to reach the least-square solution
1 Îť
â Ridge regression: minp y â Xw 2 + w 2
2
2
wâR 2 2
1
â Lasso: minp y â Xw 2 + Îť w 1
2
wâR 2
â Forward greedy:
â Initialization with empty set
â Sequentially add the variable that best reduces the square loss
⢠Each method builds a path of solutions from 0 to ordinary least-
squares solution
⢠Regularization parameters selected on the test set
60. Summary
â1-norm regularization
⢠â1-norm regularization leads to nonsmooth optimization problems
â analysis through directional derivatives or subgradients
â optimization may or may not take advantage of sparsity
⢠â1-norm regularization allows high-dimensional inference
⢠Interesting problems for â1-regularization
â Stable variable selection
â Weaker suďŹcient conditions (for weaker results)
â Estimation of regularization parameter (all bounds depend on the
unknown noise variance Ď 2)
61. Extensions
⢠Sparse methods are not limited to the square loss
â e.g., theoretical results for logistic loss (Van De Geer, 2008; Bach,
2009b)
⢠Sparse methods are not limited to supervised learning
â Learning the structure of Gaussian graphical models (Meinshausen
and B¨hlmann, 2006; Banerjee et al., 2008)
u
â Sparsity on matrices (last part of the tutorial)
⢠Sparse methods are not limited to variable selection in a linear
model
â See next part of the tutorial
63. Sparse methods for machine learning
Outline
⢠Introduction - Overview
⢠Sparse linear estimation with the â1-norm
â Convex optimization and algorithms
â Theoretical results
⢠Structured sparse methods on vectors
â Groups of features / Multiple kernel learning
â Extensions (hierarchical or overlapping groups)
⢠Sparse methods on matrices
â Multi-task learning
â Matrix factorization (low-rank, sparse PCA, dictionary learning)
64. Penalization with grouped variables
(Yuan and Lin, 2006)
⢠Assume that {1, . . . , p} is partitioned into m groups G1, . . . , Gm
m
⢠Penalization by i=1 wGi 2, often called â1-â2 norm
⢠Induces group sparsity
â Some groups entirely set to zero
â no zeros within groups
⢠In this tutorial:
â Groups may have inďŹnite size â MKL
â Groups may overlap â structured sparsity
65. Linear vs. non-linear methods
⢠All methods in this tutorial are linear in the parameters
⢠By replacing x by features Ό(x), they can be made non linear in
the data
⢠Implicit vs. explicit features
â â1-norm: explicit features
â â2-norm: representer theorem allows to consider implicit features if
their dot products can be computed easily (kernel methods)
66. Kernel methods: regularization by â2-norm
⢠Data: xi â X , yi â Y, i = 1, . . . , n, with features ÎŚ(x) â F = Rp
â Predictor f (x) = wâ¤ÎŚ(x) linear in the features
n
⤠Ν 2
⢠Optimization problem: minp â(yi, w ÎŚ(xi)) + w 2
wâR
i=1
2
67. Kernel methods: regularization by â2-norm
⢠Data: xi â X , yi â Y, i = 1, . . . , n, with features ÎŚ(x) â F = Rp
â Predictor f (x) = wâ¤ÎŚ(x) linear in the features
n
⤠Ν 2
⢠Optimization problem: minp â(yi, w ÎŚ(xi)) + w 2
wâR
i=1
2
⢠Representer theorem (Kimeldorf and Wahba, 1971): solution must
n
be of the form w = i=1 ÎąiÎŚ(xi)
n
Îť â¤
â Equivalent to solving: min â(yi, (KÎą)i) + Îą KÎą
ÎąâR n
i=1
2
â Kernel matrix Kij = k(xi, xj ) = ÎŚ(xi)â¤ÎŚ(xj )
68. Multiple kernel learning (MKL)
(Lanckriet et al., 2004b; Bach et al., 2004a)
⢠Sparse methods are linear!
⢠Sparsity with non-linearities
p â¤
â replace f (x) = j=1 wj xj with x â Rp and wj â R
p â¤
â by f (x) = j=1 wj ÎŚj (x) with x â X , ÎŚj (x) â Fj an wj â Fj
p p
⢠Replace the â1-norm j=1 |wj | by âblockâ â1-norm j=1 wj 2
⢠Remarks
â Hilbert space extension of the group Lasso (Yuan and Lin, 2006)
â Alternative sparsity-inducing norms (Ravikumar et al., 2008)
69. Multiple kernel learning (MKL)
(Lanckriet et al., 2004b; Bach et al., 2004a)
⢠Multiple feature maps / kernels on x â X :
â p âfeature mapsâ ÎŚj : X â Fj , j = 1, . . . , p.
â Minimization with respect to w1 â F1, . . . , wp â Fp
â Predictor: f (x) = w1â¤ÎŚ1(x) + ¡ ¡ ¡ + wpâ¤ÎŚp(x)
Ό1(x)⤠w1
Ö .
. .
. Ö
⤠â¤
x ââ ÎŚj (x)⤠wj ââ w1 ÎŚ1(x) + ¡ ¡ ¡ + wp ÎŚp(x)
Ö .
. .
. Ö
Όp(x)⤠wp
â Generalized additive models (Hastie and Tibshirani, 1990)
70. Regularization for multiple features
Ό1(x)⤠w1
Ö .
. .
. Ö
⤠â¤
x ââ ÎŚj (x)⤠wj ââ w1 ÎŚ1(x) + ¡ ¡ ¡ + wp ÎŚp(x)
Ö .
. .
. Ö
Όp(x)⤠wp
p 2 p
⢠Regularization by j=1 wj 2 is equivalent to using K = j=1 Kj
â Summing kernels is equivalent to concatenating feature spaces
71. Regularization for multiple features
Ό1(x)⤠w1
Ö .
. .
. Ö
⤠â¤
x ââ ÎŚj (x)⤠wj ââ w1 ÎŚ1(x) + ¡ ¡ ¡ + wp ÎŚp(x)
Ö .
. .
. Ö
Όp(x)⤠wp
p 2 p
⢠Regularization by j=1 wj 2 is equivalent to using K = j=1 Kj
p
⢠Regularization by j=1 wj 2 imposes sparsity at the group level
⢠Main questions when regularizing by block â1-norm:
1. Algorithms
2. Analysis of sparsity inducing properties (Ravikumar et al., 2008;
Bach, 2008b)
3. Does it correspond to a speciďŹc combination of kernels?
72. General kernel learning
⢠Proposition (Lanckriet et al, 2004, Bach et al., 2005, Micchelli and
Pontil, 2005):
n
G(K) = min i=1 â(yi, wâ¤ÎŚ(xi)) +Îť w
2
2
2
wâF
n
= max â
n
ââ(Νιi)
i=1 i â Îť Îąâ¤KÎą
2
ÎąâR
is a convex function of the kernel matrix K
⢠Theoretical learning bounds (Lanckriet et al., 2004, Srebro and Ben-
David, 2006)
â Less assumptions than sparsity-based bounds, but slower rates
73. Equivalence with kernel learning (Bach et al., 2004a)
⢠Block â1-norm problem:
n
⤠⤠Ν 2
â(yi, w1 ÎŚ1(xi) + ¡¡¡ + wp ÎŚp(xi)) + ( w1 2 + ¡ ¡ ¡ + wp 2 )
i=1
2
⢠Proposition: Block â1-norm regularization is equivalent to
p
minimizing with respect to Ρ the optimal value G( j=1 Ρj Kj )
⢠(sparse) weights Ρ obtained from optimality conditions
p
⢠dual parameters ι optimal for K = j=1 Ρj Kj ,
⢠Single optimization problem for learning both Ρ and ι
74. Proof of equivalence
n p p
⤠2
min â yi, wj ÎŚj (xi) + Îť wj 2
w1 ,...,wp
i=1 j=1 j=1
n p p
â¤
= min min â yi, wj ÎŚj (xi) + Îť wj 2/Ρj
2
w1 ,...,wp j Ρj =1
P
i=1 j=1 j=1
n p p
1/2 â1/2
= Pmin min â yi, Ëâ¤
Ρj wj Όj (xi) + Ν wj
Ë 2
2 with wj = wj Ρj
Ë
j Ρj =1 w1 ,...,wp
Ë Ë
i=1 j=1 j=1
n
1/2
= Pmin min â yi, wâ¤Î¨Îˇ (xi) + Îť w
Ë Ë 2
2
1/2
with ΨΡ (x) = (Ρ1 Ό1(x), . . . , Ρp Όp(x))
j Ρj =1 w
Ë
i=1
p p
⢠We have: ΨΡ (x)â¤Î¨Îˇ (xâ˛) = â˛
j=1 Ρj kj (x, x ) with j=1 Ρj = 1 (and Ρ 0)
75. Algorithms for the group Lasso / MKL
⢠Group Lasso
â Block coordinate descent (Yuan and Lin, 2006)
â Active set method (Roth and Fischer, 2008; Obozinski et al., 2009)
â Nesterovâs accelerated method (Liu et al., 2009)
⢠MKL
â Dual ascent, e.g., sequential minimal optimization (Bach et al.,
2004a)
â Ρ-trick + cutting-planes (Sonnenburg et al., 2006)
â Ρ-trick + projected gradient descent (Rakotomamonjy et al., 2008)
â Active set (Bach, 2008c)
76. Applications of multiple kernel learning
⢠Selection of hyperparameters for kernel methods
⢠Fusion from heterogeneous data sources (Lanckriet et al., 2004a)
⢠Two strategies for kernel combinations:
â Uniform combination â â2-norm
â Sparse combination â â1-norm
â MKL always leads to more interpretable models
â MKL does not always lead to better predictive performance
â In particular, with few well-designed kernels
â Be careful with normalization of kernels (Bach et al., 2004b)
77. Applications of multiple kernel learning
⢠Selection of hyperparameters for kernel methods
⢠Fusion from heterogeneous data sources (Lanckriet et al., 2004a)
⢠Two strategies for kernel combinations:
â Uniform combination â â2-norm
â Sparse combination â â1-norm
â MKL always leads to more interpretable models
â MKL does not always lead to better predictive performance
â In particular, with few well-designed kernels
â Be careful with normalization of kernels (Bach et al., 2004b)
⢠Sparse methods: new possibilities and new features
⢠See NIPS 2009 workshop âUnderstanding MKL methodsâ
78. Sparse methods for machine learning
Outline
⢠Introduction - Overview
⢠Sparse linear estimation with the â1-norm
â Convex optimization and algorithms
â Theoretical results
⢠Structured sparse methods on vectors
â Groups of features / Multiple kernel learning
â Extensions (hierarchical or overlapping groups)
⢠Sparse methods on matrices
â Multi-task learning
â Matrix factorization (low-rank, sparse PCA, dictionary learning)
79. Lasso - Two main recent theoretical results
1. Support recovery condition
2. Exponentially many irrelevant variables: under appropriate
assumptions, consistency is possible as long as
log p = O(n)
80. Lasso - Two main recent theoretical results
1. Support recovery condition
2. Exponentially many irrelevant variables: under appropriate
assumptions, consistency is possible as long as
log p = O(n)
⢠Question: is it possible to build a sparse algorithm that can learn
from more than 1080 features?
81. Lasso - Two main recent theoretical results
1. Support recovery condition
2. Exponentially many irrelevant variables: under appropriate
assumptions, consistency is possible as long as
log p = O(n)
⢠Question: is it possible to build a sparse algorithm that can learn
from more than 1080 features?
â Some type of recursivity/factorization is needed!
82. Hierarchical kernel learning (Bach, 2008c)
⢠Many kernels can be decomposed as a sum of many âsmallâ kernels
indexed by a certain set V : k(x, xâ˛) = kv (x, xâ˛)
vâV
⢠Example with x = (x1, . . . , xq ) â Rq (â non linear variable selection)
â Gaussian/ANOVA kernels: p = #(V ) = 2q
q
âÎą(xj âxⲠ)2 âÎą(xj âxⲠ)2 âÎą xJ âxⲠ2
1+e j = e j = e J 2
j=1 Jâ{1,...,q} jâJ Jâ{1,...,q}
â NB: decomposition is related to Cosso (Lin and Zhang, 2006)
⢠Goal: learning sparse combination vâV Ρv kv (x, xâ˛)
⢠Universally consistent non-linear variable selection requires all subsets
83. Restricting the set of active kernels
⢠With ďŹat structure
â Consider block â1-norm: vâV dv wv 2
â cannot avoid being linear in p = #(V ) = 2q
⢠Using the structure of the small kernels
1. for computational reasons
2. to allow more irrelevant variables
84. Restricting the set of active kernels
⢠V is endowed with a directed acyclic graph (DAG) structure:
select a kernel only after all of its ancestors have been selected
⢠Gaussian kernels: V = power set of {1, . . . , q} with inclusion DAG
â Select a subset only after all its subsets have been selected
1 2 3 4
12 13 14 23 24 34
123 124 134 234
1234
85. DAG-adapted norm (Zhao & Yu, 2008)
⢠Graph-based structured regularization
â D(v) is the set of descendants of v  V :
â 1/2
2
dv wD(v) 2 = dv ďŁ wt 2 
vâV vâV tâD(v)
⢠Main property: If v is selected, so are all its ancestors
1 2 3 4 1 2 3 4
12 13 14 23 24 34 12 13 14 23 24 34
123 124 134 234 123 124 134 234
1234 1234
86. DAG-adapted norm (Zhao & Yu, 2008)
⢠Graph-based structured regularization
â D(v) is the set of descendants of v  V :
â 1/2
2
dv wD(v) 2 = dv ďŁ wt 2 
vâV vâV tâD(v)
⢠Main property: If v is selected, so are all its ancestors
⢠Hierarchical kernel learning (Bach, 2008c) :
â polynomial-time algorithm for this norm
â necessary/suďŹcient conditions for consistent kernel selection
â Scaling between p, q, n for consistency
â Applications to variable selection or other kernels
87. Scaling between p, n
and other graph-related quantities
n = number of observations
p = number of vertices in the DAG
deg(V ) = maximum out degree in the DAG
num(V ) = number of connected components in the DAG
⢠Proposition (Bach, 2009a): Assume consistency condition satisďŹed,
Gaussian noise and data generated from a sparse function, then the
support is recovered with high-probability as soon as:
log deg(V ) + log num(V ) = O(n)
88. Scaling between p, n
and other graph-related quantities
n = number of observations
p = number of vertices in the DAG
deg(V ) = maximum out degree in the DAG
num(V ) = number of connected components in the DAG
⢠Proposition (Bach, 2009a): Assume consistency condition satisďŹed,
Gaussian noise and data generated from a sparse function, then the
support is recovered with high-probability as soon as:
log deg(V ) + log num(V ) = O(n)
⢠Unstructured case: num(V ) = p â log p = O(n)
⢠Power set of q elements: deg(V ) = q â log q = log log p = O(n)
90. Extensions to other kernels
⢠Extension to graph kernels, string kernels, pyramid match kernels
A B
AA AB BA BB
AAA AAB ABA ABB BAA BAB BBA BBB
⢠Exploring large feature spaces with structured sparsity-inducing norms
â Opposite view than traditional kernel methods
â Interpretable models
⢠Other structures than hierarchies or DAGs
91. Grouped variables
⢠Supervised learning with known groups:
â The â1-â2 norm
2 1/2
wG 2 = wj , with G a partition of {1, . . . , p}
GâG GâG jâG
â The â1-â2 norm sets to zero non-overlapping groups of variables
(as opposed to single variables for the â1 norm)
92. Grouped variables
⢠Supervised learning with known groups:
â The â1-â2 norm
2 1/2
wG 2 = wj , with G a partition of {1, . . . , p}
GâG GâG jâG
â The â1-â2 norm sets to zero non-overlapping groups of variables
(as opposed to single variables for the â1 norm).
⢠However, the â1-â2 norm encodes ďŹxed/static prior information,
requires to know in advance how to group the variables
⢠What happens if the set of groups G is not a partition anymore?
93. Structured Sparsity (Jenatton et al., 2009a)
⢠When penalizing by the â1-â2 norm
2 1/2
wG 2 = wj
GâG GâG jâG
â The â1 norm induces sparsity at the group level:
â Some wGâs are set to zero
â Inside the groups, the â2 norm does not promote sparsity
⢠Intuitively, the zero pattern of w is given by
{j â {1, . . . , p}; wj = 0} = G for some GⲠâ G.
GâGâ˛
⢠This intuition is actually true and can be formalized
94. Examples of set of groups G (1/3)
⢠Selection of contiguous patterns on a sequence, p = 6
â G is the set of blue groups
â Any union of blue groups set to zero leads to the selection of a
contiguous pattern
95. Examples of set of groups G (2/3)
⢠Selection of rectangles on a 2-D grids, p = 25
â G is the set of blue/green groups (with their complements, not
displayed)
â Any union of blue/green groups set to zero leads to the selection
of a rectangle
96. Examples of set of groups G (3/3)
⢠Selection of diamond-shaped patterns on a 2-D grids, p = 25
â It is possible to extent such settings to 3-D space, or more complex
topologies
â See applications later (sparse PCA)
97. Relationship bewteen G and Zero Patterns
(Jenatton, Audibert, and Bach, 2009a)
⢠G â Zero patterns:
â by generating the union-closure of G
⢠Zero patterns â G:
â Design groups G from any union-closed set of zero patterns
â Design groups G from any intersection-closed set of non-zero
patterns
98. Overview of other work on structured sparsity
⢠SpeciďŹc hierarchical structure (Zhao et al., 2009; Bach, 2008c)
⢠Union-closed (as opposed to intersection-closed) family of nonzero
patterns (Jacob et al., 2009; Baraniuk et al., 2008)
⢠Nonconvex penalties based on information-theoretic criteria with
greedy optimization (Huang et al., 2009)
99. Sparse methods for machine learning
Outline
⢠Introduction - Overview
⢠Sparse linear estimation with the â1-norm
â Convex optimization and algorithms
â Theoretical results
⢠Structured sparse methods on vectors
â Groups of features / Multiple kernel learning
â Extensions (hierarchical or overlapping groups)
⢠Sparse methods on matrices
â Multi-task learning
â Matrix factorization (low-rank, sparse PCA, dictionary learning)
100. Learning on matrices - Collaborative Filtering (CF)
⢠Given nX âmoviesâ x â X and nY âcustomersâ y â Y,
⢠predict the âratingâ z(x, y) â Z of customer y for movie x
⢠Training data: large nX à nY incomplete matrix Z that describes the
known ratings of some customers for some movies
⢠Goal: complete the matrix.
101. Learning on matrices - Multi-task learning
⢠k prediction tasks on same covariates x â Rp
â k weight vectors wj â Rp
â Joint matrix of predictors W = (w1, . . . , wk ) â RpĂk
⢠Many applications
â âtransfer learningâ
â Multi-category classiďŹcation (one task per class) (Amit et al., 2007)
⢠Share parameters between various tasks
â similar to ďŹxed eďŹect/random eďŹect models (Raudenbush and Bryk,
2002)
â joint variable or feature selection (Obozinski et al., 2009; Pontil
et al., 2007)
102. Learning on matrices - Image denoising
⢠Simultaneously denoise all patches of a given image
⢠Example from Mairal, Bach, Ponce, Sapiro, and Zisserman (2009c)
103. Two types of sparsity for matrices M â RnĂp
I - Directly on the elements of M
⢠Many zero elements: Mij = 0
M
⢠Many zero rows (or columns): (Mi1, . . . , Mip) = 0
M
104. Two types of sparsity for matrices M â RnĂp
II - Through a factorization of M = U V â¤
⢠M = U V â¤, U â RnĂm and V â RnĂm
⢠Low rank: m small
T
V
M = U
⢠Sparse decomposition: U sparse
T
M = U V
105. Structured matrix factorizations - Many instances
⢠M = U V â¤, U â RnĂm and V â RpĂm
⢠Structure on U and/or V
â Low-rank: U and V have few columns
â Dictionary learning / sparse PCA: U or V has many zeros
â Clustering (k-means): U â {0, 1}nĂm, U 1 = 1
â Pointwise positivity: non negative matrix factorization (NMF)
â SpeciďŹc patterns of zeros
â etc.
⢠Many applications
â e.g., source separation (F´votte et al., 2009), exploratory data
e
analysis
106. Multi-task learning
⢠Joint matrix of predictors W = (w1, . . . , wk ) â RpĂk
⢠Joint variable selection (Obozinski et al., 2009)
â Penalize by the sum of the norms of rows of W (group Lasso)
â Select variables which are predictive for all tasks
107. Multi-task learning
⢠Joint matrix of predictors W = (w1, . . . , wk ) â RpĂk
⢠Joint variable selection (Obozinski et al., 2009)
â Penalize by the sum of the norms of rows of W (group Lasso)
â Select variables which are predictive for all tasks
⢠Joint feature selection (Pontil et al., 2007)
â Penalize by the trace-norm (see later)
â Construct linear features common to all tasks
⢠Theory: allows number of observations which is sublinear in the
number of tasks (Obozinski et al., 2008; Lounici et al., 2009)
⢠Practice: more interpretable models, slightly improved performance
108. Low-rank matrix factorizations
Trace norm
⢠Given a matrix M â RnĂp
â Rank of M is the minimum size m of all factorizations of M into
M = U V â¤, U â RnĂm and V â RpĂm
â Singular value decomposition: M = U Diag(s)V ⤠where U and V
have orthonormal columns and s â Rm are singular values
+
⢠Rank of M equal to the number of non-zero singular values
109. Low-rank matrix factorizations
Trace norm
⢠Given a matrix M â RnĂp
â Rank of M is the minimum size m of all factorizations of M into
M = U V â¤, U â RnĂm and V â RpĂm
â Singular value decomposition: M = U Diag(s)V ⤠where U and V
have orthonormal columns and s â Rm are singular values
+
⢠Rank of M equal to the number of non-zero singular values
⢠Trace-norm (a.k.a. nuclear norm) = sum of singular values
⢠Convex function, leads to a semi-deďŹnite program (Fazel et al., 2001)
⢠First used for collaborative ďŹltering (Srebro et al., 2005)
110. Results for the trace norm
⢠Rank recovery condition (Bach, 2008d)
â The Hessian of the loss around the asymptotic solution should be
close to diagonal
⢠SuďŹcient condition for exact rank minimization (Recht et al., 2009)
⢠High-dimensional inference for noisy matrix completion (Srebro et al.,
2005; Cand`s and Plan, 2009a)
e
â May recover entire matrix from slightly more entries than the
minimum of the two dimensions
⢠EďŹcient algorithms:
â First-order methods based on the singular value decomposition (see,
e.g., Mazumder et al., 2009)
â Low-rank formulations (Rennie and Srebro, 2005; Abernethy et al.,
2009)
111. Spectral regularizations
⢠Extensions to any functions of singular values
⢠Extensions to bilinear forms (Abernethy et al., 2009)
(x, y) â ÎŚ(x)â¤BΨ(y)
on features ÎŚ(x) â RfX and Ψ(y) â RfY , and B â RfX ĂfY
⢠Collaborative ďŹltering with attributes
⢠Representer theorem: the solution must be of the form
nX nY
B= Îąij Ψ(xi)ÎŚ(yj )â¤
i=1 j=1
⢠Only norms invariant by orthogonal transforms (Argyriou et al., 2009)
112. Sparse principal component analysis
⢠Given data matrix X = (xâ¤, . . . , xâ¤)⤠â RnĂp, principal component
1 n
analysis (PCA) may be seen from two perspectives:
â Analysis view: ďŹnd the projection v â Rp of maximum variance
(with deďŹation to obtain more components)
â Synthesis view: ďŹnd the basis v1, . . . , vk such that all xi have low
reconstruction error when decomposed on this basis
⢠For regular PCA, the two views are equivalent
⢠Sparse extensions
â Interpretability
â High-dimensional inference
â Two views are diďŹerents
113. Sparse principal component analysis
Analysis view
1
⢠DSPCA (dâAspremont et al., 2007), with A = n X â¤X â RpĂp
â max v â¤Av relaxed into max v â¤Av
v 2 =1, v 0 k v 2 =1, v 1 k1/2
â using M = vv â¤, itself relaxed into max tr AM
M 0,tr M =1,1⤠|M |1 k
114. Sparse principal component analysis
Analysis view
1
⢠DSPCA (dâAspremont et al., 2007), with A = n X â¤X â RpĂp
â max v â¤Av relaxed into max v â¤Av
v 2 =1, v 0 k v 2 =1, v 1 k1/2
â using M = vv â¤, itself relaxed into max tr AM
M 0,tr M =1,1⤠|M |1 k
⢠Requires deďŹation for multiple components (Mackey, 2009)
⢠More reďŹned convex relaxation (dâAspremont et al., 2008)
⢠Non convex analysis (Moghaddam et al., 2006b)
⢠Applications beyond interpretable principal components
â used as suďŹcient conditions for high-dimensional inference
115. Sparse principal component analysis
Synthesis view
⢠Find v1, . . . , vm â Rp sparse so that
n m 2
min xi â
m
uj vj is small
uâR
i=1 j=1 2
⢠Equivalent to look for U â RnĂm and V â RpĂm such that V is
sparse and X â U V ⤠2 is small
F
116. Sparse principal component analysis
Synthesis view
⢠Find v1, . . . , vm â Rp sparse so that
n m 2
min xi â
m
uj vj is small
uâR
i=1 j=1 2
⢠Equivalent to look for U â RnĂm and V â RpĂm such that V is
sparse and X â U V ⤠2 is small
F
⢠Sparse formulation (Witten et al., 2009; Bach et al., 2008)
â Penalize columns vi of V by the â1-norm for sparsity
â Penalize columns ui of U by the â2-norm to avoid trivial solutions
m
min X â U V ⤠2
F +Îť ui 2
2 + vi 2
1
U,V
i=1
117. Structured matrix factorizations
m
min X â U V ⤠2
F +Îť ui 2
+ vi 2
U,V
i=1
⢠Penalizing by ui 2 + vi 2 equivalent to constraining ui 1 and
penalizing by vi (Bach et al., 2008)
⢠Optimization by alternating minimization (non-convex)
⢠ui decomposition coeďŹcients (or âcodeâ), vi dictionary elements
⢠Sparse PCA = sparse dictionary (â1-norm on ui)
⢠Dictionary learning = sparse decompositions (â1-norm on vi )
â Olshausen and Field (1997); Elad and Aharon (2006); Raina et al.
(2007)
119. Dictionary learning for image denoising
⢠Solving the denoising problem (Elad and Aharon, 2006)
â Extract all overlapping 8 Ă 8 patches xi â R64.
â Form the matrix X = [x1, . . . , xn]⤠â RnĂ64
â Solve a matrix factorization problem:
n
min ||X â U V â¤||2 =
F ||xi â V U (i, :)||2
2
U,V
i=1
where U is sparse, and V is the dictionary
â Each patch is decomposed into xi = V U (i, :)
â Average the reconstruction V U (i, :) of each patch xi to reconstruct
a full-sized image
⢠The number of patches n is large (= number of pixels)
120. Online optimization for dictionary learning
n
min ||xi â V U (i, :)||2 + Îť||U (i, :)||1
2
U âRnĂm ,V âC
i=1
âł
C = {V â RpĂm s.t. âj = 1, . . . , m, ||V (:, j)||2 1}.
⢠Classical optimization alternates between U and V
⢠Good results, but very slow!
121. Online optimization for dictionary learning
n
min ||xi â V U (i, :)||2 + Îť||U (i, :)||1
2
U âRnĂm ,V âC
i=1
âł
C = {V â RpĂm s.t. âj = 1, . . . , m, ||V (:, j)||2 1}.
⢠Classical optimization alternates between U and V .
⢠Good results, but very slow!
⢠Online learning (Mairal, Bach, Ponce, and Sapiro, 2009a) can
â handle potentially inďŹnite datasets
â adapt to dynamic training sets
129. Alternative usages of dictionary learning
⢠Uses the âcodeâ U as representation of observations for subsequent
processing (Raina et al., 2007; Yang et al., 2009)
⢠Adapt dictionary elements to speciďŹc tasks (Mairal et al., 2009b)
â Discriminative training for weakly supervised pixel classiďŹcation (Mairal
et al., 2008)
130. Sparse Structured PCA
(Jenatton, Obozinski, and Bach, 2009b)
⢠Learning sparse and structured dictionary elements:
m
min X â U V ⤠2
F +Îť ui 2
+ vi 2
U,V
i=1
⢠Structured norm on the dictionary elements
â grouped penalty with overlapping groups to select speciďŹc classes
of sparsity patterns
â use prior information for better reconstruction and/or added
robustness
⢠EďŹcient learning procedures through Ρ-tricks (closed form updates)
131. Application to face databases (1/3)
raw data (unstructured) NMF
⢠NMF obtains partially local features
132. Application to face databases (2/3)
(unstructured) sparse PCA Structured sparse PCA
⢠Enforce selection of convex nonzero patterns â robustness to
occlusion
133. Application to face databases (2/3)
(unstructured) sparse PCA Structured sparse PCA
⢠Enforce selection of convex nonzero patterns â robustness to
occlusion
134. Application to face databases (3/3)
⢠Quantitative performance evaluation on classiďŹcation task
45
raw data
PCA
40 NMF
SPCA
sharedâSPCA
35 SSPCA
% Correct classification
sharedâSSPCA
30
25
20
15
10
5
20 40 60 80 100 120 140
Dictionary size
135. Topic models and matrix factorization
⢠Latent Dirichlet allocation (Blei et al., 2003)
â For a document, sample θ â Rk from a Dirichlet(Îą)
â For the n-th word of the same document,
â sample a topic zn from a multinomial with parameter θ
â sample a word wn from a multinomial with parameter β(zn, :)
136. Topic models and matrix factorization
⢠Latent Dirichlet allocation (Blei et al., 2003)
â For a document, sample θ â Rk from a Dirichlet(Îą)
â For the n-th word of the same document,
â sample a topic zn from a multinomial with parameter θ
â sample a word wn from a multinomial with parameter β(zn, :)
⢠Interpretation as multinomial PCA (Buntine and Perttu, 2003)
â Marginalizing over topic zn, given θ, each word wn is selected from
a multinomial with parameter k θk β(z, :) = β â¤Î¸
z=1
â Row of β = dictionary elements, θ code for a document
137. Topic models and matrix factorization
⢠Two diďŹerent views on the same problem
â Interesting parallels to be made
â Common problems to be solved
⢠Structure on dictionary/decomposition coeďŹcients with adapted
priors, e.g., nested Chinese restaurant processes (Blei et al., 2004)
⢠Other priors and probabilistic formulations (GriďŹths and Ghahramani,
2006; Salakhutdinov and Mnih, 2008; Archambeau and Bach, 2008)
⢠IdentiďŹability and interpretation/evaluation of results
⢠Discriminative tasks (Blei and McAuliďŹe, 2008; Lacoste-Julien
et al., 2008; Mairal et al., 2009b)
⢠Optimization and local minima
138. Sparsifying linear methods
⢠Same pattern than with kernel methods
â High-dimensional inference rather than non-linearities
⢠Main diďŹerence: in general no unique way
⢠Sparse CCA (Sriperumbudur et al., 2009; Hardoon and Shawe-Taylor,
2008; Archambeau and Bach, 2008)
⢠Sparse LDA (Moghaddam et al., 2006a)
⢠Sparse ...
139. Sparse methods for matrices
Summary
⢠Structured matrix factorization has many applications
⢠Algorithmic issues
â Dealing with large datasets
â Dealing with structured sparsity
⢠Theoretical issues
â IdentiďŹability of structures, dictionaries or codes
â Other approaches to sparsity and structure
⢠Non-convex optimization versus convex optimization
â ConvexiďŹcation through unbounded dictionary size (Bach et al.,
2008; Bradley and Bagnell, 2009) - few performance improvements
140. Sparse methods for machine learning
Outline
⢠Introduction - Overview
⢠Sparse linear estimation with the â1-norm
â Convex optimization and algorithms
â Theoretical results
⢠Structured sparse methods on vectors
â Groups of features / Multiple kernel learning
â Extensions (hierarchical or overlapping groups)
⢠Sparse methods on matrices
â Multi-task learning
â Matrix factorization (low-rank, sparse PCA, dictionary learning)
141. Links with compressed sensing
(Baraniuk, 2007; Cand`s and Wakin, 2008)
e
⢠Goal of compressed sensing: recover a signal w â Rp from only n
measurements y = Xw â Rn
⢠Assumptions: the signal is k-sparse, n much smaller than p
⢠Algorithm: minwâRp w 1 such that y = Xw
⢠SuďŹcient condition on X and (k, n, p) for perfect recovery:
â Restricted isometry property (all small submatrices of X â¤X must
be well-conditioned)
â Such matrices are hard to come up with deterministically, but
random ones are OK with k log p = O(n)
⢠Random X for machine learning?
142. Why use sparse methods?
⢠Sparsity as a proxy to interpretability
â Structured sparsity
⢠Sparse methods are not limited to least-squares regression
⢠Faster training/testing
⢠Better predictive performance?
â Problems are sparse if you look at them the right way
â Problems are sparse if you make them sparse
143. Conclusion - Interesting questions/issues
⢠Implicit vs. explicit features
â Can we algorithmically achieve log p = O(n) with explicit
unstructured features?
⢠Norm design
â What type of behavior may be obtained with sparsity-inducing
norms?
⢠OverďŹtting convexity
â Do we actually need convexity for matrix factorization problems?
144. Hiring postdocs and PhD students
European Research Council project on
Sparse structured methods for machine learning
⢠PhD positions
⢠1-year and 2-year postdoctoral positions
⢠Machine learning (theory and algorithms), computer vision, audio
processing, signal processing
⢠Located in downtown Paris (Ecole Normale Sup´rieure - INRIA)
e
⢠http://www.di.ens.fr/~ fbach/sierra/
145. References
J. Abernethy, F. Bach, T. Evgeniou, and J.-P. Vert. A new approach to collaborative ďŹltering: Operator
estimation with spectral regularization. Journal of Machine Learning Research, 10:803â826, 2009.
Y. Amit, M. Fink, N. Srebro, and S. Ullman. Uncovering shared structures in multiclass classiďŹcation.
In Proceedings of the 24th international conference on Machine Learning (ICML), 2007.
C. Archambeau and F. Bach. Sparse probabilistic projections. In Advances in Neural Information
Processing Systems 21 (NIPS), 2008.
A. Argyriou, C.A. Micchelli, and M. Pontil. On spectral learning. Journal of Machine Learning
Research, 2009. To appear.
F. Bach. High-Dimensional Non-Linear Variable Selection through Hierarchical Kernel Learning.
Technical Report 0909.0844, arXiv, 2009a.
F. Bach. Bolasso: model consistent lasso estimation through the bootstrap. In Proceedings of the
Twenty-ďŹfth International Conference on Machine Learning (ICML), 2008a.
F. Bach. Consistency of the group Lasso and multiple kernel learning. Journal of Machine Learning
Research, 9:1179â1225, 2008b.
F. Bach. Exploring large feature spaces with hierarchical multiple kernel learning. In Advances in
Neural Information Processing Systems, 2008c.
F. Bach. Self-concordant analysis for logistic regression. Technical Report 0910.4627, ArXiv, 2009b.
F. Bach. Consistency of trace norm minimization. Journal of Machine Learning Research, 9:1019â1048,
2008d.
146. F. Bach, G. R. G. Lanckriet, and M. I. Jordan. Multiple kernel learning, conic duality, and the SMO
algorithm. In Proceedings of the International Conference on Machine Learning (ICML), 2004a.
F. Bach, R. Thibaux, and M. I. Jordan. Computing regularization paths for learning multiple kernels.
In Advances in Neural Information Processing Systems 17, 2004b.
F. Bach, J. Mairal, and J. Ponce. Convex sparse matrix factorizations. Technical Report 0812.1869,
ArXiv, 2008.
O. Banerjee, L. El Ghaoui, and A. dâAspremont. Model selection through sparse maximum likelihood
estimation for multivariate Gaussian or binary data. The Journal of Machine Learning Research, 9:
485â516, 2008.
R. Baraniuk. Compressive sensing. IEEE Signal Processing Magazine, 24(4):118â121, 2007.
R. G. Baraniuk, V. Cevher, M. F. Duarte, and C. Hegde. Model-based compressive sensing. Technical
report, arXiv:0808.3572, 2008.
A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems.
SIAM Journal on Imaging Sciences, 2(1):183â202, 2009.
D. Bertsekas. Nonlinear programming. Athena ScientiďŹc, 1995.
P. Bickel, Y. Ritov, and A. Tsybakov. Simultaneous analysis of Lasso and Dantzig selector. Annals of
Statistics, 37(4):1705â1732, 2009.
D. Blei, T.L. GriďŹths, M.I. Jordan, and J.B. Tenenbaum. Hierarchical topic models and the nested
Chinese restaurant process. Advances in neural information processing systems, 16:106, 2004.
D.M. Blei and J. McAuliďŹe. Supervised topic models. In Advances in Neural Information Processing
Systems (NIPS), volume 20, 2008.
147. D.M. Blei, A.Y. Ng, and M.I. Jordan. Latent dirichlet allocation. The Journal of Machine Learning
Research, 3:993â1022, 2003.
J. F. Bonnans, J. C. Gilbert, C. Lemar´chal, and C. A. Sagastizbal. Numerical Optimization Theoretical
e
and Practical Aspects. Springer, 2003.
J. M. Borwein and A. S. Lewis. Convex Analysis and Nonlinear Optimization. Number 3 in CMS
Books in Mathematics. Springer-Verlag, 2000.
L. Bottou and O. Bousquet. The tradeoďŹs of large scale learning. In Advances in Neural Information
Processing Systems (NIPS), volume 20, 2008.
S. P. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
D. Bradley and J. D. Bagnell. Convex coding. In Proceedings of the Proceedings of the Twenty-Fifth
Conference Annual Conference on Uncertainty in ArtiďŹcial Intelligence (UAI-09), 2009.
F. Bunea, A.B. Tsybakov, and M.H. Wegkamp. Aggregation for Gaussian regression. Annals of
Statistics, 35(4):1674â1697, 2007.
W. Buntine and S. Perttu. Is multinomial PCA multi-faceted clustering or dimensionality reduction. In
International Workshop on ArtiďŹcial Intelligence and Statistics (AISTATS), 2003.
E. Cand`s and T. Tao. The Dantzig selector: statistical estimation when p is much larger than n.
e
Annals of Statistics, 35(6):2313â2351, 2007.
E. Cand`s and M. Wakin. An introduction to compressive sampling. IEEE Signal Processing Magazine,
e
25(2):21â30, 2008.
E.J. Cand`s and Y. Plan. Matrix completion with noise. 2009a. Submitted.
e
E.J. Cand`s and Y. Plan. Near-ideal model selection by l1 minimization. The Annals of Statistics, 37
e
148. (5A):2145â2177, 2009b.
F. Caron and A. Doucet. Sparse Bayesian nonparametric regression. In 25th International Conference
on Machine Learning (ICML), 2008.
S. S. Chen, D. L. Donoho, and M. A. Saunders. Atomic decomposition by basis pursuit. SIAM Review,
43(1):129â159, 2001.
A. dâAspremont and L. El Ghaoui. Testing the nullspace property using semideďŹnite programming.
Technical Report 0807.3520v5, arXiv, 2008.
A. dâAspremont, El L. Ghaoui, M. I. Jordan, and G. R. G. Lanckriet. A direct formulation for sparse
PCA using semideďŹnite programming. SIAM Review, 49(3):434â48, 2007.
A. dâAspremont, F. Bach, and L. El Ghaoui. Optimal solutions for sparse principal component analysis.
Journal of Machine Learning Research, 9:1269â1294, 2008.
D.L. Donoho and J. Tanner. Neighborliness of randomly projected simplices in high dimensions.
Proceedings of the National Academy of Sciences of the United States of America, 102(27):9452,
2005.
B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression. Annals of statistics, 32
(2):407â451, 2004.
M. Elad and M. Aharon. Image denoising via sparse and redundant representations over learned
dictionaries. IEEE Transactions on Image Processing, 15(12):3736â3745, 2006.
J. Fan and R. Li. Variable Selection Via Nonconcave Penalized Likelihood and Its Oracle Properties.
Journal of the American Statistical Association, 96(456):1348â1361, 2001.
M. Fazel, H. Hindi, and S.P. Boyd. A rank minimization heuristic with application to minimum
149. order system approximation. In Proceedings of the American Control Conference, volume 6, pages
4734â4739, 2001.
C. F´votte, N. Bertin, and J.-L. Durrieu. Nonnegative matrix factorization with the itakura-saito
e
divergence. with application to music analysis. Neural Computation, 21(3), 2009.
J. Friedman, T. Hastie, H. H
âoďŹing, and R. Tibshirani. Pathwise coordinate optimization. Annals of Applied Statistics, 1(2):
302â332, 2007.
W. Fu. Penalized regressions: the bridge vs. the Lasso. Journal of Computational and Graphical
Statistics, 7(3):397â416, 1998).
T. GriďŹths and Z. Ghahramani. InďŹnite latent feature models and the Indian buďŹet process. Advances
in Neural Information Processing Systems (NIPS), 18, 2006.
D. R. Hardoon and J. Shawe-Taylor. Sparse canonical correlation analysis. In Sparsity and Inverse
Problems in Statistical Theory and Econometrics, 2008.
T. J. Hastie and R. J. Tibshirani. Generalized Additive Models. Chapman & Hall, 1990.
J. Huang and T. Zhang. The beneďŹt of group sparsity. Technical Report 0901.2962v2, ArXiv, 2009.
J. Huang, S. Ma, and C.H. Zhang. Adaptive Lasso for sparse high-dimensional regression models.
Statistica Sinica, 18:1603â1618, 2008.
J. Huang, T. Zhang, and D. Metaxas. Learning with structured sparsity. In Proceedings of the 26th
International Conference on Machine Learning (ICML), 2009.
H. Ishwaran and J.S. Rao. Spike and slab variable selection: frequentist and Bayesian strategies. The
Annals of Statistics, 33(2):730â773, 2005.
150. L. Jacob, G. Obozinski, and J.-P. Vert. Group Lasso with overlaps and graph Lasso. In Proceedings of
the 26th International Conference on Machine Learning (ICML), 2009.
R. Jenatton, J.Y. Audibert, and F. Bach. Structured variable selection with sparsity-inducing norms.
Technical report, arXiv:0904.3523, 2009a.
R. Jenatton, G. Obozinski, and F. Bach. Structured sparse principal component analysis. Technical
report, arXiv:0909.1440, 2009b.
A. Juditsky and A. Nemirovski. On veriďŹable suďŹcient conditions for sparse signal recovery via
â1-minimization. Technical Report 0809.2650v1, ArXiv, 2008.
G. S. Kimeldorf and G. Wahba. Some results on TchebycheďŹan spline functions. J. Math. Anal.
Applicat., 33:82â95, 1971.
S. Lacoste-Julien, F. Sha, and M.I. Jordan. DiscLDA: Discriminative learning for dimensionality
reduction and classiďŹcation. Advances in Neural Information Processing Systems (NIPS) 21, 2008.
G. R. G. Lanckriet, T. De Bie, N. Cristianini, M. I. Jordan, and W. S. Noble. A statistical framework
for genomic data fusion. Bioinformatics, 20:2626â2635, 2004a.
G. R. G. Lanckriet, N. Cristianini, L. El Ghaoui, P. Bartlett, and M. I. Jordan. Learning the kernel
matrix with semideďŹnite programming. Journal of Machine Learning Research, 5:27â72, 2004b.
H. Lee, A. Battle, R. Raina, and A. Ng. EďŹcient sparse coding algorithms. In Advances in Neural
Information Processing Systems (NIPS), 2007.
Y. Lin and H. H. Zhang. Component selection and smoothing in multivariate nonparametric regression.
Annals of Statistics, 34(5):2272â2297, 2006.
J. Liu, S. Ji, and J. Ye. Multi-Task Feature Learning Via EďŹcient l2,-Norm Minimization. Proceedings
151. of the 25th Conference on Uncertainty in ArtiďŹcial Intelligence (UAI), 2009.
K. Lounici. Sup-norm convergence rate and sign concentration property of Lasso and Dantzig
estimators. Electronic Journal of Statistics, 2:90â102, 2008.
K. Lounici, A.B. Tsybakov, M. Pontil, and S.A. van de Geer. Taking advantage of sparsity in multi-task
learning. In Conference on Computational Learning Theory (COLT), 2009.
J. Lv and Y. Fan. A uniďŹed approach to model selection and sparse recovery using regularized least
squares. Annals of Statistics, 37(6A):3498â3528, 2009.
L. Mackey. DeďŹation methods for sparse PCA. Advances in Neural Information Processing Systems
(NIPS), 21, 2009.
N. Maculan and G.J.R. GALDINO DE PAULA. A linear-time median-ďŹnding algorithm for projecting
a vector on the simplex of R?n? Operations research letters, 8(4):219â222, 1989.
J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. Discriminative learned dictionaries for local
image analysis. In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2008.
J. Mairal, F. Bach, J. Ponce, and G. Sapiro. Online dictionary learning for sparse coding. In
International Conference on Machine Learning (ICML), 2009a.
J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. Supervised dictionary learning. Advances
in Neural Information Processing Systems (NIPS), 21, 2009b.
J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. Non-local sparse models for image
restoration. In International Conference on Computer Vision (ICCV), 2009c.
H. M. Markowitz. The optimization of a quadratic function subject to linear constraints. Naval
Research Logistics Quarterly, 3:111â133, 1956.
152. P. Massart. Concentration Inequalities and Model Selection: Ecole dâ´t´ de Probabilit´s de Saint-Flour
ee e
23. Springer, 2003.
R. Mazumder, T. Hastie, and R. Tibshirani. Spectral Regularization Algorithms for Learning Large
Incomplete Matrices. 2009. Submitted.
N. Meinshausen. Relaxed Lasso. Computational Statistics and Data Analysis, 52(1):374â393, 2008.
N. Meinshausen and P. B¨hlmann. High-dimensional graphs and variable selection with the lasso.
u
Annals of statistics, 34(3):1436, 2006.
N. Meinshausen and P. B¨hlmann. Stability selection. Technical report, arXiv: 0809.2932, 2008.
u
N. Meinshausen and B. Yu. Lasso-type recovery of sparse representations for high-dimensional data.
Annals of Statistics, 37(1):246â270, 2008.
C.A. Micchelli and M. Pontil. Learning the kernel function via regularization. Journal of Machine
Learning Research, 6(2):1099, 2006.
B. Moghaddam, Y. Weiss, and S. Avidan. Generalized spectral bounds for sparse LDA. In Proceedings
of the 23rd international conference on Machine Learning (ICML), 2006a.
B. Moghaddam, Y. Weiss, and S. Avidan. Spectral bounds for sparse PCA: Exact and greedy
algorithms. In Advances in Neural Information Processing Systems, volume 18, 2006b.
R.M. Neal. Bayesian learning for neural networks. Springer Verlag, 1996.
Y. Nesterov. Introductory lectures on convex optimization: A basic course. Kluwer Academic Pub,
2003.
Y. Nesterov. Gradient methods for minimizing composite objective function. Center for Operations
Research and Econometrics (CORE), Catholic University of Louvain, Tech. Rep, 76, 2007.
153. G. Obozinski, M.J. Wainwright, and M.I. Jordan. High-dimensional union support recovery in
multivariate regression. In Advances in Neural Information Processing Systems (NIPS), 2008.
G. Obozinski, B. Taskar, and M.I. Jordan. Joint covariate selection and joint subspace selection for
multiple classiďŹcation problems. Statistics and Computing, pages 1â22, 2009.
B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy employed
by V1? Vision Research, 37:3311â3325, 1997.
M. R. Osborne, B. Presnell, and B. A. Turlach. On the lasso and its dual. Journal of Computational
and Graphical Statistics, 9(2):319â337, 2000.
M. Pontil, A. Argyriou, and T. Evgeniou. Multi-task feature learning. In Advances in Neural Information
Processing Systems, 2007.
R. Raina, A. Battle, H. Lee, B. Packer, and A.Y. Ng. Self-taught learning: Transfer learning from
unlabeled data. In Proceedings of the 24th International Conference on Machine Learning (ICML),
2007.
A. Rakotomamonjy, F. Bach, S. Canu, and Y. Grandvalet. SimpleMKL. Journal of Machine Learning
Research, 9:2491â2521, 2008.
S.W. Raudenbush and A.S. Bryk. Hierarchical linear models: Applications and data analysis methods.
Sage Pub., 2002.
P. Ravikumar, H. Liu, J. LaďŹerty, and L. Wasserman. SpAM: Sparse additive models. In Advances in
Neural Information Processing Systems (NIPS), 2008.
B. Recht, W. Xu, and B. Hassibi. Null Space Conditions and Thresholds for Rank Minimization. 2009.
Submitted.