1) The document presents the Low-Rank Regularized Heterogeneous Tensor Decomposition (LRRHTD) method for subspace clustering. LRRHTD seeks orthogonal projection matrices for all but the last tensor mode, and a low-rank projection matrix imposed with nuclear norm for the last mode, to obtain the lowest rank representation that reveals global sample structure for clustering. 2) LRRHTD models an Mth-order tensor dataset as a (M+1)th-order tensor by concatenating individual samples. It aims to find M orthogonal factor matrices for intrinsic representation and the lowest rank representation using the mapped low-dimensional tensor as a dictionary. 3) LRRHTD formulates an

1. 1
Dec. 17, 2018
Jay Chang
ADMM

2. Distributed Stochastic ADMM for
Matrix Factorization
2
Z.-Q. Yu, X.-J. Shi, L. Yan, W.-J. Li, "Distributed stochastic admm for matrixfactorization", Proceedings of the 23rd ACM Int. Conf. on Inform. and
Knowl. Manag, pp. 1259-1268, 2014.

3. 3
Matrix Factorization

4. 4
ALS-based Parallel MF Models
• ALS (Alternating Least Square) allow the columns in both U and V can be independently
updated by following equations:
Parallel ALS
: , , ,
:
randinit( , )
ra
Initial factors
1
1 Executed in parallel
ndinit , )(
m n
iter T
k T
m k
m
k
i
n
λ×
←
∈
←
←
←
Algorithm :
Input R
for to do
par fo
Initializa
r to
tio
par
n
U
V
⊳
⊳
ℝ
1 Executed in parallelj n←for to
end for
⊳

5. 5
ALS-based Improvement
CCD (Cyclic Coordinate Descent):
Instead of optimizing the whole vector or at one time, CCD adopts the coordinate descent
to optimize each element of or separately in order to avoid t
i j
i j
∗ ∗
∗ ∗
U V
U V
i
1
he matrix inverse.
CCD :
Further improves CCD's performance by changing the updating sequence in CCD.
updates one element in or each time by using coordinate descent.
kT T
d d d dd ∗ ∗ ∗ ∗=
+ +
= U V U V U V
i
CCD: Pilerl'szy, D. Zibriczky, and D. Tikk. “Fast als-based matrix factorization for explicit and implicit feedback datasets.” In RecSys, pages 71-78, 2010.
CCD++: H.-F. Yu, C.-J. Hsieh, S. Si, and I. S. Dhillon. “Scalable coordinate descent approaches to parallel matrix factorization for recommender systems.”
In ICDM, pages 765-774, 2012.

6. 6
Some Derivations for ALS
2 22
,
1
1
( , ) ( )
( , )
0
( , )
0
T
m n m k n k
T
ui u i u i
i u u i
T
u i i ui i
i iu
T
i u u ui u
u ui
Loss r
Loss
r
Loss
r
λ
λ
λ
× × ×
−
−
≈
 
= − + + 
 
∂  
=  = + ∂  
∂  
=  = + ∂  
  
 
 
R X Y
X Y x y x y
X Y
x y y I y
x
X Y
y x x I x
y

7. SGD (Stochastic Gradient Descent) randomly select one rating index ( , ) from each time,
then update the corresponding variables:
where , is the learning rate.
Conflicts ex
T
ij ij i j
i j
Rε η∗ ∗
Ω
= − U V
i
i ist between two nodes when their randomly selected ratings share
either the same user index or the same item index.
Hogwild!
DSGD (Distributed SGD)
FPSGD (fast parallel SGD)
i
i
i
7
SGD-based Parallel MF Models
Hogwild!: F. Niu, B. Recht, C. Re, and S. J. Wright. "Hogwild!: A lock-free approach to parallelizing stochastic gradient descent." In NIPS, pages
693-701, 2011.
DSGD: R. Gemulla, E. Nijkamp, P. J. Haas, and Y. Sismanis. "Large-scale matrix factorization with distributed stochastic gradient descent." In KDD,
pages 69-77, 2011.
FPSGD: Y. Zhuang, W.-S. Chin, Y.-C. Juan, and C.-J. Lin. "A fast parallel sgd for matrix factorization in shared memory systems." In RecSys, pages
249-256, 2013.

8. 8
: the number of latent factor , the learning rata , regularization parameters , , max iteration step, and the rating matrix
: Initialize a random matrix for user matrix and
U VD R
U
λη λInput
Initialization
= 1, 2,...step
( , , ) in
make prediction
error
update and
(1 )
(1 )
item matrix
T
i j
i j
i U i ij j
j V
t
u i r R
pr U V
e r pr
U V
U U e V
V
V
ηλ η
ηλ
=
= −
← − +
← −
for do
for
where T
j ij i ij ij i jV e U e R U Vη+ = −
end
end
Pseudo Code

9. 9
Some Derivations for SGD

10. 10
ADMM (Alternating Direction Methods of Multipliers)
• ADMM is used to solve the constrained problems as follows
• First gets the augmented Lagrangian as follows:
• The ADMM solution can be got be repeating the following three steps:
• If f(x) or g(z) are separable, the corresponding steps of ADMM can be done in parallel.
• So ADMM can be used to design distributed learning algorithms for large-scale problems.

11. 11
Data Split Strategy
P nodes, 1~p denote computer node id.
Local item
latent matrix
Global item
latent matrix
m n×
k n×
k n×
k n×
m k×
( )m p k×
( )m p n×

12. 12
LocalMFSplit
parallel

13. • Based on split strategy LocalMFSplit, the MF problem can be reformulated as follows:
• Define objective function by using augmented Lagrangian method:
where
13
Distributed ADMM
1 denotes the ( , ) indices of the ratiwher ngs located ine { } , nodep P p
p i j p== ΩVVVVV
Analog, but objective function Lp is non-convexconvex convex

14. 14
Distributed ADMM
• We can get
• ADMM get the solutions by repeating the following three steps:
• The solution for is:
, which can be calculated efficiently.
• The problem lies in getting efficiently.
// ? We hope we can decouple U, V and
do parallel
// parallel: locally updated on each node
1 1
1 1 1
1 1 1
scatter , , , ; update , (
, ; update (global)
scatt
in each node,
er
in parallel)
gather
in each node, in p, ; update arallel)(
p p p p p
t t t t
t t t
p p
t t t
+ +
+ + +
+ + +
U V Θ V U V
U V V
V V Θ
Problems become how to
construct a surrogate objective
function, which is convex so
that we can use ADMM three
steps update.
non-convex

15. 15
P.S. Consensus optimization via ADMM
S. P. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. “Distributed optimization and statistical learning via the alternating direction method of multipliers.”
Foundations and Trends in Machine Learning, pages 1-122, 2011.

16. 16
P.S. Consensus optimization via ADMM
S. P. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. “Distributed optimization and statistical learning via the alternating direction method of multipliers.”
Foundations and Trends in Machine Learning, pages 1-122, 2011.
g,
with regularization,
averaging in update is followed byz ρprox

17. • Batch Learning: construct a surrogate objective function Gp, which is convex and can make U and
V decouple from each other, but is still not very efficient.
• For improve efficiency Stochastic Learning are proposed for DS-ADMM
17
Stochastic Learning for Distributed ADMM
convex
access all ratings related to U*i to update each U*I, so slow.

18. 18
DS-ADMM algorithm

19. 19
Scheduler Comparison

20. 20
In x-axis why not use # iteration but time ?

21. 21
A Flexible and Efficient Algorithmic
Framework for
Constrained Matrix and Tensor
Factorization
Kejun Huang, Nicholas D Sidiropoulos, and Athanasios P Liavas. “A flexible and efficient algorithmic framework for constrained matrix and tensor
factorization.” IEEE Transactions on Signal Processing, 64(19) 5052–5065, 2016.

22. • Optimization problem
• Optimization problem is not convex in both W and H, but is convex in one if we fix the other,
thus alternating optimization (AO) is usually employed.
• ADMM for regularized least-squares
• And derived the following iterates:
22
Constrained Matrix Factorization
where ( ) and ( ) are latent factors and regularizations,
, , , .
W H
m n m k n k k n
r r
× × × ×
⋅ ⋅
∈ ∈ ∈ ∈
W H
Y W H Hɶℝ ℝ ℝ ℝ
2
cache ,
Cholesky decomposition ,
update by forward and backward substitution,
complexity ( ).
T
T T
k n
ρ+ =

W Y
W W I LL
H
OOOO
ɶ
Update while fixing using ADMM.H W

23. 23
Proximity Operator
• And derived the following iterates:
of function (1/ ) ( ) around point
T
proximity operator rρ ⋅
−H Uɶ

24. 24
Most Commonly Used Constraints/Regularizations
+
1
Non-negativity. ( ) is the indicator function of .
Lasso regularization. For ( ) the update is the well-known -
operator
Simplex constraint. In some probabilistic mode
r
r soft thresholdingλ
⋅
=H H
i ℝ
i
i l we need to constrain the columns or rows
to be elementwise non-negative and sum up to one.
Smoothness regularization. We can encourage the columns of to be smooth by
adding the regularizatio
Hi
2
n ( ) ( / 2) .
Projections onto non-convex constraints, e.g. cardinality constraints can be handled by
- . But ADMM is not guaranteed to converge to the optimal solution.
F
r
hard thresholding
λ=H TH
i

25. 25
ADMM Algorithm
Define relative primal residual
and relative dual residual where H0 is H from the previous ADMM iteration.

26. 26
AO-ADMM Algorithm
• We can initialize the current ADMM update using the previous results W and H.
• ADMM is an subroutine for alternating optimization (AO) so called AO-ADMM.

27. 27
General Loss Solution for ADMM
where is the scaled dual variable corresponding to the constraint ,
is the scaled dual variable corresponding to the equality constraint .
T
=
=
U H H
V Y WH
ɶ
ɶ ɶ

28. 28
Most Commonly Used Non-Least-Squares Loss Functions
• Missing values. In the case that only a subset of the entries in Y are available.
• Robust fitting. In the case that data entries are not uniformly corrupted by noise but only sparingly
corrupted by outliers, or when the noise is dense but heavy-tailed (e.g., Laplacian-distributed), we
can use the L1 norm as the loss function for robust fitting.
• Huber fitting. Another way to deal with possible outliers in Y is to use the Huber function to
measure the loss.
• Kullback-Leibler divergence. A commonly adopted loss function for non-negative integer data is
the Kullback-Leibler (K-L) divergence defined as

29. 29
…(12)
General Loss ADMM Algorithm
Define relative primal residual
and relative dual residual where H0 is H from the previous ADMM iteration.

30. 30
Extension to Tensor Factorization
2
(1)
(1)
(1)
arg min ( )
Khatri-Rao product and ( ), is mode-1 matricization of tensor ,
element-wise (Hadamard) product, , ( ) .
T T
F
T
T T T T T
← −
=
∗ = ∗ =
A
A Y C B A
W C B Y
W W C C B B W Y C B Y
YYYY
⊙
⊙ ⊙
⊙

31. 31
Tensor
AO-ADMM Algorithm
…(1)
• The outer AO framework naturally
provides a good initial point to the
inner ADMM iteration called warm-
start strategy.
LS loss function
General loss function

32. Low-Rank Regularized
Heterogeneous Tensor
Decomposition (LRRHTD) for
Subspace Clustering
32
J. Zhang, X. Li, P. Jing, J. Liu, Y. Su, "Low-rank regularized heterogeneous tensor decomposition for subspace clustering", IEEE Signal Process. Lett.,
vol. 25, no. 3, pp. 333-337, Mar. 2018.

33. 33
Tucker Decomposition
tensor with , , ,I J K P Q R I P J Q K R× × × × × × ×
∈ ∈ ∈ ∈ ∈A B Cℝ ℝ ℝ ℝ ℝX GX GX GX G
core tensor
factor matrix
factor matrix
factor matrix
The factor matrices (which are usually orthogonal) A, B, and C are often referred to as
the principal component in the respective tensor mode.
this will result in a compression of XXXX, with GGGG being the compressed version of XXXX.
is Kronecker product⊗

34. 34
Tucker Decomposition
• Tucker model can be generalized to N-way tensors
• The concept of n-rank (denoted by rankn(XXXX)): corresponds to the column rank of the n-th
unfolding of the tensor XXXX.
• According to the type of constraints Tucker decomposition approaches can be roughly grouped
into three categories:
• orthogonal tensor decomposition
• non-negative tensor decomposition
• sparse tensor decomposition
• Almost all of the above algorithms decompose tensors based on the isotropy hypothesis (i.e.
orthogonal, non-negative...), meaning that the factor matrices are learned in an equivalent way
for all modes.
• Not suitable for heterogeneous tensor data.
( )rank ( ) rank( )n n= XXXXX

35. • For all but the last mode, LRRHTD seeks a set of orthogonal projection matrices to map the
original tensor data into a low-dimensional common subspace.
• But for the last mode, a low-rank projection matrix is learned by imposing a nuclear-norm so that
a lowest rank representation that reveals the global structure of samples is obtained for
performing clustering.
• M-th order tensors, N is the total number of samples:
• We concatenate the N tensors to yield a (M+1)-th order tensor
• The goal of LRRHTD is to find M orthogonal factor matrices for intrinsic
low-dimensional representation and the lowest rank representation using the mapped
low-dimensional tensor as a dictionary, and D < N.
35
Low-Rank Regularized Heterogeneous Tensor
Decomposition (LRRHTD)

36. • Tucker decomposition of the concatenated tensor XXXX can be estimated in a general form as follows:
where is the core tensor and is the approximation error tensor.
• Cost function:
and
36
Low-Rank Regularized Heterogeneous Tensor
Decomposition (LRRHTD)
arg min

37. 37
Low-Rank Regularized Heterogeneous Tensor
Decomposition (LRRHTD)

38. 38
Thank you for your attention