Derivation of Least Square solution for Semi supervised learning problem using matrix differentiation rules

1. Least Square solution for Semi supervised learning
Ahmed Taha
Starting from the objective Function for the supervised learning problem
min FtLF = 1=2
X
i;j
Wij (fi fj)
Now we want to extend the same equation for the Semi supervised learning problem
So the minimization problem is now
J(F) = FtLF +
X
i
(fi yi)
Where is the weight of the Labeled Data y1; y2::::yi
J(F) = FtLF + (F Y )T (F Y ) (1)
= FtLF + FT (F Y ) Y T (F Y )
= FtLF + FT F FT Y Y T F + Y T Y
to minimise , we dierentiate and equate by zero
d J(F)
dF
=
d(FtLF)
dF
+
d(FtF)
dF
d(FtY )
dF
d(Y tF)
dF
+
d(Y tY )
dF
(2)
Based on matrix dierentiation rules 1, it can be shown that
d(XtAX)
dX
= (A + At)X
therefore
dJ(F)
dF
= 0 = (L + Lt) F + ( + t)F
d(FtY )
dF
d(Y tF)
dF
Based on matrix dierentiation rules, it can be shown that
d(XtA)
dX
= A and
d (AtX)
dX
= A (3)
1Matrix Dierentiation Rules : https://www.cs.nyu.edu/ roweis/notes/matrixid.pdf
1

2. Since is diagonal matrix and L is symmetric matrix = t;L = Lt
dJ(F)
dF
= 0 = 2LF + 2F Y Y t
t
= 2LF + 2F Y Y
= 2(L + )F 2Y
(L + )F = Y
F = (L + )1Y
This solution is called Least Square solution, it provide exact solution for classi

3. ca-
tion and object embedding problems, unfortunately this solution is very costly especially
when the number of objects is very big, Imagine you have 80 million Object, you will
have 80 x 80 million matrix , trying to calculate their inverse and calculate anity matrix
for these 80 million objects.
2