Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Why Biased Matrix
Factorization Works Well?
2018. 8. 29.
JoonyoungYi
joonyoung.yi@kaist.ac.kr

Machine Learning & Intellig...
TABLE OF CONTENTS
1. Netflix Prize and Winner’s Algorithm
2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009)
3...
TABLE OF CONTENTS
1. Netflix Prize and Winner’s Algorithm
2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009)
3...
NETFLIX PRIZE
• In October 2016, Netflix open a $1M Prize called Netflix Prize.
• To improve recommendation accuracy by 10...
DATA DESCRIPTION OF NETFLIX PRIZE
• Each set(training set, test set) is a set of (i, j, Mij)s.
• i: user index
• j: item(o...
DATA DESCRIPTION OF NETFLIX PRIZE
• RMSE(Root Mean Square Error) was used for measuring accuracy.
• Rij: predicted rating ...
RMSE on Test Set
RMSE OF NETFLIX, PRIZE WINNER AND BIASED-MF
• RMSE of Cinematch (Original Netflix Recommendation Engine):...
BIASED-MF IS FAST AND WORKS WELL
• Many Algorithms are presented after Biased-MF.
• Biased-MF is the fastest algorithm amo...
TABLE OF CONTENTS
1. Netflix Prize and Winner’s Algorithm
2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009)
3...
FUNK SVD(SINGULAR VECTOR DECOMPOSITION)
• Before introducing Biased-MF, Let’s see Funk SVD(suggested by Funk).
• Goal of N...
SOLUTION OF THE FUNK SVD
• Optimization form of the Funk SVD:
• If we know all entries (= contains all (u, i) pairs), we c...
SOLVE BY ALTERNATING MINIMIZATION
!12
M ≒ xm m
n
n
k
k
These can be solved by SVD!
(pseudo inverse)
ˆU<latexit sha1_base64...
WHAT IS BIASED MATRIX FACTORIZATION?
• The Biased Matrix Factorization is based on Funk SVD.
• Biased MF = Funk SVD + Bias...
OPTIMIZATION FORM OF BIASED-MF
• Therefore, Biased-MF introduce biased terms related to user and item respectively.
• is t...
THE ROLE OF BIAS TERMS IN BIASED-MF
• The bias terms serve the role of normalization.
• The bias terms serve to make the r...
TABLE OF CONTENTS
1. Netflix Prize and Winner’s Algorithm
2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009)
3...
OPTIMIZATION FORM OF FUNK SVD
• Recall: Optimization form of Funk SVD
• Probabilistic Matrix Factorization (PMF, NIPS’08) ...
PROBABILISTIC MATRIX FACTORIZATION
• Optimization form of PMF:
• To find and via MAP(Maximum A Posteriori).
• A low-rank a...
THE LOG POSTERIORI OF PMF
• The log posteriori of PMF is as follows:
• Hence, the following equation holds:
!19
arg max
ˆU...
TABLE OF CONTENTS
1. Netflix Prize and Winner’s Algorithm
2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009)
3...
WHY WE USE ALTERNATING MINIMIZATION?
• We can just use Gradient Descent with random initialization without alternating
min...
TABLE OF CONTENTS
1. Netflix Prize and Winner’s Algorithm
2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009)
3...
LOCAL MINIMA PROBLEM OF AM
• In otherwise, Jain et al(2013) showed that AM has no local minima problem.
• Noiseless case O...
THE MODIFIED ALGORITHM
!24
THE MODIFIED ALGORITHM
!25
Mini-batch
THE MODIFIED ALGORITHM
!26
Alternating Minimization
similar to Funk SVD
THE MODIFIED ALGORITHM
!27
When performing initialization using SVD + Clipping,
the local optimum found by the SVD method
...
MOTIVATION OF THE INCOHERENCE ASSUMPTION
• Consider the rank-1 matrix M:
• Let |Ω| be the number of observed entries of M....
THE INCOHERENCE ASSUMPTION
• More generally, it is hard to recover if the singular vectors of the matrix M are similar to
...
MAIN RESULTS
• Required entries: |Ω| = O((k4.5 log k) n log n)
• Required steps: O(log (1/ε))
• The EMCCO paper said that ...
COMPARISON TO NUCLEAR NORM MINIMIZATION
• Alternating Minimization
• Required entries: |Ω| = O((k4.5 log k) n log n)
• Req...
PROOF SKETCH
!32
PRELIMINARY
!33
BASE CASE
!34
INDUCTION STEP (DISTANCE)
!35
INDUCTION STEP (INCOHERENCE)
!36
TABLE OF CONTENTS
1. Netflix Prize and Winner’s Algorithm
2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009)
3...
WHY RANDOM INITIALIZATION WORKS?
• In prior paper, initialization steps matters to guarantee global optimality.
• In pract...
PRELIMINARY: MATRIX SENSING PROBLEM
• Quite similar to the matrix completion problem.
• A Low-rank version of linear regre...
NO SPURIOUS LOCAL MINIMA
• Rong et al (NIPS 2016), Matrix Completion has No Spurious Local Minimum
• Matrix Completion pro...
ANY QUESTION?
REFERENCES
[1] Koren,Yehuda, Robert Bell, and ChrisVolinsky. "Matrix factorization techniques for
recommender systems." Co...
REFERENCES
[9] Candès, Emmanuel J., and Benjamin Recht. "Exact matrix completion via convex
optimization." Foundations of ...
Sie haben dieses Dokument abgeschlossen.
Lade die Datei herunter und lese sie offline.
Nächste SlideShare
What to Upload to SlideShare
Weiter
Nächste SlideShare
What to Upload to SlideShare
Weiter
Herunterladen, um offline zu lesen und im Vollbildmodus anzuzeigen.

Teilen

Why biased matrix factorization works well?

Herunterladen, um offline zu lesen

This slide can answer why Biased Matrix Factorization works well?

Why biased matrix factorization works well?

  1. 1. Why Biased Matrix Factorization Works Well? 2018. 8. 29. JoonyoungYi joonyoung.yi@kaist.ac.kr
 Machine Learning & Intelligence Laboratory School of Computing
 Korea Advanced Institute of Science andTechnology
  2. 2. TABLE OF CONTENTS 1. Netflix Prize and Winner’s Algorithm 2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009) 3. Probabilistic Matrix Factorization (NIPS 2008) 4. Why We Use Alternating Minimization? 5. Overcoming Local Minima Problem of Alternating Minimization 6. Why Random Initialization Works?
  3. 3. TABLE OF CONTENTS 1. Netflix Prize and Winner’s Algorithm 2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009) 3. Probabilistic Matrix Factorization (NIPS 2008) 4. Why We Use Alternating Minimization? 5. Overcoming Local Minima Problem of Alternating Minimization 6. Why Random Initialization Works?
  4. 4. NETFLIX PRIZE • In October 2016, Netflix open a $1M Prize called Netflix Prize. • To improve recommendation accuracy by 10%. • They splitted the data for prize. • Training set: 100M • Test set: 1.4M • Training set contains 1.2 % of entries. • 480,000 users and 17,700 movies. • From a matrix viewpoint, training set contains only 1.2% entries of the matrix. !4 1 2 3 4 5 6 7 8 … 1 4 2 4 2 3 3 3 3 3 1 4 2 4 1 5 … 2 3 1 Users Movies 480,000 17,700
  5. 5. DATA DESCRIPTION OF NETFLIX PRIZE • Each set(training set, test set) is a set of (i, j, Mij)s. • i: user index • j: item(or movie) index • Mij: true rating of user u to item i. • : the set of (i, j) corresponding to the training set(=The set of known entries). • : the set of (i, j) corresponding to the test set(=The set of entries to predict). • m is the number of users. • n is the number of items. !5 ⌦<latexit sha1_base64="QevwxOiyXhYIQIu06UQswEZLGZ8=">AAAB7HicbVDLSgNBEOz1GeMr6tHLYBA8hV0R1FvAizcjuEkgWcLsZDYZM49lZlYIS/7BiwcVr36QN//GSbIHTSxoKKq66e6KU86M9f1vb2V1bX1js7RV3t7Z3duvHBw2jco0oSFRXOl2jA3lTNLQMstpO9UUi5jTVjy6mfqtJ6oNU/LBjlMaCTyQLGEEWyc1u3eCDnCvUvVr/gxomQQFqUKBRq/y1e0rkgkqLeHYmE7gpzbKsbaMcDopdzNDU0xGeEA7jkosqIny2bUTdOqUPkqUdiUtmqm/J3IsjBmL2HUKbIdm0ZuK/3mdzCZXUc5kmlkqyXxRknFkFZq+jvpMU2L52BFMNHO3IjLEGhPrAiq7EILFl5dJeF67rgX3F9W6X6RRgmM4gTMI4BLqcAsNCIHAIzzDK7x5ynvx3r2PeeuKV8wcwR94nz/GTI69</latexit><latexit sha1_base64="QevwxOiyXhYIQIu06UQswEZLGZ8=">AAAB7HicbVDLSgNBEOz1GeMr6tHLYBA8hV0R1FvAizcjuEkgWcLsZDYZM49lZlYIS/7BiwcVr36QN//GSbIHTSxoKKq66e6KU86M9f1vb2V1bX1js7RV3t7Z3duvHBw2jco0oSFRXOl2jA3lTNLQMstpO9UUi5jTVjy6mfqtJ6oNU/LBjlMaCTyQLGEEWyc1u3eCDnCvUvVr/gxomQQFqUKBRq/y1e0rkgkqLeHYmE7gpzbKsbaMcDopdzNDU0xGeEA7jkosqIny2bUTdOqUPkqUdiUtmqm/J3IsjBmL2HUKbIdm0ZuK/3mdzCZXUc5kmlkqyXxRknFkFZq+jvpMU2L52BFMNHO3IjLEGhPrAiq7EILFl5dJeF67rgX3F9W6X6RRgmM4gTMI4BLqcAsNCIHAIzzDK7x5ynvx3r2PeeuKV8wcwR94nz/GTI69</latexit><latexit sha1_base64="QevwxOiyXhYIQIu06UQswEZLGZ8=">AAAB7HicbVDLSgNBEOz1GeMr6tHLYBA8hV0R1FvAizcjuEkgWcLsZDYZM49lZlYIS/7BiwcVr36QN//GSbIHTSxoKKq66e6KU86M9f1vb2V1bX1js7RV3t7Z3duvHBw2jco0oSFRXOl2jA3lTNLQMstpO9UUi5jTVjy6mfqtJ6oNU/LBjlMaCTyQLGEEWyc1u3eCDnCvUvVr/gxomQQFqUKBRq/y1e0rkgkqLeHYmE7gpzbKsbaMcDopdzNDU0xGeEA7jkosqIny2bUTdOqUPkqUdiUtmqm/J3IsjBmL2HUKbIdm0ZuK/3mdzCZXUc5kmlkqyXxRknFkFZq+jvpMU2L52BFMNHO3IjLEGhPrAiq7EILFl5dJeF67rgX3F9W6X6RRgmM4gTMI4BLqcAsNCIHAIzzDK7x5ynvx3r2PeeuKV8wcwR94nz/GTI69</latexit><latexit sha1_base64="QevwxOiyXhYIQIu06UQswEZLGZ8=">AAAB7HicbVDLSgNBEOz1GeMr6tHLYBA8hV0R1FvAizcjuEkgWcLsZDYZM49lZlYIS/7BiwcVr36QN//GSbIHTSxoKKq66e6KU86M9f1vb2V1bX1js7RV3t7Z3duvHBw2jco0oSFRXOl2jA3lTNLQMstpO9UUi5jTVjy6mfqtJ6oNU/LBjlMaCTyQLGEEWyc1u3eCDnCvUvVr/gxomQQFqUKBRq/y1e0rkgkqLeHYmE7gpzbKsbaMcDopdzNDU0xGeEA7jkosqIny2bUTdOqUPkqUdiUtmqm/J3IsjBmL2HUKbIdm0ZuK/3mdzCZXUc5kmlkqyXxRknFkFZq+jvpMU2L52BFMNHO3IjLEGhPrAiq7EILFl5dJeF67rgX3F9W6X6RRgmM4gTMI4BLqcAsNCIHAIzzDK7x5ynvx3r2PeeuKV8wcwR94nz/GTI69</latexit> ⌦0 <latexit sha1_base64="AhOmQwf0oN8sVpt2aiAX3g06gEg=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbRU0lEUG8FL96sYGyhDWWznbRLd5OwuxFK6I/w4kHFq//Hm//GbZuDtj4YeLw3w8y8MBVcG9f9dkorq2vrG+XNytb2zu5edf/gUSeZYuizRCSqHVKNgsfoG24EtlOFVIYCW+HoZuq3nlBpnsQPZpxiIOkg5hFn1Fip1b2TOKCnvWrNrbszkGXiFaQGBZq96le3n7BMYmyYoFp3PDc1QU6V4UzgpNLNNKaUjegAO5bGVKIO8tm5E3JilT6JEmUrNmSm/p7IqdR6LEPbKakZ6kVvKv7ndTITXQU5j9PMYMzmi6JMEJOQ6e+kzxUyI8aWUKa4vZWwIVWUGZtQxYbgLb68TPzz+nXdu7+oNdwijTIcwTGcgQeX0IBbaIIPDEbwDK/w5qTOi/PufMxbS04xcwh/4Hz+ACiCju4=</latexit><latexit sha1_base64="AhOmQwf0oN8sVpt2aiAX3g06gEg=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbRU0lEUG8FL96sYGyhDWWznbRLd5OwuxFK6I/w4kHFq//Hm//GbZuDtj4YeLw3w8y8MBVcG9f9dkorq2vrG+XNytb2zu5edf/gUSeZYuizRCSqHVKNgsfoG24EtlOFVIYCW+HoZuq3nlBpnsQPZpxiIOkg5hFn1Fip1b2TOKCnvWrNrbszkGXiFaQGBZq96le3n7BMYmyYoFp3PDc1QU6V4UzgpNLNNKaUjegAO5bGVKIO8tm5E3JilT6JEmUrNmSm/p7IqdR6LEPbKakZ6kVvKv7ndTITXQU5j9PMYMzmi6JMEJOQ6e+kzxUyI8aWUKa4vZWwIVWUGZtQxYbgLb68TPzz+nXdu7+oNdwijTIcwTGcgQeX0IBbaIIPDEbwDK/w5qTOi/PufMxbS04xcwh/4Hz+ACiCju4=</latexit><latexit sha1_base64="AhOmQwf0oN8sVpt2aiAX3g06gEg=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbRU0lEUG8FL96sYGyhDWWznbRLd5OwuxFK6I/w4kHFq//Hm//GbZuDtj4YeLw3w8y8MBVcG9f9dkorq2vrG+XNytb2zu5edf/gUSeZYuizRCSqHVKNgsfoG24EtlOFVIYCW+HoZuq3nlBpnsQPZpxiIOkg5hFn1Fip1b2TOKCnvWrNrbszkGXiFaQGBZq96le3n7BMYmyYoFp3PDc1QU6V4UzgpNLNNKaUjegAO5bGVKIO8tm5E3JilT6JEmUrNmSm/p7IqdR6LEPbKakZ6kVvKv7ndTITXQU5j9PMYMzmi6JMEJOQ6e+kzxUyI8aWUKa4vZWwIVWUGZtQxYbgLb68TPzz+nXdu7+oNdwijTIcwTGcgQeX0IBbaIIPDEbwDK/w5qTOi/PufMxbS04xcwh/4Hz+ACiCju4=</latexit><latexit sha1_base64="AhOmQwf0oN8sVpt2aiAX3g06gEg=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbRU0lEUG8FL96sYGyhDWWznbRLd5OwuxFK6I/w4kHFq//Hm//GbZuDtj4YeLw3w8y8MBVcG9f9dkorq2vrG+XNytb2zu5edf/gUSeZYuizRCSqHVKNgsfoG24EtlOFVIYCW+HoZuq3nlBpnsQPZpxiIOkg5hFn1Fip1b2TOKCnvWrNrbszkGXiFaQGBZq96le3n7BMYmyYoFp3PDc1QU6V4UzgpNLNNKaUjegAO5bGVKIO8tm5E3JilT6JEmUrNmSm/p7IqdR6LEPbKakZ6kVvKv7ndTITXQU5j9PMYMzmi6JMEJOQ6e+kzxUyI8aWUKa4vZWwIVWUGZtQxYbgLb68TPzz+nXdu7+oNdwijTIcwTGcgQeX0IBbaIIPDEbwDK/w5qTOi/PufMxbS04xcwh/4Hz+ACiCju4=</latexit>
  6. 6. DATA DESCRIPTION OF NETFLIX PRIZE • RMSE(Root Mean Square Error) was used for measuring accuracy. • Rij: predicted rating of user i to item j. • is the cardinality of the . • Projection Operator • M, R are rating matrices. • !6 ⌦0 <latexit sha1_base64="AhOmQwf0oN8sVpt2aiAX3g06gEg=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbRU0lEUG8FL96sYGyhDWWznbRLd5OwuxFK6I/w4kHFq//Hm//GbZuDtj4YeLw3w8y8MBVcG9f9dkorq2vrG+XNytb2zu5edf/gUSeZYuizRCSqHVKNgsfoG24EtlOFVIYCW+HoZuq3nlBpnsQPZpxiIOkg5hFn1Fip1b2TOKCnvWrNrbszkGXiFaQGBZq96le3n7BMYmyYoFp3PDc1QU6V4UzgpNLNNKaUjegAO5bGVKIO8tm5E3JilT6JEmUrNmSm/p7IqdR6LEPbKakZ6kVvKv7ndTITXQU5j9PMYMzmi6JMEJOQ6e+kzxUyI8aWUKa4vZWwIVWUGZtQxYbgLb68TPzz+nXdu7+oNdwijTIcwTGcgQeX0IBbaIIPDEbwDK/w5qTOi/PufMxbS04xcwh/4Hz+ACiCju4=</latexit><latexit sha1_base64="AhOmQwf0oN8sVpt2aiAX3g06gEg=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbRU0lEUG8FL96sYGyhDWWznbRLd5OwuxFK6I/w4kHFq//Hm//GbZuDtj4YeLw3w8y8MBVcG9f9dkorq2vrG+XNytb2zu5edf/gUSeZYuizRCSqHVKNgsfoG24EtlOFVIYCW+HoZuq3nlBpnsQPZpxiIOkg5hFn1Fip1b2TOKCnvWrNrbszkGXiFaQGBZq96le3n7BMYmyYoFp3PDc1QU6V4UzgpNLNNKaUjegAO5bGVKIO8tm5E3JilT6JEmUrNmSm/p7IqdR6LEPbKakZ6kVvKv7ndTITXQU5j9PMYMzmi6JMEJOQ6e+kzxUyI8aWUKa4vZWwIVWUGZtQxYbgLb68TPzz+nXdu7+oNdwijTIcwTGcgQeX0IBbaIIPDEbwDK/w5qTOi/PufMxbS04xcwh/4Hz+ACiCju4=</latexit><latexit sha1_base64="AhOmQwf0oN8sVpt2aiAX3g06gEg=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbRU0lEUG8FL96sYGyhDWWznbRLd5OwuxFK6I/w4kHFq//Hm//GbZuDtj4YeLw3w8y8MBVcG9f9dkorq2vrG+XNytb2zu5edf/gUSeZYuizRCSqHVKNgsfoG24EtlOFVIYCW+HoZuq3nlBpnsQPZpxiIOkg5hFn1Fip1b2TOKCnvWrNrbszkGXiFaQGBZq96le3n7BMYmyYoFp3PDc1QU6V4UzgpNLNNKaUjegAO5bGVKIO8tm5E3JilT6JEmUrNmSm/p7IqdR6LEPbKakZ6kVvKv7ndTITXQU5j9PMYMzmi6JMEJOQ6e+kzxUyI8aWUKa4vZWwIVWUGZtQxYbgLb68TPzz+nXdu7+oNdwijTIcwTGcgQeX0IBbaIIPDEbwDK/w5qTOi/PufMxbS04xcwh/4Hz+ACiCju4=</latexit><latexit sha1_base64="AhOmQwf0oN8sVpt2aiAX3g06gEg=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbRU0lEUG8FL96sYGyhDWWznbRLd5OwuxFK6I/w4kHFq//Hm//GbZuDtj4YeLw3w8y8MBVcG9f9dkorq2vrG+XNytb2zu5edf/gUSeZYuizRCSqHVKNgsfoG24EtlOFVIYCW+HoZuq3nlBpnsQPZpxiIOkg5hFn1Fip1b2TOKCnvWrNrbszkGXiFaQGBZq96le3n7BMYmyYoFp3PDc1QU6V4UzgpNLNNKaUjegAO5bGVKIO8tm5E3JilT6JEmUrNmSm/p7IqdR6LEPbKakZ6kVvKv7ndTITXQU5j9PMYMzmi6JMEJOQ6e+kzxUyI8aWUKa4vZWwIVWUGZtQxYbgLb68TPzz+nXdu7+oNdwijTIcwTGcgQeX0IBbaIIPDEbwDK/w5qTOi/PufMxbS04xcwh/4Hz+ACiCju4=</latexit> |⌦0 |<latexit sha1_base64="lqftWvWVOn1LrUfnslhQk2K3r3M=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBbRU0lEUG8FL96sYGwhDWWz3bRLd7NhdyOUtD/DiwcVr/4bb/4bt20O2vpg4PHeDDPzopQzbVz32ymtrK6tb5Q3K1vbO7t71f2DRy0zRahPJJeqHWFNOUuob5jhtJ0qikXEaSsa3kz91hNVmsnkwYxSGgrcT1jMCDZWCsaocydoH5+icbdac+vuDGiZeAWpQYFmt/rV6UmSCZoYwrHWgeemJsyxMoxwOql0Mk1TTIa4TwNLEyyoDvPZyRN0YpUeiqWylRg0U39P5FhoPRKR7RTYDPSiNxX/84LMxFdhzpI0MzQh80VxxpGRaPo/6jFFieEjSzBRzN6KyAArTIxNqWJD8BZfXib+ef267t1f1BpukUYZjuAYzsCDS2jALTTBBwISnuEV3hzjvDjvzse8teQUM4fwB87nD58AkE4=</latexit><latexit sha1_base64="lqftWvWVOn1LrUfnslhQk2K3r3M=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBbRU0lEUG8FL96sYGwhDWWz3bRLd7NhdyOUtD/DiwcVr/4bb/4bt20O2vpg4PHeDDPzopQzbVz32ymtrK6tb5Q3K1vbO7t71f2DRy0zRahPJJeqHWFNOUuob5jhtJ0qikXEaSsa3kz91hNVmsnkwYxSGgrcT1jMCDZWCsaocydoH5+icbdac+vuDGiZeAWpQYFmt/rV6UmSCZoYwrHWgeemJsyxMoxwOql0Mk1TTIa4TwNLEyyoDvPZyRN0YpUeiqWylRg0U39P5FhoPRKR7RTYDPSiNxX/84LMxFdhzpI0MzQh80VxxpGRaPo/6jFFieEjSzBRzN6KyAArTIxNqWJD8BZfXib+ef267t1f1BpukUYZjuAYzsCDS2jALTTBBwISnuEV3hzjvDjvzse8teQUM4fwB87nD58AkE4=</latexit><latexit sha1_base64="lqftWvWVOn1LrUfnslhQk2K3r3M=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBbRU0lEUG8FL96sYGwhDWWz3bRLd7NhdyOUtD/DiwcVr/4bb/4bt20O2vpg4PHeDDPzopQzbVz32ymtrK6tb5Q3K1vbO7t71f2DRy0zRahPJJeqHWFNOUuob5jhtJ0qikXEaSsa3kz91hNVmsnkwYxSGgrcT1jMCDZWCsaocydoH5+icbdac+vuDGiZeAWpQYFmt/rV6UmSCZoYwrHWgeemJsyxMoxwOql0Mk1TTIa4TwNLEyyoDvPZyRN0YpUeiqWylRg0U39P5FhoPRKR7RTYDPSiNxX/84LMxFdhzpI0MzQh80VxxpGRaPo/6jFFieEjSzBRzN6KyAArTIxNqWJD8BZfXib+ef267t1f1BpukUYZjuAYzsCDS2jALTTBBwISnuEV3hzjvDjvzse8teQUM4fwB87nD58AkE4=</latexit><latexit sha1_base64="lqftWvWVOn1LrUfnslhQk2K3r3M=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBbRU0lEUG8FL96sYGwhDWWz3bRLd7NhdyOUtD/DiwcVr/4bb/4bt20O2vpg4PHeDDPzopQzbVz32ymtrK6tb5Q3K1vbO7t71f2DRy0zRahPJJeqHWFNOUuob5jhtJ0qikXEaSsa3kz91hNVmsnkwYxSGgrcT1jMCDZWCsaocydoH5+icbdac+vuDGiZeAWpQYFmt/rV6UmSCZoYwrHWgeemJsyxMoxwOql0Mk1TTIa4TwNLEyyoDvPZyRN0YpUeiqWylRg0U39P5FhoPRKR7RTYDPSiNxX/84LMxFdhzpI0MzQh80VxxpGRaPo/6jFFieEjSzBRzN6KyAArTIxNqWJD8BZfXib+ef267t1f1BpukUYZjuAYzsCDS2jALTTBBwISnuEV3hzjvDjvzse8teQUM4fwB87nD58AkE4=</latexit> train RMSE = s X ⌦ (Mij Rij)2 |⌦| = kP⌦(M R)kF p ⌦ test RMSE = s X ⌦0 (Mij Rij)2 |⌦0| = kP⌦0 (M R)kF p ⌦0 <latexit sha1_base64="/9IZniw0dk7d9UcO15y/+IJjYXs=">AAADHXiclVLLbhMxFPUMrxJeKSzZXBFB0gXRTBWJsqhUCYHYRIRAaKU4jDyOk5rOq/YdROT6S9jwK2xYAGLBBvE3OJkBQQsIrmXp6Nx77j1+xEUiNQbBV88/dfrM2XNr5xsXLl66fKW5fvWZzkvFxYjnSa72YqZFIjMxQomJ2CuUYGmciN344N4yv/tSKC3z7CkuCjFJ2TyTM8kZOipa93oUxSs0qJjMYNh/ct/CrW2g+lChobpMI0MfpWLOLBgwnX5k5AsLt8EMbQU3nm9aoLmb4QrgCKpqB+xqbYMxQI9g8KNPp+/kww1HRg8sfJdWA2u1ta4lbdTOhMa/GWv/n7P2n621/8Vb25lrRM1W0A1WASdBWIMWqWMQNT/Tac7LVGTIE6b1OAwKnBimUPJE2AYttSgYP2BzMXYwY6nQE7N6Xgs3HTOFWa7czhBW7M8Kw1KtF2nsKlOG+/p4bkn+LjcucbY1MTIrShQZrwbNygQwh+VfgalUgmOycIBxJZ1X4PtMMY7uRy0vITx+5JNgtNm92w0f91o7vfo21sh1coN0SEjukB3ykAzIiHDvtffWe+998N/47/yP/qeq1PdqzTXyS/hfvgGzoveX</latexit><latexit sha1_base64="/9IZniw0dk7d9UcO15y/+IJjYXs=">AAADHXiclVLLbhMxFPUMrxJeKSzZXBFB0gXRTBWJsqhUCYHYRIRAaKU4jDyOk5rOq/YdROT6S9jwK2xYAGLBBvE3OJkBQQsIrmXp6Nx77j1+xEUiNQbBV88/dfrM2XNr5xsXLl66fKW5fvWZzkvFxYjnSa72YqZFIjMxQomJ2CuUYGmciN344N4yv/tSKC3z7CkuCjFJ2TyTM8kZOipa93oUxSs0qJjMYNh/ct/CrW2g+lChobpMI0MfpWLOLBgwnX5k5AsLt8EMbQU3nm9aoLmb4QrgCKpqB+xqbYMxQI9g8KNPp+/kww1HRg8sfJdWA2u1ta4lbdTOhMa/GWv/n7P2n621/8Vb25lrRM1W0A1WASdBWIMWqWMQNT/Tac7LVGTIE6b1OAwKnBimUPJE2AYttSgYP2BzMXYwY6nQE7N6Xgs3HTOFWa7czhBW7M8Kw1KtF2nsKlOG+/p4bkn+LjcucbY1MTIrShQZrwbNygQwh+VfgalUgmOycIBxJZ1X4PtMMY7uRy0vITx+5JNgtNm92w0f91o7vfo21sh1coN0SEjukB3ykAzIiHDvtffWe+998N/47/yP/qeq1PdqzTXyS/hfvgGzoveX</latexit><latexit sha1_base64="/9IZniw0dk7d9UcO15y/+IJjYXs=">AAADHXiclVLLbhMxFPUMrxJeKSzZXBFB0gXRTBWJsqhUCYHYRIRAaKU4jDyOk5rOq/YdROT6S9jwK2xYAGLBBvE3OJkBQQsIrmXp6Nx77j1+xEUiNQbBV88/dfrM2XNr5xsXLl66fKW5fvWZzkvFxYjnSa72YqZFIjMxQomJ2CuUYGmciN344N4yv/tSKC3z7CkuCjFJ2TyTM8kZOipa93oUxSs0qJjMYNh/ct/CrW2g+lChobpMI0MfpWLOLBgwnX5k5AsLt8EMbQU3nm9aoLmb4QrgCKpqB+xqbYMxQI9g8KNPp+/kww1HRg8sfJdWA2u1ta4lbdTOhMa/GWv/n7P2n621/8Vb25lrRM1W0A1WASdBWIMWqWMQNT/Tac7LVGTIE6b1OAwKnBimUPJE2AYttSgYP2BzMXYwY6nQE7N6Xgs3HTOFWa7czhBW7M8Kw1KtF2nsKlOG+/p4bkn+LjcucbY1MTIrShQZrwbNygQwh+VfgalUgmOycIBxJZ1X4PtMMY7uRy0vITx+5JNgtNm92w0f91o7vfo21sh1coN0SEjukB3ykAzIiHDvtffWe+998N/47/yP/qeq1PdqzTXyS/hfvgGzoveX</latexit><latexit sha1_base64="/9IZniw0dk7d9UcO15y/+IJjYXs=">AAADHXiclVLLbhMxFPUMrxJeKSzZXBFB0gXRTBWJsqhUCYHYRIRAaKU4jDyOk5rOq/YdROT6S9jwK2xYAGLBBvE3OJkBQQsIrmXp6Nx77j1+xEUiNQbBV88/dfrM2XNr5xsXLl66fKW5fvWZzkvFxYjnSa72YqZFIjMxQomJ2CuUYGmciN344N4yv/tSKC3z7CkuCjFJ2TyTM8kZOipa93oUxSs0qJjMYNh/ct/CrW2g+lChobpMI0MfpWLOLBgwnX5k5AsLt8EMbQU3nm9aoLmb4QrgCKpqB+xqbYMxQI9g8KNPp+/kww1HRg8sfJdWA2u1ta4lbdTOhMa/GWv/n7P2n621/8Vb25lrRM1W0A1WASdBWIMWqWMQNT/Tac7LVGTIE6b1OAwKnBimUPJE2AYttSgYP2BzMXYwY6nQE7N6Xgs3HTOFWa7czhBW7M8Kw1KtF2nsKlOG+/p4bkn+LjcucbY1MTIrShQZrwbNygQwh+VfgalUgmOycIBxJZ1X4PtMMY7uRy0vITx+5JNgtNm92w0f91o7vfo21sh1coN0SEjukB3ykAzIiHDvtffWe+998N/47/yP/qeq1PdqzTXyS/hfvgGzoveX</latexit> P⌦(M) = ⇢ Mij if (i, j) 2 ⌦, 0 otherwise.<latexit sha1_base64="h8sB+vFxh04Y/hYuefLrNWXXg2U=">AAACd3icbVFNa9wwEJXdr3T7tUmPuQzdJmwgGDsUmh4CgV56Cd1AtwmsFiNrx14lsmwkue0i/Bv633rrD+mlp8q7JrRJBwSPN+89STNZLYWxcfwzCO/df/Dw0dbjwZOnz56/GG7vfDZVozlOeSUrfZkxg1IonFphJV7WGlmZSbzIrt93/YsvqI2o1Ce7qnFeskKJXHBmPZUOv09SRz+WWLAWxmcHcAJUYm6pGwDQDAuhHNOarVonZes5gLPUiasWYB9omVXfnMjBW8UhXB0AFQo2aYdA6Voe3wgru0T9VRiMuiCKatFHD6gWxdJGnSUdjuIoXhfcBUkPRqSvSTr8QRcVb0pUlktmzCyJazv3yVZwiT67MVgzfs0KnHmoWIlm7taDa2HPMwvIK+2PsrBm/3Y4VhqzKjOvLJldmtu9jvxfb9bY/HjuhKobi4pvLsobCbaCbguwEBq5lSsPGNfCvxX4kmnGrd9VN4Tk9pfvgulR9C5Kzt+MTo/6aWyRXfKKjElC3pJT8oFMyJRw8ivYDV4He8HvEML9cLyRhkHveUn+qTD5A4QtuSk=</latexit><latexit sha1_base64="h8sB+vFxh04Y/hYuefLrNWXXg2U=">AAACd3icbVFNa9wwEJXdr3T7tUmPuQzdJmwgGDsUmh4CgV56Cd1AtwmsFiNrx14lsmwkue0i/Bv633rrD+mlp8q7JrRJBwSPN+89STNZLYWxcfwzCO/df/Dw0dbjwZOnz56/GG7vfDZVozlOeSUrfZkxg1IonFphJV7WGlmZSbzIrt93/YsvqI2o1Ce7qnFeskKJXHBmPZUOv09SRz+WWLAWxmcHcAJUYm6pGwDQDAuhHNOarVonZes5gLPUiasWYB9omVXfnMjBW8UhXB0AFQo2aYdA6Voe3wgru0T9VRiMuiCKatFHD6gWxdJGnSUdjuIoXhfcBUkPRqSvSTr8QRcVb0pUlktmzCyJazv3yVZwiT67MVgzfs0KnHmoWIlm7taDa2HPMwvIK+2PsrBm/3Y4VhqzKjOvLJldmtu9jvxfb9bY/HjuhKobi4pvLsobCbaCbguwEBq5lSsPGNfCvxX4kmnGrd9VN4Tk9pfvgulR9C5Kzt+MTo/6aWyRXfKKjElC3pJT8oFMyJRw8ivYDV4He8HvEML9cLyRhkHveUn+qTD5A4QtuSk=</latexit><latexit sha1_base64="h8sB+vFxh04Y/hYuefLrNWXXg2U=">AAACd3icbVFNa9wwEJXdr3T7tUmPuQzdJmwgGDsUmh4CgV56Cd1AtwmsFiNrx14lsmwkue0i/Bv633rrD+mlp8q7JrRJBwSPN+89STNZLYWxcfwzCO/df/Dw0dbjwZOnz56/GG7vfDZVozlOeSUrfZkxg1IonFphJV7WGlmZSbzIrt93/YsvqI2o1Ce7qnFeskKJXHBmPZUOv09SRz+WWLAWxmcHcAJUYm6pGwDQDAuhHNOarVonZes5gLPUiasWYB9omVXfnMjBW8UhXB0AFQo2aYdA6Voe3wgru0T9VRiMuiCKatFHD6gWxdJGnSUdjuIoXhfcBUkPRqSvSTr8QRcVb0pUlktmzCyJazv3yVZwiT67MVgzfs0KnHmoWIlm7taDa2HPMwvIK+2PsrBm/3Y4VhqzKjOvLJldmtu9jvxfb9bY/HjuhKobi4pvLsobCbaCbguwEBq5lSsPGNfCvxX4kmnGrd9VN4Tk9pfvgulR9C5Kzt+MTo/6aWyRXfKKjElC3pJT8oFMyJRw8ivYDV4He8HvEML9cLyRhkHveUn+qTD5A4QtuSk=</latexit><latexit sha1_base64="h8sB+vFxh04Y/hYuefLrNWXXg2U=">AAACd3icbVFNa9wwEJXdr3T7tUmPuQzdJmwgGDsUmh4CgV56Cd1AtwmsFiNrx14lsmwkue0i/Bv633rrD+mlp8q7JrRJBwSPN+89STNZLYWxcfwzCO/df/Dw0dbjwZOnz56/GG7vfDZVozlOeSUrfZkxg1IonFphJV7WGlmZSbzIrt93/YsvqI2o1Ce7qnFeskKJXHBmPZUOv09SRz+WWLAWxmcHcAJUYm6pGwDQDAuhHNOarVonZes5gLPUiasWYB9omVXfnMjBW8UhXB0AFQo2aYdA6Voe3wgru0T9VRiMuiCKatFHD6gWxdJGnSUdjuIoXhfcBUkPRqSvSTr8QRcVb0pUlktmzCyJazv3yVZwiT67MVgzfs0KnHmoWIlm7taDa2HPMwvIK+2PsrBm/3Y4VhqzKjOvLJldmtu9jvxfb9bY/HjuhKobi4pvLsobCbaCbguwEBq5lSsPGNfCvxX4kmnGrd9VN4Tk9pfvgulR9C5Kzt+MTo/6aWyRXfKKjElC3pJT8oFMyJRw8ivYDV4He8HvEML9cLyRhkHveUn+qTD5A4QtuSk=</latexit> M, R 2 Rm⇥n <latexit sha1_base64="9fnhV6oxrlHnJGzB7Tuj95171cQ=">AAACCHicbVBNS8NAEN34WetX1KOXxSJ4kJKIoN4KXrwItRhbaGLZbDft0t1N2N0IJeTqxb/ixYOKV3+CN/+NmzYHbX0w8Hhvhpl5YcKo0o7zbS0sLi2vrFbWqusbm1vb9s7unYpTiYmHYxbLTogUYVQQT1PNSCeRBPGQkXY4uiz89gORisbiVo8TEnA0EDSiGGkj9Wx4fQxb0KcC+hzpYRhmrfw+49DXlBMFRd6za07dmQDOE7ckNVCi2bO//H6MU06Exgwp1XWdRAcZkppiRvKqnyqSIDxCA9I1VCCzJ8gmn+Tw0Ch9GMXSlNBwov6eyBBXasxD01mcq2a9QvzP66Y6Og8yKpJUE4Gni6KUQR3DIhbYp5JgzcaGICypuRXiIZIIaxNe1YTgzr48T7yT+kXdvTmtNZwyjQrYBwfgCLjgDDTAFWgCD2DwCJ7BK3iznqwX6936mLYuWOXMHvgD6/MHZxCZDg==</latexit><latexit sha1_base64="9fnhV6oxrlHnJGzB7Tuj95171cQ=">AAACCHicbVBNS8NAEN34WetX1KOXxSJ4kJKIoN4KXrwItRhbaGLZbDft0t1N2N0IJeTqxb/ixYOKV3+CN/+NmzYHbX0w8Hhvhpl5YcKo0o7zbS0sLi2vrFbWqusbm1vb9s7unYpTiYmHYxbLTogUYVQQT1PNSCeRBPGQkXY4uiz89gORisbiVo8TEnA0EDSiGGkj9Wx4fQxb0KcC+hzpYRhmrfw+49DXlBMFRd6za07dmQDOE7ckNVCi2bO//H6MU06Exgwp1XWdRAcZkppiRvKqnyqSIDxCA9I1VCCzJ8gmn+Tw0Ch9GMXSlNBwov6eyBBXasxD01mcq2a9QvzP66Y6Og8yKpJUE4Gni6KUQR3DIhbYp5JgzcaGICypuRXiIZIIaxNe1YTgzr48T7yT+kXdvTmtNZwyjQrYBwfgCLjgDDTAFWgCD2DwCJ7BK3iznqwX6936mLYuWOXMHvgD6/MHZxCZDg==</latexit><latexit sha1_base64="9fnhV6oxrlHnJGzB7Tuj95171cQ=">AAACCHicbVBNS8NAEN34WetX1KOXxSJ4kJKIoN4KXrwItRhbaGLZbDft0t1N2N0IJeTqxb/ixYOKV3+CN/+NmzYHbX0w8Hhvhpl5YcKo0o7zbS0sLi2vrFbWqusbm1vb9s7unYpTiYmHYxbLTogUYVQQT1PNSCeRBPGQkXY4uiz89gORisbiVo8TEnA0EDSiGGkj9Wx4fQxb0KcC+hzpYRhmrfw+49DXlBMFRd6za07dmQDOE7ckNVCi2bO//H6MU06Exgwp1XWdRAcZkppiRvKqnyqSIDxCA9I1VCCzJ8gmn+Tw0Ch9GMXSlNBwov6eyBBXasxD01mcq2a9QvzP66Y6Og8yKpJUE4Gni6KUQR3DIhbYp5JgzcaGICypuRXiIZIIaxNe1YTgzr48T7yT+kXdvTmtNZwyjQrYBwfgCLjgDDTAFWgCD2DwCJ7BK3iznqwX6936mLYuWOXMHvgD6/MHZxCZDg==</latexit><latexit sha1_base64="9fnhV6oxrlHnJGzB7Tuj95171cQ=">AAACCHicbVBNS8NAEN34WetX1KOXxSJ4kJKIoN4KXrwItRhbaGLZbDft0t1N2N0IJeTqxb/ixYOKV3+CN/+NmzYHbX0w8Hhvhpl5YcKo0o7zbS0sLi2vrFbWqusbm1vb9s7unYpTiYmHYxbLTogUYVQQT1PNSCeRBPGQkXY4uiz89gORisbiVo8TEnA0EDSiGGkj9Wx4fQxb0KcC+hzpYRhmrfw+49DXlBMFRd6za07dmQDOE7ckNVCi2bO//H6MU06Exgwp1XWdRAcZkppiRvKqnyqSIDxCA9I1VCCzJ8gmn+Tw0Ch9GMXSlNBwov6eyBBXasxD01mcq2a9QvzP66Y6Og8yKpJUE4Gni6KUQR3DIhbYp5JgzcaGICypuRXiIZIIaxNe1YTgzr48T7yT+kXdvTmtNZwyjQrYBwfgCLjgDDTAFWgCD2DwCJ7BK3iznqwX6936mLYuWOXMHvgD6/MHZxCZDg==</latexit>
  7. 7. RMSE on Test Set RMSE OF NETFLIX, PRIZE WINNER AND BIASED-MF • RMSE of Cinematch (Original Netflix Recommendation Engine): 0.9514 • RMSE of Winner: 0.8553 (10% enhancement) • They ensembled a lot of algorithms. • RMSE of Biased-MF ONLY: 0.8799 (7.5% enhancement) • Some materials said 6% enhancement only by Biased-MF. • RMSE of Biased-MF(with tuned hyper-parameters): 0.844 (11.3% enhancement) • From the AutoRec Paper in WWW’15. !7
  8. 8. BIASED-MF IS FAST AND WORKS WELL • Many Algorithms are presented after Biased-MF. • Biased-MF is the fastest algorithm among them. • The convex relaxation(nuclear norm minimization) algorithm is slower than Biased-MF. • Biased-MF works quite well even these days. • And, the some of algorithms with good performance are based on Biased-MF. • SMA and MRMA are ensemble algorithms based on Biased-MF. • LLoRMA and ABCF are modification of Biased-MF. !8 Movielens 1M Movielens 10M Netflix(100M) Biased MF(IEEE’09) 0.845 0.803 0.844 LLoRMA(ICML’13) 0.8333 0.7815 0.8337 AutoRec(WWW’15) 0.831 0.782 0.823 CF-NADE(ICML ’16) 0.829 0.771 0.803 SMA(ICML’16) - 0.7682 0.8036 ABCF(Neurocomputing’18) 0.836 0.766 0.795 MRMA(NIPS’17) - 0.7634 0.7973
  9. 9. TABLE OF CONTENTS 1. Netflix Prize and Winner’s Algorithm 2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009) 3. Probabilistic Matrix Factorization (NIPS 2008) 4. Why We Use Alternating Minimization? 5. Overcoming Local Minima Problem of Alternating Minimization 6. Why Random Initialization Works?
  10. 10. FUNK SVD(SINGULAR VECTOR DECOMPOSITION) • Before introducing Biased-MF, Let’s see Funk SVD(suggested by Funk). • Goal of Netflix Prize: Minimize RMSE • There is no way to solve this optimization form without any assumption. • In rating recommendation, it is natural to assume the matrix is low-rank. • New Optimization Form with Low-rank Assumption: • p and q are low-rank matrices: • To avoid overfitting: !10 ˆrui<latexit sha1_base64="ExB9hem06ilaSw+uGS5343B2V7A=">AAAB8nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoN4CXjxGcE0gu4TZySQZMvtgpkcIy/6GFw8qXv0ab/6Nk2QPmljQUFR1090VZVJodN1vp7K2vrG5Vd2u7ezu7R/UD48edWoU4z5LZaq6EdVcioT7KFDybqY4jSPJO9HkduZ3nrjSIk0ecJrxMKajRAwFo2ilIBhTzFXRz40o+vWG23TnIKvEK0kDSrT79a9gkDIT8wSZpFr3PDfDMKcKBZO8qAVG84yyCR3xnqUJjbkO8/nNBTmzyoAMU2UrQTJXf0/kNNZ6Gke2M6Y41sveTPzP6xkcXoe5SDKDPGGLRUMjCaZkFgAZCMUZyqkllClhbyVsTBVlaGOq2RC85ZdXiX/RvGl695eNllumUYUTOIVz8OAKWnAHbfCBQQbP8ApvjnFenHfnY9FaccqZY/gD5/MHL/KR7A==</latexit><latexit sha1_base64="ExB9hem06ilaSw+uGS5343B2V7A=">AAAB8nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoN4CXjxGcE0gu4TZySQZMvtgpkcIy/6GFw8qXv0ab/6Nk2QPmljQUFR1090VZVJodN1vp7K2vrG5Vd2u7ezu7R/UD48edWoU4z5LZaq6EdVcioT7KFDybqY4jSPJO9HkduZ3nrjSIk0ecJrxMKajRAwFo2ilIBhTzFXRz40o+vWG23TnIKvEK0kDSrT79a9gkDIT8wSZpFr3PDfDMKcKBZO8qAVG84yyCR3xnqUJjbkO8/nNBTmzyoAMU2UrQTJXf0/kNNZ6Gke2M6Y41sveTPzP6xkcXoe5SDKDPGGLRUMjCaZkFgAZCMUZyqkllClhbyVsTBVlaGOq2RC85ZdXiX/RvGl695eNllumUYUTOIVz8OAKWnAHbfCBQQbP8ApvjnFenHfnY9FaccqZY/gD5/MHL/KR7A==</latexit><latexit sha1_base64="ExB9hem06ilaSw+uGS5343B2V7A=">AAAB8nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoN4CXjxGcE0gu4TZySQZMvtgpkcIy/6GFw8qXv0ab/6Nk2QPmljQUFR1090VZVJodN1vp7K2vrG5Vd2u7ezu7R/UD48edWoU4z5LZaq6EdVcioT7KFDybqY4jSPJO9HkduZ3nrjSIk0ecJrxMKajRAwFo2ilIBhTzFXRz40o+vWG23TnIKvEK0kDSrT79a9gkDIT8wSZpFr3PDfDMKcKBZO8qAVG84yyCR3xnqUJjbkO8/nNBTmzyoAMU2UrQTJXf0/kNNZ6Gke2M6Y41sveTPzP6xkcXoe5SDKDPGGLRUMjCaZkFgAZCMUZyqkllClhbyVsTBVlaGOq2RC85ZdXiX/RvGl695eNllumUYUTOIVz8OAKWnAHbfCBQQbP8ApvjnFenHfnY9FaccqZY/gD5/MHL/KR7A==</latexit><latexit sha1_base64="ExB9hem06ilaSw+uGS5343B2V7A=">AAAB8nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoN4CXjxGcE0gu4TZySQZMvtgpkcIy/6GFw8qXv0ab/6Nk2QPmljQUFR1090VZVJodN1vp7K2vrG5Vd2u7ezu7R/UD48edWoU4z5LZaq6EdVcioT7KFDybqY4jSPJO9HkduZ3nrjSIk0ecJrxMKajRAwFo2ilIBhTzFXRz40o+vWG23TnIKvEK0kDSrT79a9gkDIT8wSZpFr3PDfDMKcKBZO8qAVG84yyCR3xnqUJjbkO8/nNBTmzyoAMU2UrQTJXf0/kNNZ6Gke2M6Y41sveTPzP6xkcXoe5SDKDPGGLRUMjCaZkFgAZCMUZyqkllClhbyVsTBVlaGOq2RC85ZdXiX/RvGl695eNllumUYUTOIVz8OAKWnAHbfCBQQbP8ApvjnFenHfnY9FaccqZY/gD5/MHL/KR7A==</latexit> arg min R v u u t X (i,j)2⌦ (Mij Rij)2 |⌦| = arg min R X (i,j)2⌦ (Mij Rij)2 <latexit sha1_base64="CW8tcQThzWoM9yMmC0TflQgxgI4=">AAACknicdVHRTtswFHUyxrpubGFoT7xcUW0qEquSahqbpmlFvPAAgqEVkJoSOa5bTG0nsx2kyuSH+Bze+BvcNg8UxpUsH597zr32dZpzpk0Y3nn+i6WXy69qr+tv3q68ex+sfjjRWaEI7ZKMZ+osxZpyJmnXMMPpWa4oFimnp+l4d5o/vaJKs0z+NZOc9gUeSTZkBBtHJcHNZ4ixGkEsmEzscQmx/qeMdVshEttkW3C5CTGTEB8KOsIlWLDNg8SyyxIAvoCzzA+b521nzlwvJ7mu5HBdgtOV8GuhC5TP118sf/ygeBI0wlY4C3gKogo0UBVHSXAbDzJSCCoN4VjrXhTmpm+xMoxwWtbjQtMckzEe0Z6DEguq+3Y20xI+OWYAw0y5JQ3M2IcOi4XWE5E6pcDmQj/OTcn/5XqFGX7vWybzwlBJ5o2GBQeTwfSDYMAUJYZPHMBEMXdXIBdYYWLcN9bdEKLHT34Kuu3Wj1b052uj066mUUPraAM1UYS2UQftoSPURcQLvG/eb6/jf/R/+jv+7lzqe5VnDS2Ev38P3HjB6g==</latexit><latexit sha1_base64="CW8tcQThzWoM9yMmC0TflQgxgI4=">AAACknicdVHRTtswFHUyxrpubGFoT7xcUW0qEquSahqbpmlFvPAAgqEVkJoSOa5bTG0nsx2kyuSH+Bze+BvcNg8UxpUsH597zr32dZpzpk0Y3nn+i6WXy69qr+tv3q68ex+sfjjRWaEI7ZKMZ+osxZpyJmnXMMPpWa4oFimnp+l4d5o/vaJKs0z+NZOc9gUeSTZkBBtHJcHNZ4ixGkEsmEzscQmx/qeMdVshEttkW3C5CTGTEB8KOsIlWLDNg8SyyxIAvoCzzA+b521nzlwvJ7mu5HBdgtOV8GuhC5TP118sf/ygeBI0wlY4C3gKogo0UBVHSXAbDzJSCCoN4VjrXhTmpm+xMoxwWtbjQtMckzEe0Z6DEguq+3Y20xI+OWYAw0y5JQ3M2IcOi4XWE5E6pcDmQj/OTcn/5XqFGX7vWybzwlBJ5o2GBQeTwfSDYMAUJYZPHMBEMXdXIBdYYWLcN9bdEKLHT34Kuu3Wj1b052uj066mUUPraAM1UYS2UQftoSPURcQLvG/eb6/jf/R/+jv+7lzqe5VnDS2Ev38P3HjB6g==</latexit><latexit sha1_base64="CW8tcQThzWoM9yMmC0TflQgxgI4=">AAACknicdVHRTtswFHUyxrpubGFoT7xcUW0qEquSahqbpmlFvPAAgqEVkJoSOa5bTG0nsx2kyuSH+Bze+BvcNg8UxpUsH597zr32dZpzpk0Y3nn+i6WXy69qr+tv3q68ex+sfjjRWaEI7ZKMZ+osxZpyJmnXMMPpWa4oFimnp+l4d5o/vaJKs0z+NZOc9gUeSTZkBBtHJcHNZ4ixGkEsmEzscQmx/qeMdVshEttkW3C5CTGTEB8KOsIlWLDNg8SyyxIAvoCzzA+b521nzlwvJ7mu5HBdgtOV8GuhC5TP118sf/ygeBI0wlY4C3gKogo0UBVHSXAbDzJSCCoN4VjrXhTmpm+xMoxwWtbjQtMckzEe0Z6DEguq+3Y20xI+OWYAw0y5JQ3M2IcOi4XWE5E6pcDmQj/OTcn/5XqFGX7vWybzwlBJ5o2GBQeTwfSDYMAUJYZPHMBEMXdXIBdYYWLcN9bdEKLHT34Kuu3Wj1b052uj066mUUPraAM1UYS2UQftoSPURcQLvG/eb6/jf/R/+jv+7lzqe5VnDS2Ev38P3HjB6g==</latexit><latexit sha1_base64="CW8tcQThzWoM9yMmC0TflQgxgI4=">AAACknicdVHRTtswFHUyxrpubGFoT7xcUW0qEquSahqbpmlFvPAAgqEVkJoSOa5bTG0nsx2kyuSH+Bze+BvcNg8UxpUsH597zr32dZpzpk0Y3nn+i6WXy69qr+tv3q68ex+sfjjRWaEI7ZKMZ+osxZpyJmnXMMPpWa4oFimnp+l4d5o/vaJKs0z+NZOc9gUeSTZkBBtHJcHNZ4ixGkEsmEzscQmx/qeMdVshEttkW3C5CTGTEB8KOsIlWLDNg8SyyxIAvoCzzA+b521nzlwvJ7mu5HBdgtOV8GuhC5TP118sf/ygeBI0wlY4C3gKogo0UBVHSXAbDzJSCCoN4VjrXhTmpm+xMoxwWtbjQtMckzEe0Z6DEguq+3Y20xI+OWYAw0y5JQ3M2IcOi4XWE5E6pcDmQj/OTcn/5XqFGX7vWybzwlBJ5o2GBQeTwfSDYMAUJYZPHMBEMXdXIBdYYWLcN9bdEKLHT34Kuu3Wj1b052uj066mUUPraAM1UYS2UQftoSPURcQLvG/eb6/jf/R/+jv+7lzqe5VnDS2Ev38P3HjB6g==</latexit> arg min ˆU, ˆV X (i,j)2⌦ (Mij ˆUi ˆV † j )2 <latexit sha1_base64="sYlUpp/x2K67VgHp7EuvG0Zqvg8=">AAACUnicbVJdaxQxFM2sX3WtdrWPvlxchC3UZaYU1LeCL76IFZy2sNkOd7LZ2WyTzJBkhCXkP0qhD/4RX/qg2e0I2nohcHLOPdzkJGUjhXVp+iPp3bv/4OGjrcf9J9tPn+0Mnr84sXVrGM9ZLWtzVqLlUmieO+EkP2sMR1VKflpefFjrp9+4saLWX92q4VOFlRZzwdBFqhgsKZoKqBK68HSBzudhHzbgJEAAaltV+JHYh+UeUKGBfla8wgDgR/5TKLxYBngDnTNuwx9zsTz3dIZVxU2AvfODAMVgmI7TTcFdkHVgSLo6LgaXdFazVnHtmERrJ1nauKlH4wSTPPRpa3mD7AIrPolQo+J26jeZBHgdmRnMaxOXdrBh/3Z4VNauVBk7FbqFva2tyf9pk9bN30290E3ruGY3g+atBFfDOmCYCcOZk6sIkBkRzwpsgQaZi8/QjyFkt698F+QH4/fj7Mvh8OiwS2OLvCSvyIhk5C05Ih/JMckJI9/JT/IrIclVct2Lv+SmtZd0nl3yT/W2fwM5kLKW</latexit><latexit sha1_base64="sYlUpp/x2K67VgHp7EuvG0Zqvg8=">AAACUnicbVJdaxQxFM2sX3WtdrWPvlxchC3UZaYU1LeCL76IFZy2sNkOd7LZ2WyTzJBkhCXkP0qhD/4RX/qg2e0I2nohcHLOPdzkJGUjhXVp+iPp3bv/4OGjrcf9J9tPn+0Mnr84sXVrGM9ZLWtzVqLlUmieO+EkP2sMR1VKflpefFjrp9+4saLWX92q4VOFlRZzwdBFqhgsKZoKqBK68HSBzudhHzbgJEAAaltV+JHYh+UeUKGBfla8wgDgR/5TKLxYBngDnTNuwx9zsTz3dIZVxU2AvfODAMVgmI7TTcFdkHVgSLo6LgaXdFazVnHtmERrJ1nauKlH4wSTPPRpa3mD7AIrPolQo+J26jeZBHgdmRnMaxOXdrBh/3Z4VNauVBk7FbqFva2tyf9pk9bN30290E3ruGY3g+atBFfDOmCYCcOZk6sIkBkRzwpsgQaZi8/QjyFkt698F+QH4/fj7Mvh8OiwS2OLvCSvyIhk5C05Ih/JMckJI9/JT/IrIclVct2Lv+SmtZd0nl3yT/W2fwM5kLKW</latexit><latexit sha1_base64="sYlUpp/x2K67VgHp7EuvG0Zqvg8=">AAACUnicbVJdaxQxFM2sX3WtdrWPvlxchC3UZaYU1LeCL76IFZy2sNkOd7LZ2WyTzJBkhCXkP0qhD/4RX/qg2e0I2nohcHLOPdzkJGUjhXVp+iPp3bv/4OGjrcf9J9tPn+0Mnr84sXVrGM9ZLWtzVqLlUmieO+EkP2sMR1VKflpefFjrp9+4saLWX92q4VOFlRZzwdBFqhgsKZoKqBK68HSBzudhHzbgJEAAaltV+JHYh+UeUKGBfla8wgDgR/5TKLxYBngDnTNuwx9zsTz3dIZVxU2AvfODAMVgmI7TTcFdkHVgSLo6LgaXdFazVnHtmERrJ1nauKlH4wSTPPRpa3mD7AIrPolQo+J26jeZBHgdmRnMaxOXdrBh/3Z4VNauVBk7FbqFva2tyf9pk9bN30290E3ruGY3g+atBFfDOmCYCcOZk6sIkBkRzwpsgQaZi8/QjyFkt698F+QH4/fj7Mvh8OiwS2OLvCSvyIhk5C05Ih/JMckJI9/JT/IrIclVct2Lv+SmtZd0nl3yT/W2fwM5kLKW</latexit><latexit sha1_base64="sYlUpp/x2K67VgHp7EuvG0Zqvg8=">AAACUnicbVJdaxQxFM2sX3WtdrWPvlxchC3UZaYU1LeCL76IFZy2sNkOd7LZ2WyTzJBkhCXkP0qhD/4RX/qg2e0I2nohcHLOPdzkJGUjhXVp+iPp3bv/4OGjrcf9J9tPn+0Mnr84sXVrGM9ZLWtzVqLlUmieO+EkP2sMR1VKflpefFjrp9+4saLWX92q4VOFlRZzwdBFqhgsKZoKqBK68HSBzudhHzbgJEAAaltV+JHYh+UeUKGBfla8wgDgR/5TKLxYBngDnTNuwx9zsTz3dIZVxU2AvfODAMVgmI7TTcFdkHVgSLo6LgaXdFazVnHtmERrJ1nauKlH4wSTPPRpa3mD7AIrPolQo+J26jeZBHgdmRnMaxOXdrBh/3Z4VNauVBk7FbqFva2tyf9pk9bN30290E3ruGY3g+atBFfDOmCYCcOZk6sIkBkRzwpsgQaZi8/QjyFkt698F+QH4/fj7Mvh8OiwS2OLvCSvyIhk5C05Ih/JMckJI9/JT/IrIclVct2Lv+SmtZd0nl3yT/W2fwM5kLKW</latexit> arg min ˆU, ˆV X (i,j)2⌦ (Mij ˆUi ˆV † j )2 + U k ˆUk2 F + V k ˆV k2 F <latexit sha1_base64="QbHbHTDK8iJKyzMIsXFV+BYgY7g=">AAACjnicbVFdixMxFM2MX2tdtauPvlwsQhfXMlMWVlFxQZB9EVdwugtNO6SZdJpukhmSjFBi/o4/yDf/jWl3KuuuFwIn59xDbs6d1YIbmyS/o/jW7Tt37+3c7zzYffjocXfvychUjaYso5Wo9PmMGCa4YpnlVrDzWjMiZ4KdzS4+rvWz70wbXqlvdlWziSSl4nNOiQ1U3v2JiS4BS65yhxfEuswfwAaMPHjAppG56/MDWO4D5grwF8lK4gFc3332ueNLD6+gdYar35rz5dThgpQl0x72p8NgeQlYhMkKkmeAf2xNAeafpsMr6uivOtqqebeXDJJNwU2QtqCH2jrNu79wUdFGMmWpIMaM06S2E0e05VQw38GNYTWhF6Rk4wAVkcxM3CZPDy8CU8C80uEoCxv2qsMRacxKzkKnJHZhrmtr8n/auLHz1xPHVd1YpujlQ/NGgK1gvRwouGbUilUAhGoeZgW6IJpQG1bYCSGk1798E2TDwZtB+vWwd3zYprGDnqHnqI9SdISO0Qk6RRmi0W40jN5G7+K9+Ch+H3+4bI2j1vMU/VPxyR/Vt8QD</latexit><latexit sha1_base64="QbHbHTDK8iJKyzMIsXFV+BYgY7g=">AAACjnicbVFdixMxFM2MX2tdtauPvlwsQhfXMlMWVlFxQZB9EVdwugtNO6SZdJpukhmSjFBi/o4/yDf/jWl3KuuuFwIn59xDbs6d1YIbmyS/o/jW7Tt37+3c7zzYffjocXfvychUjaYso5Wo9PmMGCa4YpnlVrDzWjMiZ4KdzS4+rvWz70wbXqlvdlWziSSl4nNOiQ1U3v2JiS4BS65yhxfEuswfwAaMPHjAppG56/MDWO4D5grwF8lK4gFc3332ueNLD6+gdYar35rz5dThgpQl0x72p8NgeQlYhMkKkmeAf2xNAeafpsMr6uivOtqqebeXDJJNwU2QtqCH2jrNu79wUdFGMmWpIMaM06S2E0e05VQw38GNYTWhF6Rk4wAVkcxM3CZPDy8CU8C80uEoCxv2qsMRacxKzkKnJHZhrmtr8n/auLHz1xPHVd1YpujlQ/NGgK1gvRwouGbUilUAhGoeZgW6IJpQG1bYCSGk1798E2TDwZtB+vWwd3zYprGDnqHnqI9SdISO0Qk6RRmi0W40jN5G7+K9+Ch+H3+4bI2j1vMU/VPxyR/Vt8QD</latexit><latexit sha1_base64="QbHbHTDK8iJKyzMIsXFV+BYgY7g=">AAACjnicbVFdixMxFM2MX2tdtauPvlwsQhfXMlMWVlFxQZB9EVdwugtNO6SZdJpukhmSjFBi/o4/yDf/jWl3KuuuFwIn59xDbs6d1YIbmyS/o/jW7Tt37+3c7zzYffjocXfvychUjaYso5Wo9PmMGCa4YpnlVrDzWjMiZ4KdzS4+rvWz70wbXqlvdlWziSSl4nNOiQ1U3v2JiS4BS65yhxfEuswfwAaMPHjAppG56/MDWO4D5grwF8lK4gFc3332ueNLD6+gdYar35rz5dThgpQl0x72p8NgeQlYhMkKkmeAf2xNAeafpsMr6uivOtqqebeXDJJNwU2QtqCH2jrNu79wUdFGMmWpIMaM06S2E0e05VQw38GNYTWhF6Rk4wAVkcxM3CZPDy8CU8C80uEoCxv2qsMRacxKzkKnJHZhrmtr8n/auLHz1xPHVd1YpujlQ/NGgK1gvRwouGbUilUAhGoeZgW6IJpQG1bYCSGk1798E2TDwZtB+vWwd3zYprGDnqHnqI9SdISO0Qk6RRmi0W40jN5G7+K9+Ch+H3+4bI2j1vMU/VPxyR/Vt8QD</latexit><latexit sha1_base64="QbHbHTDK8iJKyzMIsXFV+BYgY7g=">AAACjnicbVFdixMxFM2MX2tdtauPvlwsQhfXMlMWVlFxQZB9EVdwugtNO6SZdJpukhmSjFBi/o4/yDf/jWl3KuuuFwIn59xDbs6d1YIbmyS/o/jW7Tt37+3c7zzYffjocXfvychUjaYso5Wo9PmMGCa4YpnlVrDzWjMiZ4KdzS4+rvWz70wbXqlvdlWziSSl4nNOiQ1U3v2JiS4BS65yhxfEuswfwAaMPHjAppG56/MDWO4D5grwF8lK4gFc3332ueNLD6+gdYar35rz5dThgpQl0x72p8NgeQlYhMkKkmeAf2xNAeafpsMr6uivOtqqebeXDJJNwU2QtqCH2jrNu79wUdFGMmWpIMaM06S2E0e05VQw38GNYTWhF6Rk4wAVkcxM3CZPDy8CU8C80uEoCxv2qsMRacxKzkKnJHZhrmtr8n/auLHz1xPHVd1YpujlQ/NGgK1gvRwouGbUilUAhGoeZgW6IJpQG1bYCSGk1798E2TDwZtB+vWwd3zYprGDnqHnqI9SdISO0Qk6RRmi0W40jN5G7+K9+Ch+H3+4bI2j1vMU/VPxyR/Vt8QD</latexit> max(rank( ˆU), rank( ˆV )) ⌧ dim(M)<latexit sha1_base64="fXL2aN8MX5rq1Tyzyywy1m+UmlE=">AAACJHicbVA9SwNBEN3z2/gVtbRZDEICEu5EUMFCsLERFDwVciHMbTZmye7esTsnCcf9GRv/io2FioWNv8VNTKHGBwOP92aYmRenUlj0/Q9vanpmdm5+YbG0tLyyulZe37i2SWYYD1kiE3Mbg+VSaB6iQMlvU8NBxZLfxL3ToX9zz40Vib7CQcqbCu606AgG6KRW+ThS0K9GyPuYG9C9ohp1AfOwqO3SCfW6qNVoJCWN2kJVz2utcsWv+yPQSRKMSYWMcdEqv0bthGWKa2QSrG0EforNHAwKJnlRijLLU2A9uOMNRzUobpv56MuC7jilTTuJcaWRjtSfEzkoawcqdp0KsGv/ekPxP6+RYeewmQudZsg1+17UySTFhA4jo21hOEM5cASYEe5WyrpggKELtuRCCP6+PEnCvfpRPbjcr5z44zQWyBbZJlUSkANyQs7IBQkJIw/kibyQV+/Re/bevPfv1ilvPLNJfsH7/AJcoaSl</latexit><latexit sha1_base64="fXL2aN8MX5rq1Tyzyywy1m+UmlE=">AAACJHicbVA9SwNBEN3z2/gVtbRZDEICEu5EUMFCsLERFDwVciHMbTZmye7esTsnCcf9GRv/io2FioWNv8VNTKHGBwOP92aYmRenUlj0/Q9vanpmdm5+YbG0tLyyulZe37i2SWYYD1kiE3Mbg+VSaB6iQMlvU8NBxZLfxL3ToX9zz40Vib7CQcqbCu606AgG6KRW+ThS0K9GyPuYG9C9ohp1AfOwqO3SCfW6qNVoJCWN2kJVz2utcsWv+yPQSRKMSYWMcdEqv0bthGWKa2QSrG0EforNHAwKJnlRijLLU2A9uOMNRzUobpv56MuC7jilTTuJcaWRjtSfEzkoawcqdp0KsGv/ekPxP6+RYeewmQudZsg1+17UySTFhA4jo21hOEM5cASYEe5WyrpggKELtuRCCP6+PEnCvfpRPbjcr5z44zQWyBbZJlUSkANyQs7IBQkJIw/kibyQV+/Re/bevPfv1ilvPLNJfsH7/AJcoaSl</latexit><latexit sha1_base64="fXL2aN8MX5rq1Tyzyywy1m+UmlE=">AAACJHicbVA9SwNBEN3z2/gVtbRZDEICEu5EUMFCsLERFDwVciHMbTZmye7esTsnCcf9GRv/io2FioWNv8VNTKHGBwOP92aYmRenUlj0/Q9vanpmdm5+YbG0tLyyulZe37i2SWYYD1kiE3Mbg+VSaB6iQMlvU8NBxZLfxL3ToX9zz40Vib7CQcqbCu606AgG6KRW+ThS0K9GyPuYG9C9ohp1AfOwqO3SCfW6qNVoJCWN2kJVz2utcsWv+yPQSRKMSYWMcdEqv0bthGWKa2QSrG0EforNHAwKJnlRijLLU2A9uOMNRzUobpv56MuC7jilTTuJcaWRjtSfEzkoawcqdp0KsGv/ekPxP6+RYeewmQudZsg1+17UySTFhA4jo21hOEM5cASYEe5WyrpggKELtuRCCP6+PEnCvfpRPbjcr5z44zQWyBbZJlUSkANyQs7IBQkJIw/kibyQV+/Re/bevPfv1ilvPLNJfsH7/AJcoaSl</latexit><latexit sha1_base64="fXL2aN8MX5rq1Tyzyywy1m+UmlE=">AAACJHicbVA9SwNBEN3z2/gVtbRZDEICEu5EUMFCsLERFDwVciHMbTZmye7esTsnCcf9GRv/io2FioWNv8VNTKHGBwOP92aYmRenUlj0/Q9vanpmdm5+YbG0tLyyulZe37i2SWYYD1kiE3Mbg+VSaB6iQMlvU8NBxZLfxL3ToX9zz40Vib7CQcqbCu606AgG6KRW+ThS0K9GyPuYG9C9ohp1AfOwqO3SCfW6qNVoJCWN2kJVz2utcsWv+yPQSRKMSYWMcdEqv0bthGWKa2QSrG0EforNHAwKJnlRijLLU2A9uOMNRzUobpv56MuC7jilTTuJcaWRjtSfEzkoawcqdp0KsGv/ekPxP6+RYeewmQudZsg1+17UySTFhA4jo21hOEM5cASYEe5WyrpggKELtuRCCP6+PEnCvfpRPbjcr5z44zQWyBbZJlUSkANyQs7IBQkJIw/kibyQV+/Re/bevPfv1ilvPLNJfsH7/AJcoaSl</latexit>
  11. 11. SOLUTION OF THE FUNK SVD • Optimization form of the Funk SVD: • If we know all entries (= contains all (u, i) pairs), we can solve by SVD (Singular vector decomposition). • However, we can’t know the all entries. • If we don’t know all entries, this is NP-hard problem. • Then, how to solve this optimization problem? • One of approaches is convex relaxation. • But, the Funk-SVD use another algorithm. !11 ⌦<latexit sha1_base64="CXYjJQmltPnYcikH2EiL4mSI51A=">AAAB7HicbVDLSgNBEOyNrxhfUY9eBoPgKewGQb0FvHgzgnlAsoTZyWwyZh7LzKwQlvyDFw8qXv0gb/6Nk2QPmljQUFR1090VJZwZ6/vfXmFtfWNzq7hd2tnd2z8oHx61jEo1oU2iuNKdCBvKmaRNyyynnURTLCJO29H4Zua3n6g2TMkHO0loKPBQspgRbJ3U6t0JOsT9csWv+nOgVRLkpAI5Gv3yV2+gSCqotIRjY7qBn9gww9oywum01EsNTTAZ4yHtOiqxoCbM5tdO0ZlTBihW2pW0aK7+nsiwMGYiItcpsB2ZZW8m/ud1UxtfhRmTSWqpJItFccqRVWj2OhowTYnlE0cw0czdisgIa0ysC6jkQgiWX14lzVr1uhrcX1TqtTyNIpzAKZxDAJdQh1toQBMIPMIzvMKbp7wX7937WLQWvHzmGP7A+/wBxuaOvw==</latexit><latexit sha1_base64="CXYjJQmltPnYcikH2EiL4mSI51A=">AAAB7HicbVDLSgNBEOyNrxhfUY9eBoPgKewGQb0FvHgzgnlAsoTZyWwyZh7LzKwQlvyDFw8qXv0gb/6Nk2QPmljQUFR1090VJZwZ6/vfXmFtfWNzq7hd2tnd2z8oHx61jEo1oU2iuNKdCBvKmaRNyyynnURTLCJO29H4Zua3n6g2TMkHO0loKPBQspgRbJ3U6t0JOsT9csWv+nOgVRLkpAI5Gv3yV2+gSCqotIRjY7qBn9gww9oywum01EsNTTAZ4yHtOiqxoCbM5tdO0ZlTBihW2pW0aK7+nsiwMGYiItcpsB2ZZW8m/ud1UxtfhRmTSWqpJItFccqRVWj2OhowTYnlE0cw0czdisgIa0ysC6jkQgiWX14lzVr1uhrcX1TqtTyNIpzAKZxDAJdQh1toQBMIPMIzvMKbp7wX7937WLQWvHzmGP7A+/wBxuaOvw==</latexit><latexit sha1_base64="CXYjJQmltPnYcikH2EiL4mSI51A=">AAAB7HicbVDLSgNBEOyNrxhfUY9eBoPgKewGQb0FvHgzgnlAsoTZyWwyZh7LzKwQlvyDFw8qXv0gb/6Nk2QPmljQUFR1090VJZwZ6/vfXmFtfWNzq7hd2tnd2z8oHx61jEo1oU2iuNKdCBvKmaRNyyynnURTLCJO29H4Zua3n6g2TMkHO0loKPBQspgRbJ3U6t0JOsT9csWv+nOgVRLkpAI5Gv3yV2+gSCqotIRjY7qBn9gww9oywum01EsNTTAZ4yHtOiqxoCbM5tdO0ZlTBihW2pW0aK7+nsiwMGYiItcpsB2ZZW8m/ud1UxtfhRmTSWqpJItFccqRVWj2OhowTYnlE0cw0czdisgIa0ysC6jkQgiWX14lzVr1uhrcX1TqtTyNIpzAKZxDAJdQh1toQBMIPMIzvMKbp7wX7937WLQWvHzmGP7A+/wBxuaOvw==</latexit><latexit sha1_base64="CXYjJQmltPnYcikH2EiL4mSI51A=">AAAB7HicbVDLSgNBEOyNrxhfUY9eBoPgKewGQb0FvHgzgnlAsoTZyWwyZh7LzKwQlvyDFw8qXv0gb/6Nk2QPmljQUFR1090VJZwZ6/vfXmFtfWNzq7hd2tnd2z8oHx61jEo1oU2iuNKdCBvKmaRNyyynnURTLCJO29H4Zua3n6g2TMkHO0loKPBQspgRbJ3U6t0JOsT9csWv+nOgVRLkpAI5Gv3yV2+gSCqotIRjY7qBn9gww9oywum01EsNTTAZ4yHtOiqxoCbM5tdO0ZlTBihW2pW0aK7+nsiwMGYiItcpsB2ZZW8m/ud1UxtfhRmTSWqpJItFccqRVWj2OhowTYnlE0cw0czdisgIa0ysC6jkQgiWX14lzVr1uhrcX1TqtTyNIpzAKZxDAJdQh1toQBMIPMIzvMKbp7wX7937WLQWvHzmGP7A+/wBxuaOvw==</latexit> arg min ˆU, ˆV X (i,j)2⌦ (Mij ˆUi ˆV † j )2 + U k ˆUk2 F + V k ˆV k2 F <latexit sha1_base64="QbHbHTDK8iJKyzMIsXFV+BYgY7g=">AAACjnicbVFdixMxFM2MX2tdtauPvlwsQhfXMlMWVlFxQZB9EVdwugtNO6SZdJpukhmSjFBi/o4/yDf/jWl3KuuuFwIn59xDbs6d1YIbmyS/o/jW7Tt37+3c7zzYffjocXfvychUjaYso5Wo9PmMGCa4YpnlVrDzWjMiZ4KdzS4+rvWz70wbXqlvdlWziSSl4nNOiQ1U3v2JiS4BS65yhxfEuswfwAaMPHjAppG56/MDWO4D5grwF8lK4gFc3332ueNLD6+gdYar35rz5dThgpQl0x72p8NgeQlYhMkKkmeAf2xNAeafpsMr6uivOtqqebeXDJJNwU2QtqCH2jrNu79wUdFGMmWpIMaM06S2E0e05VQw38GNYTWhF6Rk4wAVkcxM3CZPDy8CU8C80uEoCxv2qsMRacxKzkKnJHZhrmtr8n/auLHz1xPHVd1YpujlQ/NGgK1gvRwouGbUilUAhGoeZgW6IJpQG1bYCSGk1798E2TDwZtB+vWwd3zYprGDnqHnqI9SdISO0Qk6RRmi0W40jN5G7+K9+Ch+H3+4bI2j1vMU/VPxyR/Vt8QD</latexit><latexit sha1_base64="QbHbHTDK8iJKyzMIsXFV+BYgY7g=">AAACjnicbVFdixMxFM2MX2tdtauPvlwsQhfXMlMWVlFxQZB9EVdwugtNO6SZdJpukhmSjFBi/o4/yDf/jWl3KuuuFwIn59xDbs6d1YIbmyS/o/jW7Tt37+3c7zzYffjocXfvychUjaYso5Wo9PmMGCa4YpnlVrDzWjMiZ4KdzS4+rvWz70wbXqlvdlWziSSl4nNOiQ1U3v2JiS4BS65yhxfEuswfwAaMPHjAppG56/MDWO4D5grwF8lK4gFc3332ueNLD6+gdYar35rz5dThgpQl0x72p8NgeQlYhMkKkmeAf2xNAeafpsMr6uivOtqqebeXDJJNwU2QtqCH2jrNu79wUdFGMmWpIMaM06S2E0e05VQw38GNYTWhF6Rk4wAVkcxM3CZPDy8CU8C80uEoCxv2qsMRacxKzkKnJHZhrmtr8n/auLHz1xPHVd1YpujlQ/NGgK1gvRwouGbUilUAhGoeZgW6IJpQG1bYCSGk1798E2TDwZtB+vWwd3zYprGDnqHnqI9SdISO0Qk6RRmi0W40jN5G7+K9+Ch+H3+4bI2j1vMU/VPxyR/Vt8QD</latexit><latexit sha1_base64="QbHbHTDK8iJKyzMIsXFV+BYgY7g=">AAACjnicbVFdixMxFM2MX2tdtauPvlwsQhfXMlMWVlFxQZB9EVdwugtNO6SZdJpukhmSjFBi/o4/yDf/jWl3KuuuFwIn59xDbs6d1YIbmyS/o/jW7Tt37+3c7zzYffjocXfvychUjaYso5Wo9PmMGCa4YpnlVrDzWjMiZ4KdzS4+rvWz70wbXqlvdlWziSSl4nNOiQ1U3v2JiS4BS65yhxfEuswfwAaMPHjAppG56/MDWO4D5grwF8lK4gFc3332ueNLD6+gdYar35rz5dThgpQl0x72p8NgeQlYhMkKkmeAf2xNAeafpsMr6uivOtqqebeXDJJNwU2QtqCH2jrNu79wUdFGMmWpIMaM06S2E0e05VQw38GNYTWhF6Rk4wAVkcxM3CZPDy8CU8C80uEoCxv2qsMRacxKzkKnJHZhrmtr8n/auLHz1xPHVd1YpujlQ/NGgK1gvRwouGbUilUAhGoeZgW6IJpQG1bYCSGk1798E2TDwZtB+vWwd3zYprGDnqHnqI9SdISO0Qk6RRmi0W40jN5G7+K9+Ch+H3+4bI2j1vMU/VPxyR/Vt8QD</latexit><latexit sha1_base64="QbHbHTDK8iJKyzMIsXFV+BYgY7g=">AAACjnicbVFdixMxFM2MX2tdtauPvlwsQhfXMlMWVlFxQZB9EVdwugtNO6SZdJpukhmSjFBi/o4/yDf/jWl3KuuuFwIn59xDbs6d1YIbmyS/o/jW7Tt37+3c7zzYffjocXfvychUjaYso5Wo9PmMGCa4YpnlVrDzWjMiZ4KdzS4+rvWz70wbXqlvdlWziSSl4nNOiQ1U3v2JiS4BS65yhxfEuswfwAaMPHjAppG56/MDWO4D5grwF8lK4gFc3332ueNLD6+gdYar35rz5dThgpQl0x72p8NgeQlYhMkKkmeAf2xNAeafpsMr6uivOtqqebeXDJJNwU2QtqCH2jrNu79wUdFGMmWpIMaM06S2E0e05VQw38GNYTWhF6Rk4wAVkcxM3CZPDy8CU8C80uEoCxv2qsMRacxKzkKnJHZhrmtr8n/auLHz1xPHVd1YpujlQ/NGgK1gvRwouGbUilUAhGoeZgW6IJpQG1bYCSGk1798E2TDwZtB+vWwd3zYprGDnqHnqI9SdISO0Qk6RRmi0W40jN5G7+K9+Ch+H3+4bI2j1vMU/VPxyR/Vt8QD</latexit>
  12. 12. SOLVE BY ALTERNATING MINIMIZATION !12 M ≒ xm m n n k k These can be solved by SVD! (pseudo inverse) ˆU<latexit sha1_base64="qEBNw/NF9I5Pysh/o7KQOMhnYcA=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEaL0VvHisYGyhDWWz3bRLN5uwOxFK6I/w4kHFq//Hm//GbZuDtj4YeLw3w8y8MJXCoOt+O6WNza3tnfJuZW//4PCoenzyaJJMM+6zRCa6G1LDpVDcR4GSd1PNaRxK3gknt3O/88S1EYl6wGnKg5iOlIgEo2ilTn9MMfdng2rNrbsLkHXiFaQGBdqD6ld/mLAs5gqZpMb0PDfFIKcaBZN8VulnhqeUTeiI9yxVNOYmyBfnzsiFVYYkSrQthWSh/p7IaWzMNA5tZ0xxbFa9ufif18swaga5UGmGXLHloiiTBBMy/50MheYM5dQSyrSwtxI2ppoytAlVbAje6svrxL+q39S9++taq1mkUYYzOIdL8KABLbiDNvjAYALP8ApvTuq8OO/Ox7K15BQzp/AHzucP4+6PcA==</latexit><latexit sha1_base64="qEBNw/NF9I5Pysh/o7KQOMhnYcA=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEaL0VvHisYGyhDWWz3bRLN5uwOxFK6I/w4kHFq//Hm//GbZuDtj4YeLw3w8y8MJXCoOt+O6WNza3tnfJuZW//4PCoenzyaJJMM+6zRCa6G1LDpVDcR4GSd1PNaRxK3gknt3O/88S1EYl6wGnKg5iOlIgEo2ilTn9MMfdng2rNrbsLkHXiFaQGBdqD6ld/mLAs5gqZpMb0PDfFIKcaBZN8VulnhqeUTeiI9yxVNOYmyBfnzsiFVYYkSrQthWSh/p7IaWzMNA5tZ0xxbFa9ufif18swaga5UGmGXLHloiiTBBMy/50MheYM5dQSyrSwtxI2ppoytAlVbAje6svrxL+q39S9++taq1mkUYYzOIdL8KABLbiDNvjAYALP8ApvTuq8OO/Ox7K15BQzp/AHzucP4+6PcA==</latexit><latexit sha1_base64="qEBNw/NF9I5Pysh/o7KQOMhnYcA=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEaL0VvHisYGyhDWWz3bRLN5uwOxFK6I/w4kHFq//Hm//GbZuDtj4YeLw3w8y8MJXCoOt+O6WNza3tnfJuZW//4PCoenzyaJJMM+6zRCa6G1LDpVDcR4GSd1PNaRxK3gknt3O/88S1EYl6wGnKg5iOlIgEo2ilTn9MMfdng2rNrbsLkHXiFaQGBdqD6ld/mLAs5gqZpMb0PDfFIKcaBZN8VulnhqeUTeiI9yxVNOYmyBfnzsiFVYYkSrQthWSh/p7IaWzMNA5tZ0xxbFa9ufif18swaga5UGmGXLHloiiTBBMy/50MheYM5dQSyrSwtxI2ppoytAlVbAje6svrxL+q39S9++taq1mkUYYzOIdL8KABLbiDNvjAYALP8ApvTuq8OO/Ox7K15BQzp/AHzucP4+6PcA==</latexit><latexit sha1_base64="qEBNw/NF9I5Pysh/o7KQOMhnYcA=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEaL0VvHisYGyhDWWz3bRLN5uwOxFK6I/w4kHFq//Hm//GbZuDtj4YeLw3w8y8MJXCoOt+O6WNza3tnfJuZW//4PCoenzyaJJMM+6zRCa6G1LDpVDcR4GSd1PNaRxK3gknt3O/88S1EYl6wGnKg5iOlIgEo2ilTn9MMfdng2rNrbsLkHXiFaQGBdqD6ld/mLAs5gqZpMb0PDfFIKcaBZN8VulnhqeUTeiI9yxVNOYmyBfnzsiFVYYkSrQthWSh/p7IaWzMNA5tZ0xxbFa9ufif18swaga5UGmGXLHloiiTBBMy/50MheYM5dQSyrSwtxI2ppoytAlVbAje6svrxL+q39S9++taq1mkUYYzOIdL8KABLbiDNvjAYALP8ApvTuq8OO/Ox7K15BQzp/AHzucP4+6PcA==</latexit> ˆV † <latexit sha1_base64="gToXFTlZVnuHTXJsUTeaDo+Bsko=">AAAB93icbVBNS8NAEN34WetHox69LBbBU0lEsN4KXjxWMG2hjWWz2aRLN5uwOxFq6C/x4kHFq3/Fm//GbZuDtj4YeLw3w8y8IBNcg+N8W2vrG5tb25Wd6u7e/kHNPjzq6DRXlHk0FanqBUQzwSXzgINgvUwxkgSCdYPxzczvPjKleSrvYZIxPyGx5BGnBIw0tGuDEYGiM30YhCSOmRradafhzIFXiVuSOirRHtpfgzClecIkUEG07rtOBn5BFHAq2LQ6yDXLCB2TmPUNlSRh2i/mh0/xmVFCHKXKlAQ8V39PFCTRepIEpjMhMNLL3kz8z+vnEDX9gsssBybpYlGUCwwpnqWAQ64YBTExhFDFza2YjogiFExWVROCu/zyKvEuGtcN9+6y3mqWaVTQCTpF58hFV6iFblEbeYiiHD2jV/RmPVkv1rv1sWhds8qZY/QH1ucPZl+TFg==</latexit><latexit sha1_base64="gToXFTlZVnuHTXJsUTeaDo+Bsko=">AAAB93icbVBNS8NAEN34WetHox69LBbBU0lEsN4KXjxWMG2hjWWz2aRLN5uwOxFq6C/x4kHFq3/Fm//GbZuDtj4YeLw3w8y8IBNcg+N8W2vrG5tb25Wd6u7e/kHNPjzq6DRXlHk0FanqBUQzwSXzgINgvUwxkgSCdYPxzczvPjKleSrvYZIxPyGx5BGnBIw0tGuDEYGiM30YhCSOmRradafhzIFXiVuSOirRHtpfgzClecIkUEG07rtOBn5BFHAq2LQ6yDXLCB2TmPUNlSRh2i/mh0/xmVFCHKXKlAQ8V39PFCTRepIEpjMhMNLL3kz8z+vnEDX9gsssBybpYlGUCwwpnqWAQ64YBTExhFDFza2YjogiFExWVROCu/zyKvEuGtcN9+6y3mqWaVTQCTpF58hFV6iFblEbeYiiHD2jV/RmPVkv1rv1sWhds8qZY/QH1ucPZl+TFg==</latexit><latexit sha1_base64="gToXFTlZVnuHTXJsUTeaDo+Bsko=">AAAB93icbVBNS8NAEN34WetHox69LBbBU0lEsN4KXjxWMG2hjWWz2aRLN5uwOxFq6C/x4kHFq3/Fm//GbZuDtj4YeLw3w8y8IBNcg+N8W2vrG5tb25Wd6u7e/kHNPjzq6DRXlHk0FanqBUQzwSXzgINgvUwxkgSCdYPxzczvPjKleSrvYZIxPyGx5BGnBIw0tGuDEYGiM30YhCSOmRradafhzIFXiVuSOirRHtpfgzClecIkUEG07rtOBn5BFHAq2LQ6yDXLCB2TmPUNlSRh2i/mh0/xmVFCHKXKlAQ8V39PFCTRepIEpjMhMNLL3kz8z+vnEDX9gsssBybpYlGUCwwpnqWAQ64YBTExhFDFza2YjogiFExWVROCu/zyKvEuGtcN9+6y3mqWaVTQCTpF58hFV6iFblEbeYiiHD2jV/RmPVkv1rv1sWhds8qZY/QH1ucPZl+TFg==</latexit><latexit sha1_base64="gToXFTlZVnuHTXJsUTeaDo+Bsko=">AAAB93icbVBNS8NAEN34WetHox69LBbBU0lEsN4KXjxWMG2hjWWz2aRLN5uwOxFq6C/x4kHFq3/Fm//GbZuDtj4YeLw3w8y8IBNcg+N8W2vrG5tb25Wd6u7e/kHNPjzq6DRXlHk0FanqBUQzwSXzgINgvUwxkgSCdYPxzczvPjKleSrvYZIxPyGx5BGnBIw0tGuDEYGiM30YhCSOmRradafhzIFXiVuSOirRHtpfgzClecIkUEG07rtOBn5BFHAq2LQ6yDXLCB2TmPUNlSRh2i/mh0/xmVFCHKXKlAQ8V39PFCTRepIEpjMhMNLL3kz8z+vnEDX9gsssBybpYlGUCwwpnqWAQ64YBTExhFDFza2YjogiFExWVROCu/zyKvEuGtcN9+6y3mqWaVTQCTpF58hFV6iFblEbeYiiHD2jV/RmPVkv1rv1sWhds8qZY/QH1ucPZl+TFg==</latexit> 1: Input: observed set ⌦, values P⌦(M) 2: Initialize ˆV 0 randomly. 3: for t = 1, · · · , T: 4: ˆUt arg min U2Rm⇥k kP⌦(M U( ˆV t 1 )† )k2 F + U kUk2 F 5: ˆV t arg min V 2Rk⇥n kP⌦(M ˆUt V † )k2 F + V kV k2 F 6: Return R = ˆUt ( ˆV t )† <latexit sha1_base64="pQ+/bsDaW5e0HjtjvbKSiNfXu+o=">AAAEBXicfVNNb9NAEHUdPkr4auEIhxEVKBVtFIfyFalSJSQEh4pS1W6lbmKt15t0lfU62h0XgusTF/4KFw6AuPIfuPFvWDdxCW3F+vI0M2/2vfFsNJLCYKv1e86tXbh46fL8lfrVa9dv3FxYvBWYNNOM+yyVqd6LqOFSKO6jQMn3RprTJJJ8Nxq+KPO7h1wbkaodHI94N6EDJfqCUbShcNG9+4Agf4+514HXapRhB9LIcH3IYzAcoSBvEj6gK5MiOKQy4waKrXASb8AmLAMh9WmXdtlFoKBSfOCWfEAxD4pea0rXVMVpIsfNYobzqGMLS4SY91MNBSCseytA4hRNtAI7MKmzZTOstQ4Q+02u8ItejjYreR+p1uk7IFQPgCRChbkPRCiLKR5EUb5tSxPbUSTWx7CwpCOo3EBjE1bBh0alO8dVr1jukZgOBlxbo0fhy14bHtqb7IhjGvol3a/ihEAl7/GsvOB/8oIz8oaVPHWevMowQlAJg3OUBSUz+KusEvakA9scM63sNLdhfabdiWs8cRwuLLWareMDZ4E3BUvO9GyFC7/sX2NZwhUySY3Z91oj7OZUo2CSF3WSGT6ibEgHfN9CRa3Nbn68xgXct5EYyhXopwrhODrLyGlizDiJbGU5LXM6VwbPy+1n2H/WzUW53lyxyUX9TAKmUL4JiIXmDOXYAsq0XV8G7IBqytC+nLodgnfa8lngt5vPm97btaWN9nQa884d557TcDznqbPhvHK2HN9h7kf3s/vV/Vb7VPtS+177MSl156ac284/p/bzD1A7SFE=</latexit><latexit sha1_base64="pQ+/bsDaW5e0HjtjvbKSiNfXu+o=">AAAEBXicfVNNb9NAEHUdPkr4auEIhxEVKBVtFIfyFalSJSQEh4pS1W6lbmKt15t0lfU62h0XgusTF/4KFw6AuPIfuPFvWDdxCW3F+vI0M2/2vfFsNJLCYKv1e86tXbh46fL8lfrVa9dv3FxYvBWYNNOM+yyVqd6LqOFSKO6jQMn3RprTJJJ8Nxq+KPO7h1wbkaodHI94N6EDJfqCUbShcNG9+4Agf4+514HXapRhB9LIcH3IYzAcoSBvEj6gK5MiOKQy4waKrXASb8AmLAMh9WmXdtlFoKBSfOCWfEAxD4pea0rXVMVpIsfNYobzqGMLS4SY91MNBSCseytA4hRNtAI7MKmzZTOstQ4Q+02u8ItejjYreR+p1uk7IFQPgCRChbkPRCiLKR5EUb5tSxPbUSTWx7CwpCOo3EBjE1bBh0alO8dVr1jukZgOBlxbo0fhy14bHtqb7IhjGvol3a/ihEAl7/GsvOB/8oIz8oaVPHWevMowQlAJg3OUBSUz+KusEvakA9scM63sNLdhfabdiWs8cRwuLLWareMDZ4E3BUvO9GyFC7/sX2NZwhUySY3Z91oj7OZUo2CSF3WSGT6ibEgHfN9CRa3Nbn68xgXct5EYyhXopwrhODrLyGlizDiJbGU5LXM6VwbPy+1n2H/WzUW53lyxyUX9TAKmUL4JiIXmDOXYAsq0XV8G7IBqytC+nLodgnfa8lngt5vPm97btaWN9nQa884d557TcDznqbPhvHK2HN9h7kf3s/vV/Vb7VPtS+177MSl156ac284/p/bzD1A7SFE=</latexit><latexit sha1_base64="pQ+/bsDaW5e0HjtjvbKSiNfXu+o=">AAAEBXicfVNNb9NAEHUdPkr4auEIhxEVKBVtFIfyFalSJSQEh4pS1W6lbmKt15t0lfU62h0XgusTF/4KFw6AuPIfuPFvWDdxCW3F+vI0M2/2vfFsNJLCYKv1e86tXbh46fL8lfrVa9dv3FxYvBWYNNOM+yyVqd6LqOFSKO6jQMn3RprTJJJ8Nxq+KPO7h1wbkaodHI94N6EDJfqCUbShcNG9+4Agf4+514HXapRhB9LIcH3IYzAcoSBvEj6gK5MiOKQy4waKrXASb8AmLAMh9WmXdtlFoKBSfOCWfEAxD4pea0rXVMVpIsfNYobzqGMLS4SY91MNBSCseytA4hRNtAI7MKmzZTOstQ4Q+02u8ItejjYreR+p1uk7IFQPgCRChbkPRCiLKR5EUb5tSxPbUSTWx7CwpCOo3EBjE1bBh0alO8dVr1jukZgOBlxbo0fhy14bHtqb7IhjGvol3a/ihEAl7/GsvOB/8oIz8oaVPHWevMowQlAJg3OUBSUz+KusEvakA9scM63sNLdhfabdiWs8cRwuLLWareMDZ4E3BUvO9GyFC7/sX2NZwhUySY3Z91oj7OZUo2CSF3WSGT6ibEgHfN9CRa3Nbn68xgXct5EYyhXopwrhODrLyGlizDiJbGU5LXM6VwbPy+1n2H/WzUW53lyxyUX9TAKmUL4JiIXmDOXYAsq0XV8G7IBqytC+nLodgnfa8lngt5vPm97btaWN9nQa884d557TcDznqbPhvHK2HN9h7kf3s/vV/Vb7VPtS+177MSl156ac284/p/bzD1A7SFE=</latexit><latexit sha1_base64="pQ+/bsDaW5e0HjtjvbKSiNfXu+o=">AAAEBXicfVNNb9NAEHUdPkr4auEIhxEVKBVtFIfyFalSJSQEh4pS1W6lbmKt15t0lfU62h0XgusTF/4KFw6AuPIfuPFvWDdxCW3F+vI0M2/2vfFsNJLCYKv1e86tXbh46fL8lfrVa9dv3FxYvBWYNNOM+yyVqd6LqOFSKO6jQMn3RprTJJJ8Nxq+KPO7h1wbkaodHI94N6EDJfqCUbShcNG9+4Agf4+514HXapRhB9LIcH3IYzAcoSBvEj6gK5MiOKQy4waKrXASb8AmLAMh9WmXdtlFoKBSfOCWfEAxD4pea0rXVMVpIsfNYobzqGMLS4SY91MNBSCseytA4hRNtAI7MKmzZTOstQ4Q+02u8ItejjYreR+p1uk7IFQPgCRChbkPRCiLKR5EUb5tSxPbUSTWx7CwpCOo3EBjE1bBh0alO8dVr1jukZgOBlxbo0fhy14bHtqb7IhjGvol3a/ihEAl7/GsvOB/8oIz8oaVPHWevMowQlAJg3OUBSUz+KusEvakA9scM63sNLdhfabdiWs8cRwuLLWareMDZ4E3BUvO9GyFC7/sX2NZwhUySY3Z91oj7OZUo2CSF3WSGT6ibEgHfN9CRa3Nbn68xgXct5EYyhXopwrhODrLyGlizDiJbGU5LXM6VwbPy+1n2H/WzUW53lyxyUX9TAKmUL4JiIXmDOXYAsq0XV8G7IBqytC+nLodgnfa8lngt5vPm97btaWN9nQa884d557TcDznqbPhvHK2HN9h7kf3s/vV/Vb7VPtS+177MSl156ac284/p/bzD1A7SFE=</latexit>
  13. 13. WHAT IS BIASED MATRIX FACTORIZATION? • The Biased Matrix Factorization is based on Funk SVD. • Biased MF = Funk SVD + Bias Terms • Some users tend to give higher ratings, and some users tend to give lower ratings. • The tendencies of movies are similar to that of users. • Some movies tend to get higher ratings, and some movies tend to get lower ratings. • Biased Matrix Factorization wants to handle this phenomenon. • To introduce bias terms in the optimization form. !13
  14. 14. OPTIMIZATION FORM OF BIASED-MF • Therefore, Biased-MF introduce biased terms related to user and item respectively. • is the average rating of training set(constant). • is the bias vector related to user. • is the bias vector related to item. • This optimization form can be solved by alternating minimization. • Similar to the solution of Funk SVD. • The Biased Matrix Factorization is also called SVD++. !14 µ<latexit sha1_base64="VlnKo72qdPNK/XpDYE6WLCIIdz8=">AAAB6XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5EULuAjWVEzwSSI+xt5pIlu3vH7p4QQn6CjYWKrf/Izn/jJrlCow8GHu/NMDMvzgQ31ve/vNLK6tr6RnmzsrW9s7tX3T94MGmuGYYsFalux9Sg4ApDy63AdqaRylhgKx5dz/zWI2rDU3VvxxlGkg4UTzij1kl3XZn3qjW/7s9B/pKgIDUo0OxVP7v9lOUSlWWCGtMJ/MxGE6otZwKnlW5uMKNsRAfYcVRRiSaazE+dkhOn9EmSalfKkrn6c2JCpTFjGbtOSe3QLHsz8T+vk9vkMppwleUWFVssSnJBbEpmf5M+18isGDtCmebuVsKGVFNmXToVF0Kw/PJfEp7Vr+rB7Xmt4RdplOEIjuEUAriABtxAE0JgMIAneIFXT3jP3pv3vmgtecXMIfyC9/ENxYONmA==</latexit><latexit sha1_base64="VlnKo72qdPNK/XpDYE6WLCIIdz8=">AAAB6XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5EULuAjWVEzwSSI+xt5pIlu3vH7p4QQn6CjYWKrf/Izn/jJrlCow8GHu/NMDMvzgQ31ve/vNLK6tr6RnmzsrW9s7tX3T94MGmuGYYsFalux9Sg4ApDy63AdqaRylhgKx5dz/zWI2rDU3VvxxlGkg4UTzij1kl3XZn3qjW/7s9B/pKgIDUo0OxVP7v9lOUSlWWCGtMJ/MxGE6otZwKnlW5uMKNsRAfYcVRRiSaazE+dkhOn9EmSalfKkrn6c2JCpTFjGbtOSe3QLHsz8T+vk9vkMppwleUWFVssSnJBbEpmf5M+18isGDtCmebuVsKGVFNmXToVF0Kw/PJfEp7Vr+rB7Xmt4RdplOEIjuEUAriABtxAE0JgMIAneIFXT3jP3pv3vmgtecXMIfyC9/ENxYONmA==</latexit><latexit sha1_base64="VlnKo72qdPNK/XpDYE6WLCIIdz8=">AAAB6XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5EULuAjWVEzwSSI+xt5pIlu3vH7p4QQn6CjYWKrf/Izn/jJrlCow8GHu/NMDMvzgQ31ve/vNLK6tr6RnmzsrW9s7tX3T94MGmuGYYsFalux9Sg4ApDy63AdqaRylhgKx5dz/zWI2rDU3VvxxlGkg4UTzij1kl3XZn3qjW/7s9B/pKgIDUo0OxVP7v9lOUSlWWCGtMJ/MxGE6otZwKnlW5uMKNsRAfYcVRRiSaazE+dkhOn9EmSalfKkrn6c2JCpTFjGbtOSe3QLHsz8T+vk9vkMppwleUWFVssSnJBbEpmf5M+18isGDtCmebuVsKGVFNmXToVF0Kw/PJfEp7Vr+rB7Xmt4RdplOEIjuEUAriABtxAE0JgMIAneIFXT3jP3pv3vmgtecXMIfyC9/ENxYONmA==</latexit><latexit sha1_base64="VlnKo72qdPNK/XpDYE6WLCIIdz8=">AAAB6XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5EULuAjWVEzwSSI+xt5pIlu3vH7p4QQn6CjYWKrf/Izn/jJrlCow8GHu/NMDMvzgQ31ve/vNLK6tr6RnmzsrW9s7tX3T94MGmuGYYsFalux9Sg4ApDy63AdqaRylhgKx5dz/zWI2rDU3VvxxlGkg4UTzij1kl3XZn3qjW/7s9B/pKgIDUo0OxVP7v9lOUSlWWCGtMJ/MxGE6otZwKnlW5uMKNsRAfYcVRRiSaazE+dkhOn9EmSalfKkrn6c2JCpTFjGbtOSe3QLHsz8T+vk9vkMppwleUWFVssSnJBbEpmf5M+18isGDtCmebuVsKGVFNmXToVF0Kw/PJfEp7Vr+rB7Xmt4RdplOEIjuEUAriABtxAE0JgMIAneIFXT3jP3pv3vmgtecXMIfyC9/ENxYONmA==</latexit> buser <latexit sha1_base64="DXbG6GcW+vrGhhOT8KyvQGgSL1c=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FLx4rGFtoY9lsJ+3SzSbuboQS+ie8eFDx6u/x5r9x2+agrQ8WHu/NzM68MBVcG9f9dkorq2vrG+XNytb2zu5edf/gXieZYuizRCSqHVKNgkv0DTcC26lCGocCW+Hoeuq3nlBpnsg7M04xiOlA8ogzaqzUDh/yTKOa9Ko1t+7OQJaJV5AaFGj2ql/dfsKyGKVhgmrd8dzUBDlVhjOBk0rXjk0pG9EBdiyVNEYd5LN9J+TEKn0SJco+achM/d2R01jrcRzaypiaoV70puJ/Xicz0WWQc5lmBiWbfxRlgpiETI8nfa6QGTG2hDLF7a6EDamizNiIKjYEb/HkZeKf1a/q3u15reEWaZThCI7hFDy4gAbcQBN8YCDgGV7hzXl0Xpx352NeWnKKnkP4A+fzB+UOkAM=</latexit><latexit sha1_base64="DXbG6GcW+vrGhhOT8KyvQGgSL1c=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FLx4rGFtoY9lsJ+3SzSbuboQS+ie8eFDx6u/x5r9x2+agrQ8WHu/NzM68MBVcG9f9dkorq2vrG+XNytb2zu5edf/gXieZYuizRCSqHVKNgkv0DTcC26lCGocCW+Hoeuq3nlBpnsg7M04xiOlA8ogzaqzUDh/yTKOa9Ko1t+7OQJaJV5AaFGj2ql/dfsKyGKVhgmrd8dzUBDlVhjOBk0rXjk0pG9EBdiyVNEYd5LN9J+TEKn0SJco+achM/d2R01jrcRzaypiaoV70puJ/Xicz0WWQc5lmBiWbfxRlgpiETI8nfa6QGTG2hDLF7a6EDamizNiIKjYEb/HkZeKf1a/q3u15reEWaZThCI7hFDy4gAbcQBN8YCDgGV7hzXl0Xpx352NeWnKKnkP4A+fzB+UOkAM=</latexit><latexit sha1_base64="DXbG6GcW+vrGhhOT8KyvQGgSL1c=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FLx4rGFtoY9lsJ+3SzSbuboQS+ie8eFDx6u/x5r9x2+agrQ8WHu/NzM68MBVcG9f9dkorq2vrG+XNytb2zu5edf/gXieZYuizRCSqHVKNgkv0DTcC26lCGocCW+Hoeuq3nlBpnsg7M04xiOlA8ogzaqzUDh/yTKOa9Ko1t+7OQJaJV5AaFGj2ql/dfsKyGKVhgmrd8dzUBDlVhjOBk0rXjk0pG9EBdiyVNEYd5LN9J+TEKn0SJco+achM/d2R01jrcRzaypiaoV70puJ/Xicz0WWQc5lmBiWbfxRlgpiETI8nfa6QGTG2hDLF7a6EDamizNiIKjYEb/HkZeKf1a/q3u15reEWaZThCI7hFDy4gAbcQBN8YCDgGV7hzXl0Xpx352NeWnKKnkP4A+fzB+UOkAM=</latexit><latexit sha1_base64="DXbG6GcW+vrGhhOT8KyvQGgSL1c=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FLx4rGFtoY9lsJ+3SzSbuboQS+ie8eFDx6u/x5r9x2+agrQ8WHu/NzM68MBVcG9f9dkorq2vrG+XNytb2zu5edf/gXieZYuizRCSqHVKNgkv0DTcC26lCGocCW+Hoeuq3nlBpnsg7M04xiOlA8ogzaqzUDh/yTKOa9Ko1t+7OQJaJV5AaFGj2ql/dfsKyGKVhgmrd8dzUBDlVhjOBk0rXjk0pG9EBdiyVNEYd5LN9J+TEKn0SJco+achM/d2R01jrcRzaypiaoV70puJ/Xicz0WWQc5lmBiWbfxRlgpiETI8nfa6QGTG2hDLF7a6EDamizNiIKjYEb/HkZeKf1a/q3u15reEWaZThCI7hFDy4gAbcQBN8YCDgGV7hzXl0Xpx352NeWnKKnkP4A+fzB+UOkAM=</latexit> bitem <latexit sha1_base64="ihgt0y/lts5uDQPDa3MALu8LZCM=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FLx4rGFtoY9lsJ+3SzSbuboQS+ie8eFDx6u/x5r9x2+agrQ8GHu/NMDMvTAXXxnW/ndLK6tr6RnmzsrW9s7tX3T+410mmGPosEYlqh1Sj4BJ9w43AdqqQxqHAVji6nvqtJ1SaJ/LOjFMMYjqQPOKMGiu1w4ecG4wnvWrNrbszkGXiFaQGBZq96le3n7AsRmmYoFp3PDc1QU6V4UzgpNLNNKaUjegAO5ZKGqMO8tm9E3JilT6JEmVLGjJTf0/kNNZ6HIe2M6ZmqBe9qfif18lMdBnkXKaZQcnmi6JMEJOQ6fOkzxUyI8aWUKa4vZWwIVWUGRtRxYbgLb68TPyz+lXduz2vNdwijTIcwTGcggcX0IAbaIIPDAQ8wyu8OY/Oi/PufMxbS04xcwh/4Hz+AMysj/M=</latexit><latexit sha1_base64="ihgt0y/lts5uDQPDa3MALu8LZCM=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FLx4rGFtoY9lsJ+3SzSbuboQS+ie8eFDx6u/x5r9x2+agrQ8GHu/NMDMvTAXXxnW/ndLK6tr6RnmzsrW9s7tX3T+410mmGPosEYlqh1Sj4BJ9w43AdqqQxqHAVji6nvqtJ1SaJ/LOjFMMYjqQPOKMGiu1w4ecG4wnvWrNrbszkGXiFaQGBZq96le3n7AsRmmYoFp3PDc1QU6V4UzgpNLNNKaUjegAO5ZKGqMO8tm9E3JilT6JEmVLGjJTf0/kNNZ6HIe2M6ZmqBe9qfif18lMdBnkXKaZQcnmi6JMEJOQ6fOkzxUyI8aWUKa4vZWwIVWUGRtRxYbgLb68TPyz+lXduz2vNdwijTIcwTGcggcX0IAbaIIPDAQ8wyu8OY/Oi/PufMxbS04xcwh/4Hz+AMysj/M=</latexit><latexit sha1_base64="ihgt0y/lts5uDQPDa3MALu8LZCM=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FLx4rGFtoY9lsJ+3SzSbuboQS+ie8eFDx6u/x5r9x2+agrQ8GHu/NMDMvTAXXxnW/ndLK6tr6RnmzsrW9s7tX3T+410mmGPosEYlqh1Sj4BJ9w43AdqqQxqHAVji6nvqtJ1SaJ/LOjFMMYjqQPOKMGiu1w4ecG4wnvWrNrbszkGXiFaQGBZq96le3n7AsRmmYoFp3PDc1QU6V4UzgpNLNNKaUjegAO5ZKGqMO8tm9E3JilT6JEmVLGjJTf0/kNNZ6HIe2M6ZmqBe9qfif18lMdBnkXKaZQcnmi6JMEJOQ6fOkzxUyI8aWUKa4vZWwIVWUGRtRxYbgLb68TPyz+lXduz2vNdwijTIcwTGcggcX0IAbaIIPDAQ8wyu8OY/Oi/PufMxbS04xcwh/4Hz+AMysj/M=</latexit><latexit sha1_base64="ihgt0y/lts5uDQPDa3MALu8LZCM=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FLx4rGFtoY9lsJ+3SzSbuboQS+ie8eFDx6u/x5r9x2+agrQ8GHu/NMDMvTAXXxnW/ndLK6tr6RnmzsrW9s7tX3T+410mmGPosEYlqh1Sj4BJ9w43AdqqQxqHAVji6nvqtJ1SaJ/LOjFMMYjqQPOKMGiu1w4ecG4wnvWrNrbszkGXiFaQGBZq96le3n7AsRmmYoFp3PDc1QU6V4UzgpNLNNKaUjegAO5ZKGqMO8tm9E3JilT6JEmVLGjJTf0/kNNZ6HIe2M6ZmqBe9qfif18lMdBnkXKaZQcnmi6JMEJOQ6fOkzxUyI8aWUKa4vZWwIVWUGRtRxYbgLb68TPyz+lXduz2vNdwijTIcwTGcggcX0IAbaIIPDAQ8wyu8OY/Oi/PufMxbS04xcwh/4Hz+AMysj/M=</latexit> min ˆU, ˆV ,buser,bitem X (i,j)2⌦ (Mij µ buser i bitem j ˆUi ˆV † j )2 + U k ˆUk2 F + V k ˆV k2 F + buser kbuser k2 2 + bitem kbitem k2 2<latexit sha1_base64="utwFR+JySkMZlz+0CF9UitgaWsY=">AAADEHicbVJNj9MwEHXC11I+tgtHLiMqlq7YrZJqJeC2EhLiglgk0l2pbiPHdVN34ySyHaTK5C9w4a9w4QCIK0du/BucNFlguyNZfp73xjNjT5QnXGnP++24V65eu35j62bn1u07d7e7O/dGKiskZQHNkkyeRkSxhKcs0Fwn7DSXjIgoYSfR2YuKP3nPpOJZ+k6vcjYRJE75nFOirSvccR5jwdPQ4AXRJij3oQYjC6KpKRSTa8Q1EyWUsAuAVSFC0+f7sNwDzFPAbwSLSQlg+q9Dw5clHAAWBdgtCnlzS31YthcdQJMv5G1CS+IZiWMmYW86LAHjzu4TwIltZUbCAPCHNsbC8OV0CH/Z0Tk7uoQ1bScVd95WpRtu6OryWt261kYXdnvewKsNNoHfgB5q7Djs/sKzjBaCpZomRKmx7+V6YojUnCas7GBbRk7oGYnZ2MKUCKYmpv7REh5ZzwzmmbQr1VB7/40wRCi1EpFVCqIX6iJXOS/jxoWeP5sYnuaFZildJ5oXCegMqvGAGZeM6mRlAaGS21qBLogkVNsh6thH8C+2vAmC4eD5wH972Ds6bF5jCz1AD1Ef+egpOkKv0DEKEHU+Op+dr84395P7xf3u/lhLXaeJuY/+M/fnH3m99G4=</latexit><latexit sha1_base64="utwFR+JySkMZlz+0CF9UitgaWsY=">AAADEHicbVJNj9MwEHXC11I+tgtHLiMqlq7YrZJqJeC2EhLiglgk0l2pbiPHdVN34ySyHaTK5C9w4a9w4QCIK0du/BucNFlguyNZfp73xjNjT5QnXGnP++24V65eu35j62bn1u07d7e7O/dGKiskZQHNkkyeRkSxhKcs0Fwn7DSXjIgoYSfR2YuKP3nPpOJZ+k6vcjYRJE75nFOirSvccR5jwdPQ4AXRJij3oQYjC6KpKRSTa8Q1EyWUsAuAVSFC0+f7sNwDzFPAbwSLSQlg+q9Dw5clHAAWBdgtCnlzS31YthcdQJMv5G1CS+IZiWMmYW86LAHjzu4TwIltZUbCAPCHNsbC8OV0CH/Z0Tk7uoQ1bScVd95WpRtu6OryWt261kYXdnvewKsNNoHfgB5q7Djs/sKzjBaCpZomRKmx7+V6YojUnCas7GBbRk7oGYnZ2MKUCKYmpv7REh5ZzwzmmbQr1VB7/40wRCi1EpFVCqIX6iJXOS/jxoWeP5sYnuaFZildJ5oXCegMqvGAGZeM6mRlAaGS21qBLogkVNsh6thH8C+2vAmC4eD5wH972Ds6bF5jCz1AD1Ef+egpOkKv0DEKEHU+Op+dr84395P7xf3u/lhLXaeJuY/+M/fnH3m99G4=</latexit><latexit sha1_base64="utwFR+JySkMZlz+0CF9UitgaWsY=">AAADEHicbVJNj9MwEHXC11I+tgtHLiMqlq7YrZJqJeC2EhLiglgk0l2pbiPHdVN34ySyHaTK5C9w4a9w4QCIK0du/BucNFlguyNZfp73xjNjT5QnXGnP++24V65eu35j62bn1u07d7e7O/dGKiskZQHNkkyeRkSxhKcs0Fwn7DSXjIgoYSfR2YuKP3nPpOJZ+k6vcjYRJE75nFOirSvccR5jwdPQ4AXRJij3oQYjC6KpKRSTa8Q1EyWUsAuAVSFC0+f7sNwDzFPAbwSLSQlg+q9Dw5clHAAWBdgtCnlzS31YthcdQJMv5G1CS+IZiWMmYW86LAHjzu4TwIltZUbCAPCHNsbC8OV0CH/Z0Tk7uoQ1bScVd95WpRtu6OryWt261kYXdnvewKsNNoHfgB5q7Djs/sKzjBaCpZomRKmx7+V6YojUnCas7GBbRk7oGYnZ2MKUCKYmpv7REh5ZzwzmmbQr1VB7/40wRCi1EpFVCqIX6iJXOS/jxoWeP5sYnuaFZildJ5oXCegMqvGAGZeM6mRlAaGS21qBLogkVNsh6thH8C+2vAmC4eD5wH972Ds6bF5jCz1AD1Ef+egpOkKv0DEKEHU+Op+dr84395P7xf3u/lhLXaeJuY/+M/fnH3m99G4=</latexit><latexit sha1_base64="utwFR+JySkMZlz+0CF9UitgaWsY=">AAADEHicbVJNj9MwEHXC11I+tgtHLiMqlq7YrZJqJeC2EhLiglgk0l2pbiPHdVN34ySyHaTK5C9w4a9w4QCIK0du/BucNFlguyNZfp73xjNjT5QnXGnP++24V65eu35j62bn1u07d7e7O/dGKiskZQHNkkyeRkSxhKcs0Fwn7DSXjIgoYSfR2YuKP3nPpOJZ+k6vcjYRJE75nFOirSvccR5jwdPQ4AXRJij3oQYjC6KpKRSTa8Q1EyWUsAuAVSFC0+f7sNwDzFPAbwSLSQlg+q9Dw5clHAAWBdgtCnlzS31YthcdQJMv5G1CS+IZiWMmYW86LAHjzu4TwIltZUbCAPCHNsbC8OV0CH/Z0Tk7uoQ1bScVd95WpRtu6OryWt261kYXdnvewKsNNoHfgB5q7Djs/sKzjBaCpZomRKmx7+V6YojUnCas7GBbRk7oGYnZ2MKUCKYmpv7REh5ZzwzmmbQr1VB7/40wRCi1EpFVCqIX6iJXOS/jxoWeP5sYnuaFZildJ5oXCegMqvGAGZeM6mRlAaGS21qBLogkVNsh6thH8C+2vAmC4eD5wH972Ds6bF5jCz1AD1Ef+egpOkKv0DEKEHU+Op+dr84395P7xf3u/lhLXaeJuY/+M/fnH3m99G4=</latexit>
  15. 15. THE ROLE OF BIAS TERMS IN BIASED-MF • The bias terms serve the role of normalization. • The bias terms serve to make the rating matrix r be zero mean. • Meanwhile, the Biased-MF without bias terms(=Funk SVD) also works quite well. • The RMSE performance on Netflix Dataset is similar to original recommendation engine of Netflix. • Therefore, knowing why Biased-MF works well is equivalent to
 knowing why Funk SVD works well. • Now, we will investigate why Funk SVD works well on low-rank matrices. !15
  16. 16. TABLE OF CONTENTS 1. Netflix Prize and Winner’s Algorithm 2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009) 3. Probabilistic Matrix Factorization (NIPS 2008) 4. Why We Use Alternating Minimization? 5. Overcoming Local Minima Problem of Alternating Minimization 6. Why Random Initialization Works?
  17. 17. OPTIMIZATION FORM OF FUNK SVD • Recall: Optimization form of Funk SVD • Probabilistic Matrix Factorization (PMF, NIPS’08) interprets Funk SVD 
 in the view point of posterior. • Funk SVD can be interpreted as an algorithm that finds the most probable low-rank matrices p and q when a star is observed. !17 arg min ˆU, ˆV X (i,j)2⌦ (Mij ˆUi ˆV † j )2 + U k ˆUk2 F + V k ˆV k2 F <latexit sha1_base64="QbHbHTDK8iJKyzMIsXFV+BYgY7g=">AAACjnicbVFdixMxFM2MX2tdtauPvlwsQhfXMlMWVlFxQZB9EVdwugtNO6SZdJpukhmSjFBi/o4/yDf/jWl3KuuuFwIn59xDbs6d1YIbmyS/o/jW7Tt37+3c7zzYffjocXfvychUjaYso5Wo9PmMGCa4YpnlVrDzWjMiZ4KdzS4+rvWz70wbXqlvdlWziSSl4nNOiQ1U3v2JiS4BS65yhxfEuswfwAaMPHjAppG56/MDWO4D5grwF8lK4gFc3332ueNLD6+gdYar35rz5dThgpQl0x72p8NgeQlYhMkKkmeAf2xNAeafpsMr6uivOtqqebeXDJJNwU2QtqCH2jrNu79wUdFGMmWpIMaM06S2E0e05VQw38GNYTWhF6Rk4wAVkcxM3CZPDy8CU8C80uEoCxv2qsMRacxKzkKnJHZhrmtr8n/auLHz1xPHVd1YpujlQ/NGgK1gvRwouGbUilUAhGoeZgW6IJpQG1bYCSGk1798E2TDwZtB+vWwd3zYprGDnqHnqI9SdISO0Qk6RRmi0W40jN5G7+K9+Ch+H3+4bI2j1vMU/VPxyR/Vt8QD</latexit><latexit sha1_base64="QbHbHTDK8iJKyzMIsXFV+BYgY7g=">AAACjnicbVFdixMxFM2MX2tdtauPvlwsQhfXMlMWVlFxQZB9EVdwugtNO6SZdJpukhmSjFBi/o4/yDf/jWl3KuuuFwIn59xDbs6d1YIbmyS/o/jW7Tt37+3c7zzYffjocXfvychUjaYso5Wo9PmMGCa4YpnlVrDzWjMiZ4KdzS4+rvWz70wbXqlvdlWziSSl4nNOiQ1U3v2JiS4BS65yhxfEuswfwAaMPHjAppG56/MDWO4D5grwF8lK4gFc3332ueNLD6+gdYar35rz5dThgpQl0x72p8NgeQlYhMkKkmeAf2xNAeafpsMr6uivOtqqebeXDJJNwU2QtqCH2jrNu79wUdFGMmWpIMaM06S2E0e05VQw38GNYTWhF6Rk4wAVkcxM3CZPDy8CU8C80uEoCxv2qsMRacxKzkKnJHZhrmtr8n/auLHz1xPHVd1YpujlQ/NGgK1gvRwouGbUilUAhGoeZgW6IJpQG1bYCSGk1798E2TDwZtB+vWwd3zYprGDnqHnqI9SdISO0Qk6RRmi0W40jN5G7+K9+Ch+H3+4bI2j1vMU/VPxyR/Vt8QD</latexit><latexit sha1_base64="QbHbHTDK8iJKyzMIsXFV+BYgY7g=">AAACjnicbVFdixMxFM2MX2tdtauPvlwsQhfXMlMWVlFxQZB9EVdwugtNO6SZdJpukhmSjFBi/o4/yDf/jWl3KuuuFwIn59xDbs6d1YIbmyS/o/jW7Tt37+3c7zzYffjocXfvychUjaYso5Wo9PmMGCa4YpnlVrDzWjMiZ4KdzS4+rvWz70wbXqlvdlWziSSl4nNOiQ1U3v2JiS4BS65yhxfEuswfwAaMPHjAppG56/MDWO4D5grwF8lK4gFc3332ueNLD6+gdYar35rz5dThgpQl0x72p8NgeQlYhMkKkmeAf2xNAeafpsMr6uivOtqqebeXDJJNwU2QtqCH2jrNu79wUdFGMmWpIMaM06S2E0e05VQw38GNYTWhF6Rk4wAVkcxM3CZPDy8CU8C80uEoCxv2qsMRacxKzkKnJHZhrmtr8n/auLHz1xPHVd1YpujlQ/NGgK1gvRwouGbUilUAhGoeZgW6IJpQG1bYCSGk1798E2TDwZtB+vWwd3zYprGDnqHnqI9SdISO0Qk6RRmi0W40jN5G7+K9+Ch+H3+4bI2j1vMU/VPxyR/Vt8QD</latexit><latexit sha1_base64="QbHbHTDK8iJKyzMIsXFV+BYgY7g=">AAACjnicbVFdixMxFM2MX2tdtauPvlwsQhfXMlMWVlFxQZB9EVdwugtNO6SZdJpukhmSjFBi/o4/yDf/jWl3KuuuFwIn59xDbs6d1YIbmyS/o/jW7Tt37+3c7zzYffjocXfvychUjaYso5Wo9PmMGCa4YpnlVrDzWjMiZ4KdzS4+rvWz70wbXqlvdlWziSSl4nNOiQ1U3v2JiS4BS65yhxfEuswfwAaMPHjAppG56/MDWO4D5grwF8lK4gFc3332ueNLD6+gdYar35rz5dThgpQl0x72p8NgeQlYhMkKkmeAf2xNAeafpsMr6uivOtqqebeXDJJNwU2QtqCH2jrNu79wUdFGMmWpIMaM06S2E0e05VQw38GNYTWhF6Rk4wAVkcxM3CZPDy8CU8C80uEoCxv2qsMRacxKzkKnJHZhrmtr8n/auLHz1xPHVd1YpujlQ/NGgK1gvRwouGbUilUAhGoeZgW6IJpQG1bYCSGk1798E2TDwZtB+vWwd3zYprGDnqHnqI9SdISO0Qk6RRmi0W40jN5G7+K9+Ch+H3+4bI2j1vMU/VPxyR/Vt8QD</latexit>
  18. 18. PROBABILISTIC MATRIX FACTORIZATION • Optimization form of PMF: • To find and via MAP(Maximum A Posteriori). • A low-rank assumption ( ) and Gaussian noise assumptions. !18 Mij i=1,…,m j=1,…,n σ σU σV arg max ˆU, ˆV Pr[ ˆU, ˆV |P⌦(M), ⌦, 2 , 2 U , 2 V ] <latexit sha1_base64="W0+3TbimD40TlGA8JBqmkMPUT+k=">AAACWXicbZFfa9swFMVlb2uz7E+z9nEvYmHQwQh2KGx7K/SlL2MpzGkhds21cuOISrYnXY8G119yL2OwrzKo4mbQpbsg9OOce5F0lFVKWgqCX57/6PGTnd3e0/6z5y9e7g1e7U9tWRuBkShVaS4ysKhkgRFJUnhRGQSdKTzPrk7W/vl3NFaWxVdaVZhoyAu5kALISengWwwm57GG67SJl0BN1L7nHUzblseE19RMTDvb9m74JI2/aMyBH35+5+SO3W5lruFy/JfS6B5PL8cJTwfDYBR0xR9CuIEh29QkHfyI56WoNRYkFFg7C4OKkgYMSaGw7ce1xQrEFeQ4c1iARps0XTQtf+uUOV+Uxq2CeKfen2hAW7vSmevUQEu77a3F/3mzmhYfk0YWVU1YiLuDFrXiVPJ1znwuDQpSKwcgjHR35WIJBgS53+i7EMLtJz+EaDz6NArPjobHR5s0euw1e8MOWcg+sGN2yiYsYoL9ZH+8HW/X++37fs/v37X63mbmgP1T/sEtCBayuA==</latexit><latexit sha1_base64="W0+3TbimD40TlGA8JBqmkMPUT+k=">AAACWXicbZFfa9swFMVlb2uz7E+z9nEvYmHQwQh2KGx7K/SlL2MpzGkhds21cuOISrYnXY8G119yL2OwrzKo4mbQpbsg9OOce5F0lFVKWgqCX57/6PGTnd3e0/6z5y9e7g1e7U9tWRuBkShVaS4ysKhkgRFJUnhRGQSdKTzPrk7W/vl3NFaWxVdaVZhoyAu5kALISengWwwm57GG67SJl0BN1L7nHUzblseE19RMTDvb9m74JI2/aMyBH35+5+SO3W5lruFy/JfS6B5PL8cJTwfDYBR0xR9CuIEh29QkHfyI56WoNRYkFFg7C4OKkgYMSaGw7ce1xQrEFeQ4c1iARps0XTQtf+uUOV+Uxq2CeKfen2hAW7vSmevUQEu77a3F/3mzmhYfk0YWVU1YiLuDFrXiVPJ1znwuDQpSKwcgjHR35WIJBgS53+i7EMLtJz+EaDz6NArPjobHR5s0euw1e8MOWcg+sGN2yiYsYoL9ZH+8HW/X++37fs/v37X63mbmgP1T/sEtCBayuA==</latexit><latexit sha1_base64="W0+3TbimD40TlGA8JBqmkMPUT+k=">AAACWXicbZFfa9swFMVlb2uz7E+z9nEvYmHQwQh2KGx7K/SlL2MpzGkhds21cuOISrYnXY8G119yL2OwrzKo4mbQpbsg9OOce5F0lFVKWgqCX57/6PGTnd3e0/6z5y9e7g1e7U9tWRuBkShVaS4ysKhkgRFJUnhRGQSdKTzPrk7W/vl3NFaWxVdaVZhoyAu5kALISengWwwm57GG67SJl0BN1L7nHUzblseE19RMTDvb9m74JI2/aMyBH35+5+SO3W5lruFy/JfS6B5PL8cJTwfDYBR0xR9CuIEh29QkHfyI56WoNRYkFFg7C4OKkgYMSaGw7ce1xQrEFeQ4c1iARps0XTQtf+uUOV+Uxq2CeKfen2hAW7vSmevUQEu77a3F/3mzmhYfk0YWVU1YiLuDFrXiVPJ1znwuDQpSKwcgjHR35WIJBgS53+i7EMLtJz+EaDz6NArPjobHR5s0euw1e8MOWcg+sGN2yiYsYoL9ZH+8HW/X++37fs/v37X63mbmgP1T/sEtCBayuA==</latexit><latexit sha1_base64="W0+3TbimD40TlGA8JBqmkMPUT+k=">AAACWXicbZFfa9swFMVlb2uz7E+z9nEvYmHQwQh2KGx7K/SlL2MpzGkhds21cuOISrYnXY8G119yL2OwrzKo4mbQpbsg9OOce5F0lFVKWgqCX57/6PGTnd3e0/6z5y9e7g1e7U9tWRuBkShVaS4ysKhkgRFJUnhRGQSdKTzPrk7W/vl3NFaWxVdaVZhoyAu5kALISengWwwm57GG67SJl0BN1L7nHUzblseE19RMTDvb9m74JI2/aMyBH35+5+SO3W5lruFy/JfS6B5PL8cJTwfDYBR0xR9CuIEh29QkHfyI56WoNRYkFFg7C4OKkgYMSaGw7ce1xQrEFeQ4c1iARps0XTQtf+uUOV+Uxq2CeKfen2hAW7vSmevUQEu77a3F/3mzmhYfk0YWVU1YiLuDFrXiVPJ1znwuDQpSKwcgjHR35WIJBgS53+i7EMLtJz+EaDz6NArPjobHR5s0euw1e8MOWcg+sGN2yiYsYoL9ZH+8HW/X++37fs/v37X63mbmgP1T/sEtCBayuA==</latexit> ˆUi<latexit sha1_base64="DmZux6dYard+s7S159poIGMBQ98=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoN4KXjxWMLbShrLZbtqlm03YnQgl9Fd48aDi1b/jzX/jts1BWx8MPN6bYWZemEph0HW/ndLa+sbmVnm7srO7t39QPTx6MEmmGfdZIhPdCanhUijuo0DJO6nmNA4lb4fjm5nffuLaiETd4yTlQUyHSkSCUbTSY29EMfenfdGv1ty6OwdZJV5BalCg1a9+9QYJy2KukElqTNdzUwxyqlEwyaeVXmZ4StmYDnnXUkVjboJ8fvCUnFllQKJE21JI5urviZzGxkzi0HbGFEdm2ZuJ/3ndDKOrIBcqzZArtlgUZZJgQmbfk4HQnKGcWEKZFvZWwkZUU4Y2o4oNwVt+eZX4jfp13bu7qDUbRRplOIFTOAcPLqEJt9ACHxjE8Ayv8OZo58V5dz4WrSWnmDmGP3A+fwBgoZBG</latexit><latexit sha1_base64="DmZux6dYard+s7S159poIGMBQ98=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoN4KXjxWMLbShrLZbtqlm03YnQgl9Fd48aDi1b/jzX/jts1BWx8MPN6bYWZemEph0HW/ndLa+sbmVnm7srO7t39QPTx6MEmmGfdZIhPdCanhUijuo0DJO6nmNA4lb4fjm5nffuLaiETd4yTlQUyHSkSCUbTSY29EMfenfdGv1ty6OwdZJV5BalCg1a9+9QYJy2KukElqTNdzUwxyqlEwyaeVXmZ4StmYDnnXUkVjboJ8fvCUnFllQKJE21JI5urviZzGxkzi0HbGFEdm2ZuJ/3ndDKOrIBcqzZArtlgUZZJgQmbfk4HQnKGcWEKZFvZWwkZUU4Y2o4oNwVt+eZX4jfp13bu7qDUbRRplOIFTOAcPLqEJt9ACHxjE8Ayv8OZo58V5dz4WrSWnmDmGP3A+fwBgoZBG</latexit><latexit sha1_base64="DmZux6dYard+s7S159poIGMBQ98=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoN4KXjxWMLbShrLZbtqlm03YnQgl9Fd48aDi1b/jzX/jts1BWx8MPN6bYWZemEph0HW/ndLa+sbmVnm7srO7t39QPTx6MEmmGfdZIhPdCanhUijuo0DJO6nmNA4lb4fjm5nffuLaiETd4yTlQUyHSkSCUbTSY29EMfenfdGv1ty6OwdZJV5BalCg1a9+9QYJy2KukElqTNdzUwxyqlEwyaeVXmZ4StmYDnnXUkVjboJ8fvCUnFllQKJE21JI5urviZzGxkzi0HbGFEdm2ZuJ/3ndDKOrIBcqzZArtlgUZZJgQmbfk4HQnKGcWEKZFvZWwkZUU4Y2o4oNwVt+eZX4jfp13bu7qDUbRRplOIFTOAcPLqEJt9ACHxjE8Ayv8OZo58V5dz4WrSWnmDmGP3A+fwBgoZBG</latexit><latexit sha1_base64="DmZux6dYard+s7S159poIGMBQ98=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoN4KXjxWMLbShrLZbtqlm03YnQgl9Fd48aDi1b/jzX/jts1BWx8MPN6bYWZemEph0HW/ndLa+sbmVnm7srO7t39QPTx6MEmmGfdZIhPdCanhUijuo0DJO6nmNA4lb4fjm5nffuLaiETd4yTlQUyHSkSCUbTSY29EMfenfdGv1ty6OwdZJV5BalCg1a9+9QYJy2KukElqTNdzUwxyqlEwyaeVXmZ4StmYDnnXUkVjboJ8fvCUnFllQKJE21JI5urviZzGxkzi0HbGFEdm2ZuJ/3ndDKOrIBcqzZArtlgUZZJgQmbfk4HQnKGcWEKZFvZWwkZUU4Y2o4oNwVt+eZX4jfp13bu7qDUbRRplOIFTOAcPLqEJt9ACHxjE8Ayv8OZo58V5dz4WrSWnmDmGP3A+fwBgoZBG</latexit> ˆVj<latexit sha1_base64="H+66y9gu8sYQVMmwNQcT8xMRc3A=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoN4KXjxWMG2lDWWz3bZrd5OwOxFK6K/w4kHFq3/Hm//GbZuDtj4YeLw3w8y8MJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUNHGqGfdZLGPdDqnhUkTcR4GStxPNqQolb4Xjm5nfeuLaiDi6x0nCA0WHkRgIRtFKD90Rxaw57T32yhW36s5BVomXkwrkaPTKX91+zFLFI2SSGtPx3ASDjGoUTPJpqZsanlA2pkPesTSiipsgmx88JWdW6ZNBrG1FSObq74mMKmMmKrSdiuLILHsz8T+vk+LgKshElKTII7ZYNEglwZjMvid9oTlDObGEMi3srYSNqKYMbUYlG4K3/PIq8WvV66p3d1Gp1/I0inACp3AOHlxCHW6hAT4wUPAMr/DmaOfFeXc+Fq0FJ585hj9wPn8AY6qQSA==</latexit><latexit sha1_base64="H+66y9gu8sYQVMmwNQcT8xMRc3A=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoN4KXjxWMG2lDWWz3bZrd5OwOxFK6K/w4kHFq3/Hm//GbZuDtj4YeLw3w8y8MJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUNHGqGfdZLGPdDqnhUkTcR4GStxPNqQolb4Xjm5nfeuLaiDi6x0nCA0WHkRgIRtFKD90Rxaw57T32yhW36s5BVomXkwrkaPTKX91+zFLFI2SSGtPx3ASDjGoUTPJpqZsanlA2pkPesTSiipsgmx88JWdW6ZNBrG1FSObq74mMKmMmKrSdiuLILHsz8T+vk+LgKshElKTII7ZYNEglwZjMvid9oTlDObGEMi3srYSNqKYMbUYlG4K3/PIq8WvV66p3d1Gp1/I0inACp3AOHlxCHW6hAT4wUPAMr/DmaOfFeXc+Fq0FJ585hj9wPn8AY6qQSA==</latexit><latexit sha1_base64="H+66y9gu8sYQVMmwNQcT8xMRc3A=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoN4KXjxWMG2lDWWz3bZrd5OwOxFK6K/w4kHFq3/Hm//GbZuDtj4YeLw3w8y8MJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUNHGqGfdZLGPdDqnhUkTcR4GStxPNqQolb4Xjm5nfeuLaiDi6x0nCA0WHkRgIRtFKD90Rxaw57T32yhW36s5BVomXkwrkaPTKX91+zFLFI2SSGtPx3ASDjGoUTPJpqZsanlA2pkPesTSiipsgmx88JWdW6ZNBrG1FSObq74mMKmMmKrSdiuLILHsz8T+vk+LgKshElKTII7ZYNEglwZjMvid9oTlDObGEMi3srYSNqKYMbUYlG4K3/PIq8WvV66p3d1Gp1/I0inACp3AOHlxCHW6hAT4wUPAMr/DmaOfFeXc+Fq0FJ585hj9wPn8AY6qQSA==</latexit><latexit sha1_base64="H+66y9gu8sYQVMmwNQcT8xMRc3A=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoN4KXjxWMG2lDWWz3bZrd5OwOxFK6K/w4kHFq3/Hm//GbZuDtj4YeLw3w8y8MJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUNHGqGfdZLGPdDqnhUkTcR4GStxPNqQolb4Xjm5nfeuLaiDi6x0nCA0WHkRgIRtFKD90Rxaw57T32yhW36s5BVomXkwrkaPTKX91+zFLFI2SSGtPx3ASDjGoUTPJpqZsanlA2pkPesTSiipsgmx88JWdW6ZNBrG1FSObq74mMKmMmKrSdiuLILHsz8T+vk+LgKshElKTII7ZYNEglwZjMvid9oTlDObGEMi3srYSNqKYMbUYlG4K3/PIq8WvV66p3d1Gp1/I0inACp3AOHlxCHW6hAT4wUPAMr/DmaOfFeXc+Fq0FJ585hj9wPn8AY6qQSA==</latexit> Mij = ˆUi ˆV † j<latexit sha1_base64="/xMSLN6PbfmwT3kOQ+z4TtUfAnI=">AAACDHicbZDLSsNAFIYn9VbrLerSzWAVXJWkCOpCKLhxI1QwbaGJYTKZpNNOLsxMhBLyAm58FTcuVNz6AO58G6dtFtr6w8DHf87hzPm9lFEhDeNbqywtr6yuVddrG5tb2zv67l5HJBnHxMIJS3jPQ4IwGhNLUslIL+UERR4jXW90Nal3HwgXNInv5DglToTCmAYUI6ksVz+6cXM6LOAltAdI5lbh0hl1Cnd4b/soDAl39brRMKaCi2CWUAel2q7+ZfsJziISS8yQEH3TSKWTIy4pZqSo2ZkgKcIjFJK+whhFRDj59JoCHivHh0HC1YslnLq/J3IUCTGOPNUZITkQ87WJ+V+tn8ng3MlpnGaSxHi2KMgYlAmcRAN9ygmWbKwAYU7VXyEeII6wVAHWVAjm/MmLYDUbFw3z9rTeapZpVMEBOAQnwARnoAWuQRtYAINH8AxewZv2pL1o79rHrLWilTP74I+0zx+WTJtt</latexit><latexit sha1_base64="/xMSLN6PbfmwT3kOQ+z4TtUfAnI=">AAACDHicbZDLSsNAFIYn9VbrLerSzWAVXJWkCOpCKLhxI1QwbaGJYTKZpNNOLsxMhBLyAm58FTcuVNz6AO58G6dtFtr6w8DHf87hzPm9lFEhDeNbqywtr6yuVddrG5tb2zv67l5HJBnHxMIJS3jPQ4IwGhNLUslIL+UERR4jXW90Nal3HwgXNInv5DglToTCmAYUI6ksVz+6cXM6LOAltAdI5lbh0hl1Cnd4b/soDAl39brRMKaCi2CWUAel2q7+ZfsJziISS8yQEH3TSKWTIy4pZqSo2ZkgKcIjFJK+whhFRDj59JoCHivHh0HC1YslnLq/J3IUCTGOPNUZITkQ87WJ+V+tn8ng3MlpnGaSxHi2KMgYlAmcRAN9ygmWbKwAYU7VXyEeII6wVAHWVAjm/MmLYDUbFw3z9rTeapZpVMEBOAQnwARnoAWuQRtYAINH8AxewZv2pL1o79rHrLWilTP74I+0zx+WTJtt</latexit><latexit sha1_base64="/xMSLN6PbfmwT3kOQ+z4TtUfAnI=">AAACDHicbZDLSsNAFIYn9VbrLerSzWAVXJWkCOpCKLhxI1QwbaGJYTKZpNNOLsxMhBLyAm58FTcuVNz6AO58G6dtFtr6w8DHf87hzPm9lFEhDeNbqywtr6yuVddrG5tb2zv67l5HJBnHxMIJS3jPQ4IwGhNLUslIL+UERR4jXW90Nal3HwgXNInv5DglToTCmAYUI6ksVz+6cXM6LOAltAdI5lbh0hl1Cnd4b/soDAl39brRMKaCi2CWUAel2q7+ZfsJziISS8yQEH3TSKWTIy4pZqSo2ZkgKcIjFJK+whhFRDj59JoCHivHh0HC1YslnLq/J3IUCTGOPNUZITkQ87WJ+V+tn8ng3MlpnGaSxHi2KMgYlAmcRAN9ygmWbKwAYU7VXyEeII6wVAHWVAjm/MmLYDUbFw3z9rTeapZpVMEBOAQnwARnoAWuQRtYAINH8AxewZv2pL1o79rHrLWilTP74I+0zx+WTJtt</latexit><latexit sha1_base64="/xMSLN6PbfmwT3kOQ+z4TtUfAnI=">AAACDHicbZDLSsNAFIYn9VbrLerSzWAVXJWkCOpCKLhxI1QwbaGJYTKZpNNOLsxMhBLyAm58FTcuVNz6AO58G6dtFtr6w8DHf87hzPm9lFEhDeNbqywtr6yuVddrG5tb2zv67l5HJBnHxMIJS3jPQ4IwGhNLUslIL+UERR4jXW90Nal3HwgXNInv5DglToTCmAYUI6ksVz+6cXM6LOAltAdI5lbh0hl1Cnd4b/soDAl39brRMKaCi2CWUAel2q7+ZfsJziISS8yQEH3TSKWTIy4pZqSo2ZkgKcIjFJK+whhFRDj59JoCHivHh0HC1YslnLq/J3IUCTGOPNUZITkQ87WJ+V+tn8ng3MlpnGaSxHi2KMgYlAmcRAN9ygmWbKwAYU7VXyEeII6wVAHWVAjm/MmLYDUbFw3z9rTeapZpVMEBOAQnwARnoAWuQRtYAINH8AxewZv2pL1o79rHrLWilTP74I+0zx+WTJtt</latexit> Pr[M| ˆU, ˆV , 2 ] = mY i=1 nY j=1 [N[Mij| ˆUi ˆV † j , 2 ]]I⌦ ij , Pr[ ˆU| 2 U ] = mY i=1 N[ ˆUi|0, 2 U I], Pr[ ˆV | 2 V ] = nY j=1 N[ ˆVj|0, 2 V I], I⌦ ij = ⇢ 1 (i, j) 2 ⌦ 0 otherwise.<latexit sha1_base64="4vyAlKNdd7n8kEzBLhheSN736jk=">AAAD7XicdVPLbhMxFJ1OCpTwaApLNldEVEWqopkKCVhUqsQGFoUgkbRSPBl5HGfixPOQ7UAjZz6CDQtAbPkfdvwN9mSSpk2xNNK5j3POvZYnyjmTyvP+brm17Vu37+zcrd+7/+DhbmPvUVdmU0Foh2Q8E+cRlpSzlHYUU5ye54LiJOL0LJq8sfWzz1RIlqWf1CynQYLjlA0Zwcqkwj13Gyl6oXRbFL3TORphpTvFIZSga4FkcYL7RwHsHwPKRTbo66QINTv2i2Wcmnhs4x6gBKsRwVy/N9GpaRsXMIdKNmRL3XDcRwMcx1TAukXQ1yuBd0UffUhojEsROwmqX45aKc4X3LDzvwHXx7mcYg7e0tZSF21RZDyDm3y6K5/uVZ/V4ps+dscrPt1NH6ivbVuuWa1cWnA6VEjXAVBEY5ZqLASeFZrzwuQAfIB9OGCHMH4OiKVQMc30tuqZIkqi7EJnakTFFyZpy/IQTQeVUh0JFo9Uy1LCRtNreeWBTeBXoOlUpx02/qBBRqYJTRXhWMqe7+UqMMqKEU6N9lTSHJMJjmnPwBQnVAa6fK4FPDOZAQwzYb5UQZldZ2icSDlLItNp70der9nkTbXeVA1fBZql+VTRlCyMhlMOKgP79mHABCWKzwzARDAzK5ARFpgo84fYS/Cvr7wJOket1y3/44vmiVfdxo7zxHnqHDi+89I5cd46bafjEHfifnW/uz9qee1b7Wft16LV3ao4j50rp/b7Hw+5QzI=</latexit><latexit sha1_base64="4vyAlKNdd7n8kEzBLhheSN736jk=">AAAD7XicdVPLbhMxFJ1OCpTwaApLNldEVEWqopkKCVhUqsQGFoUgkbRSPBl5HGfixPOQ7UAjZz6CDQtAbPkfdvwN9mSSpk2xNNK5j3POvZYnyjmTyvP+brm17Vu37+zcrd+7/+DhbmPvUVdmU0Foh2Q8E+cRlpSzlHYUU5ye54LiJOL0LJq8sfWzz1RIlqWf1CynQYLjlA0Zwcqkwj13Gyl6oXRbFL3TORphpTvFIZSga4FkcYL7RwHsHwPKRTbo66QINTv2i2Wcmnhs4x6gBKsRwVy/N9GpaRsXMIdKNmRL3XDcRwMcx1TAukXQ1yuBd0UffUhojEsROwmqX45aKc4X3LDzvwHXx7mcYg7e0tZSF21RZDyDm3y6K5/uVZ/V4ps+dscrPt1NH6ivbVuuWa1cWnA6VEjXAVBEY5ZqLASeFZrzwuQAfIB9OGCHMH4OiKVQMc30tuqZIkqi7EJnakTFFyZpy/IQTQeVUh0JFo9Uy1LCRtNreeWBTeBXoOlUpx02/qBBRqYJTRXhWMqe7+UqMMqKEU6N9lTSHJMJjmnPwBQnVAa6fK4FPDOZAQwzYb5UQZldZ2icSDlLItNp70der9nkTbXeVA1fBZql+VTRlCyMhlMOKgP79mHABCWKzwzARDAzK5ARFpgo84fYS/Cvr7wJOket1y3/44vmiVfdxo7zxHnqHDi+89I5cd46bafjEHfifnW/uz9qee1b7Wft16LV3ao4j50rp/b7Hw+5QzI=</latexit><latexit sha1_base64="4vyAlKNdd7n8kEzBLhheSN736jk=">AAAD7XicdVPLbhMxFJ1OCpTwaApLNldEVEWqopkKCVhUqsQGFoUgkbRSPBl5HGfixPOQ7UAjZz6CDQtAbPkfdvwN9mSSpk2xNNK5j3POvZYnyjmTyvP+brm17Vu37+zcrd+7/+DhbmPvUVdmU0Foh2Q8E+cRlpSzlHYUU5ye54LiJOL0LJq8sfWzz1RIlqWf1CynQYLjlA0Zwcqkwj13Gyl6oXRbFL3TORphpTvFIZSga4FkcYL7RwHsHwPKRTbo66QINTv2i2Wcmnhs4x6gBKsRwVy/N9GpaRsXMIdKNmRL3XDcRwMcx1TAukXQ1yuBd0UffUhojEsROwmqX45aKc4X3LDzvwHXx7mcYg7e0tZSF21RZDyDm3y6K5/uVZ/V4ps+dscrPt1NH6ivbVuuWa1cWnA6VEjXAVBEY5ZqLASeFZrzwuQAfIB9OGCHMH4OiKVQMc30tuqZIkqi7EJnakTFFyZpy/IQTQeVUh0JFo9Uy1LCRtNreeWBTeBXoOlUpx02/qBBRqYJTRXhWMqe7+UqMMqKEU6N9lTSHJMJjmnPwBQnVAa6fK4FPDOZAQwzYb5UQZldZ2icSDlLItNp70der9nkTbXeVA1fBZql+VTRlCyMhlMOKgP79mHABCWKzwzARDAzK5ARFpgo84fYS/Cvr7wJOket1y3/44vmiVfdxo7zxHnqHDi+89I5cd46bafjEHfifnW/uz9qee1b7Wft16LV3ao4j50rp/b7Hw+5QzI=</latexit><latexit sha1_base64="4vyAlKNdd7n8kEzBLhheSN736jk=">AAAD7XicdVPLbhMxFJ1OCpTwaApLNldEVEWqopkKCVhUqsQGFoUgkbRSPBl5HGfixPOQ7UAjZz6CDQtAbPkfdvwN9mSSpk2xNNK5j3POvZYnyjmTyvP+brm17Vu37+zcrd+7/+DhbmPvUVdmU0Foh2Q8E+cRlpSzlHYUU5ye54LiJOL0LJq8sfWzz1RIlqWf1CynQYLjlA0Zwcqkwj13Gyl6oXRbFL3TORphpTvFIZSga4FkcYL7RwHsHwPKRTbo66QINTv2i2Wcmnhs4x6gBKsRwVy/N9GpaRsXMIdKNmRL3XDcRwMcx1TAukXQ1yuBd0UffUhojEsROwmqX45aKc4X3LDzvwHXx7mcYg7e0tZSF21RZDyDm3y6K5/uVZ/V4ps+dscrPt1NH6ivbVuuWa1cWnA6VEjXAVBEY5ZqLASeFZrzwuQAfIB9OGCHMH4OiKVQMc30tuqZIkqi7EJnakTFFyZpy/IQTQeVUh0JFo9Uy1LCRtNreeWBTeBXoOlUpx02/qBBRqYJTRXhWMqe7+UqMMqKEU6N9lTSHJMJjmnPwBQnVAa6fK4FPDOZAQwzYb5UQZldZ2icSDlLItNp70der9nkTbXeVA1fBZql+VTRlCyMhlMOKgP79mHABCWKzwzARDAzK5ARFpgo84fYS/Cvr7wJOket1y3/44vmiVfdxo7zxHnqHDi+89I5cd46bafjEHfifnW/uz9qee1b7Wft16LV3ao4j50rp/b7Hw+5QzI=</latexit> ̂U ̂V
  19. 19. THE LOG POSTERIORI OF PMF • The log posteriori of PMF is as follows: • Hence, the following equation holds: !19 arg max ˆU, ˆV Pr[ ˆU, ˆV |P⌦(M), ⌦, 2 , 2 U , 2 V ] = arg min ˆU, ˆV X (i,j)2⌦ (Mij ˆUi ˆV † j )2 + U k ˆUk2 F + V k ˆV k2 F <latexit sha1_base64="XlBMvMAdTTRcsp7OF0gzCIi5a5E=">AAADAHicbZJNj9MwEIad8LWUry4cOHAZUbFqxVKlERJwQFoJCXFZUSSSXalOLdd1U3fjJLIdtFU2XPgrXDjAaq/8DG78G9xuikrLSFYeve+Mx554lCdCG8/77bhXrl67fmPnZuPW7Tt37zV374c6KxTjAcuSTB2PqOaJSHlghEn4ca44laOEH41O3iz8o09caZGlH80855GkcSomglFjJbLrPNzDVMWAJT0lJZ5SUwbVPiwhrCrAhp+asq+qwaZ3Bn2C30seU2gfdqy8ZPvVIpZ06K+IBGscDv0IMG683oO6rUi324LtqwtJyrbYh1kHsEjr7SuAsn1ISjGr4BnUdUSsCslsiMc0jrmCztCv4CngxM5ibA8B+GyVb5G8HfprbvjXDVcuaba8rrcM2IZeDS1UR580f+FxxgrJU8MSqvWg5+UmKqkygiW8auBC85yyExrzgcWUSq6jcvkHK3hilTFMMmVXamCprleUVGo9lyObKamZ6k1vIf7PGxRm8jIqRZoXhqfsstGkSMBksHgOMBaKM5PMLVCmhD0rsClVlBn7aBp2CL3NK29D4HdfdXsfnrcO/HoaO+gReozaqIdeoAP0DvVRgJjz2fnqfHd+uF/cb+65e3GZ6jp1zQP0T7g//wA+U+2w</latexit><latexit sha1_base64="XlBMvMAdTTRcsp7OF0gzCIi5a5E=">AAADAHicbZJNj9MwEIad8LWUry4cOHAZUbFqxVKlERJwQFoJCXFZUSSSXalOLdd1U3fjJLIdtFU2XPgrXDjAaq/8DG78G9xuikrLSFYeve+Mx554lCdCG8/77bhXrl67fmPnZuPW7Tt37zV374c6KxTjAcuSTB2PqOaJSHlghEn4ca44laOEH41O3iz8o09caZGlH80855GkcSomglFjJbLrPNzDVMWAJT0lJZ5SUwbVPiwhrCrAhp+asq+qwaZ3Bn2C30seU2gfdqy8ZPvVIpZ06K+IBGscDv0IMG683oO6rUi324LtqwtJyrbYh1kHsEjr7SuAsn1ISjGr4BnUdUSsCslsiMc0jrmCztCv4CngxM5ibA8B+GyVb5G8HfprbvjXDVcuaba8rrcM2IZeDS1UR580f+FxxgrJU8MSqvWg5+UmKqkygiW8auBC85yyExrzgcWUSq6jcvkHK3hilTFMMmVXamCprleUVGo9lyObKamZ6k1vIf7PGxRm8jIqRZoXhqfsstGkSMBksHgOMBaKM5PMLVCmhD0rsClVlBn7aBp2CL3NK29D4HdfdXsfnrcO/HoaO+gReozaqIdeoAP0DvVRgJjz2fnqfHd+uF/cb+65e3GZ6jp1zQP0T7g//wA+U+2w</latexit><latexit sha1_base64="XlBMvMAdTTRcsp7OF0gzCIi5a5E=">AAADAHicbZJNj9MwEIad8LWUry4cOHAZUbFqxVKlERJwQFoJCXFZUSSSXalOLdd1U3fjJLIdtFU2XPgrXDjAaq/8DG78G9xuikrLSFYeve+Mx554lCdCG8/77bhXrl67fmPnZuPW7Tt37zV374c6KxTjAcuSTB2PqOaJSHlghEn4ca44laOEH41O3iz8o09caZGlH80855GkcSomglFjJbLrPNzDVMWAJT0lJZ5SUwbVPiwhrCrAhp+asq+qwaZ3Bn2C30seU2gfdqy8ZPvVIpZ06K+IBGscDv0IMG683oO6rUi324LtqwtJyrbYh1kHsEjr7SuAsn1ISjGr4BnUdUSsCslsiMc0jrmCztCv4CngxM5ibA8B+GyVb5G8HfprbvjXDVcuaba8rrcM2IZeDS1UR580f+FxxgrJU8MSqvWg5+UmKqkygiW8auBC85yyExrzgcWUSq6jcvkHK3hilTFMMmVXamCprleUVGo9lyObKamZ6k1vIf7PGxRm8jIqRZoXhqfsstGkSMBksHgOMBaKM5PMLVCmhD0rsClVlBn7aBp2CL3NK29D4HdfdXsfnrcO/HoaO+gReozaqIdeoAP0DvVRgJjz2fnqfHd+uF/cb+65e3GZ6jp1zQP0T7g//wA+U+2w</latexit><latexit sha1_base64="XlBMvMAdTTRcsp7OF0gzCIi5a5E=">AAADAHicbZJNj9MwEIad8LWUry4cOHAZUbFqxVKlERJwQFoJCXFZUSSSXalOLdd1U3fjJLIdtFU2XPgrXDjAaq/8DG78G9xuikrLSFYeve+Mx554lCdCG8/77bhXrl67fmPnZuPW7Tt37zV374c6KxTjAcuSTB2PqOaJSHlghEn4ca44laOEH41O3iz8o09caZGlH80855GkcSomglFjJbLrPNzDVMWAJT0lJZ5SUwbVPiwhrCrAhp+asq+qwaZ3Bn2C30seU2gfdqy8ZPvVIpZ06K+IBGscDv0IMG683oO6rUi324LtqwtJyrbYh1kHsEjr7SuAsn1ISjGr4BnUdUSsCslsiMc0jrmCztCv4CngxM5ibA8B+GyVb5G8HfprbvjXDVcuaba8rrcM2IZeDS1UR580f+FxxgrJU8MSqvWg5+UmKqkygiW8auBC85yyExrzgcWUSq6jcvkHK3hilTFMMmVXamCprleUVGo9lyObKamZ6k1vIf7PGxRm8jIqRZoXhqfsstGkSMBksHgOMBaKM5PMLVCmhD0rsClVlBn7aBp2CL3NK29D4HdfdXsfnrcO/HoaO+gReozaqIdeoAP0DvVRgJjz2fnqfHd+uF/cb+65e3GZ6jp1zQP0T7g//wA+U+2w</latexit> ln Pr[ ˆU, ˆV |P⌦(M), ⌦, 2 , 2 U , 2 V ] = 1 2 2 mX i=1 nX j=1 I⌦ ij(Mij ˆUi ˆV † j )2 1 2 2 U mX i=1 ˆU† i ˆUi 1 2 2 V nX j=1 ˆV † j ˆVj 1 2 (( mX i=1 nX j=1 I⌦ ij) ln 2 + mk ln 2 U + nk ln 2 V ) + C <latexit sha1_base64="FacZlGnfamKX+f7FRk0QGJSESm8=">AAAD2HicnVNNj9MwEE0TPpbysV04chlRUbVaWjUREnBYaaW9wAFRJNJdqWki13VTt3ESxQ6i8ubAARBXfho3fgV/ASdtttluT4wU+fmNZ+bNxJ7EAeWi3/9T041bt+/cPbhXv//g4aPDxtHjIY/SBBMbR0GUXEwQJwENiS2oCMhFnBDEJgE5nyzPcv/5Z5JwGoWfxComY4b8kM4oRkJR3lHtb8sJQnAE+SLkIMlGzhwJaWcvoADD7BIGnvOBER9B+31H0QVWK6c+Q65VIs+u4KFrjcFx6ict6EppZuBESgRI6yoMMkXylHmSnpiZK1m5XRTbUG0ZEnOMAvkuc9dF1dlFplSs1y5spHq01OotXGeKfJ8kHdeqd0GaV4W3IvcULvOU0ZXM+5IMK0m2cnclbJl8Eq1qJkt1Ae3/ax86UPywco7HwGBZofIWFRleJ5VkFXgMZ+A1mv1evzC4CcwNaGobG3iN3840wikjocAB4nxk9mMxligRFAckqzspJzHCS+STkYIhYoSPZXEzM3iumCnMokR9oYCCrUZIxDhfsYk6mffLd305uc83SsXs9VjSME4FCfG60CwNQESQX3OY0oRgEawUQDihSivgOUoQFuox1NUQzN2WbwLb6r3pmR9fNk+tzTQOtKfaM62tmdor7VR7qw00W8P6UL/Uv+nfjZHx1fhh/Fwf1WubmCfaNTN+/QMTIjkx</latexit><latexit sha1_base64="FacZlGnfamKX+f7FRk0QGJSESm8=">AAAD2HicnVNNj9MwEE0TPpbysV04chlRUbVaWjUREnBYaaW9wAFRJNJdqWki13VTt3ESxQ6i8ubAARBXfho3fgV/ASdtttluT4wU+fmNZ+bNxJ7EAeWi3/9T041bt+/cPbhXv//g4aPDxtHjIY/SBBMbR0GUXEwQJwENiS2oCMhFnBDEJgE5nyzPcv/5Z5JwGoWfxComY4b8kM4oRkJR3lHtb8sJQnAE+SLkIMlGzhwJaWcvoADD7BIGnvOBER9B+31H0QVWK6c+Q65VIs+u4KFrjcFx6ict6EppZuBESgRI6yoMMkXylHmSnpiZK1m5XRTbUG0ZEnOMAvkuc9dF1dlFplSs1y5spHq01OotXGeKfJ8kHdeqd0GaV4W3IvcULvOU0ZXM+5IMK0m2cnclbJl8Eq1qJkt1Ae3/ax86UPywco7HwGBZofIWFRleJ5VkFXgMZ+A1mv1evzC4CcwNaGobG3iN3840wikjocAB4nxk9mMxligRFAckqzspJzHCS+STkYIhYoSPZXEzM3iumCnMokR9oYCCrUZIxDhfsYk6mffLd305uc83SsXs9VjSME4FCfG60CwNQESQX3OY0oRgEawUQDihSivgOUoQFuox1NUQzN2WbwLb6r3pmR9fNk+tzTQOtKfaM62tmdor7VR7qw00W8P6UL/Uv+nfjZHx1fhh/Fwf1WubmCfaNTN+/QMTIjkx</latexit><latexit sha1_base64="FacZlGnfamKX+f7FRk0QGJSESm8=">AAAD2HicnVNNj9MwEE0TPpbysV04chlRUbVaWjUREnBYaaW9wAFRJNJdqWki13VTt3ESxQ6i8ubAARBXfho3fgV/ASdtttluT4wU+fmNZ+bNxJ7EAeWi3/9T041bt+/cPbhXv//g4aPDxtHjIY/SBBMbR0GUXEwQJwENiS2oCMhFnBDEJgE5nyzPcv/5Z5JwGoWfxComY4b8kM4oRkJR3lHtb8sJQnAE+SLkIMlGzhwJaWcvoADD7BIGnvOBER9B+31H0QVWK6c+Q65VIs+u4KFrjcFx6ict6EppZuBESgRI6yoMMkXylHmSnpiZK1m5XRTbUG0ZEnOMAvkuc9dF1dlFplSs1y5spHq01OotXGeKfJ8kHdeqd0GaV4W3IvcULvOU0ZXM+5IMK0m2cnclbJl8Eq1qJkt1Ae3/ax86UPywco7HwGBZofIWFRleJ5VkFXgMZ+A1mv1evzC4CcwNaGobG3iN3840wikjocAB4nxk9mMxligRFAckqzspJzHCS+STkYIhYoSPZXEzM3iumCnMokR9oYCCrUZIxDhfsYk6mffLd305uc83SsXs9VjSME4FCfG60CwNQESQX3OY0oRgEawUQDihSivgOUoQFuox1NUQzN2WbwLb6r3pmR9fNk+tzTQOtKfaM62tmdor7VR7qw00W8P6UL/Uv+nfjZHx1fhh/Fwf1WubmCfaNTN+/QMTIjkx</latexit><latexit sha1_base64="FacZlGnfamKX+f7FRk0QGJSESm8=">AAAD2HicnVNNj9MwEE0TPpbysV04chlRUbVaWjUREnBYaaW9wAFRJNJdqWki13VTt3ESxQ6i8ubAARBXfho3fgV/ASdtttluT4wU+fmNZ+bNxJ7EAeWi3/9T041bt+/cPbhXv//g4aPDxtHjIY/SBBMbR0GUXEwQJwENiS2oCMhFnBDEJgE5nyzPcv/5Z5JwGoWfxComY4b8kM4oRkJR3lHtb8sJQnAE+SLkIMlGzhwJaWcvoADD7BIGnvOBER9B+31H0QVWK6c+Q65VIs+u4KFrjcFx6ict6EppZuBESgRI6yoMMkXylHmSnpiZK1m5XRTbUG0ZEnOMAvkuc9dF1dlFplSs1y5spHq01OotXGeKfJ8kHdeqd0GaV4W3IvcULvOU0ZXM+5IMK0m2cnclbJl8Eq1qJkt1Ae3/ax86UPywco7HwGBZofIWFRleJ5VkFXgMZ+A1mv1evzC4CcwNaGobG3iN3840wikjocAB4nxk9mMxligRFAckqzspJzHCS+STkYIhYoSPZXEzM3iumCnMokR9oYCCrUZIxDhfsYk6mffLd305uc83SsXs9VjSME4FCfG60CwNQESQX3OY0oRgEawUQDihSivgOUoQFuox1NUQzN2WbwLb6r3pmR9fNk+tzTQOtKfaM62tmdor7VR7qw00W8P6UL/Uv+nfjZHx1fhh/Fwf1WubmCfaNTN+/QMTIjkx</latexit>
  20. 20. TABLE OF CONTENTS 1. Netflix Prize and Winner’s Algorithm 2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009) 3. Probabilistic Matrix Factorization (NIPS 2008) 4. Why We Use Alternating Minimization? 5. Overcoming Local Minima Problem of Alternating Minimization 6. Why Random Initialization Works?
  21. 21. WHY WE USE ALTERNATING MINIMIZATION? • We can just use Gradient Descent with random initialization without alternating minimization. • Why we should use alternating minimization? • First of all, Gradient Descent can oscillate without alternating minimization. • The MRMA paper(NIPS’17) said that it is possible to overfit without alternating minimization. • Alternating Minimization is not the only way to avoid overfitting. • However, Besag(1986) said that it is a good way to avoid overfitting. • Glendinning(1989) said that alternating minimization is robust to initial point empirically. • However, the initial point is nonetheless important. • Because it can fall into the local minima. !21
  22. 22. TABLE OF CONTENTS 1. Netflix Prize and Winner’s Algorithm 2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009) 3. Probabilistic Matrix Factorization (NIPS 2008) 4. Why We Use Alternating Minimization? 5. Overcoming Local Minima Problem of Alternating Minimization 6. Why Random Initialization Works?
  23. 23. LOCAL MINIMA PROBLEM OF AM • In otherwise, Jain et al(2013) showed that AM has no local minima problem. • Noiseless case ONLY. Exact Completion Setting. • With the slightly modified algorithm. • With some assumptions. • Let’s look at the modified algorithm first. !23
  24. 24. THE MODIFIED ALGORITHM !24
  25. 25. THE MODIFIED ALGORITHM !25 Mini-batch
  26. 26. THE MODIFIED ALGORITHM !26 Alternating Minimization similar to Funk SVD
  27. 27. THE MODIFIED ALGORITHM !27 When performing initialization using SVD + Clipping, the local optimum found by the SVD method
 is close enough to the global optimum!
  28. 28. MOTIVATION OF THE INCOHERENCE ASSUMPTION • Consider the rank-1 matrix M: • Let |Ω| be the number of observed entries of M. • Then, we can see only 0 with probability 1 - |Ω| / (mn). • If sample set doesn’t contain 1, we can not complete matrix exactly. • Therefore, it is impossible to recover all low-rank matrices. !28 M = e1e⇤ n = 2 6 6 6 4 0 0 · · · 0 1 0 0 · · · 0 0 ... ... ... ... ... 0 0 · · · 0 0 3 7 7 7 5 <latexit sha1_base64="BDwoBlvzuP9vRxmJB62/VzKKa7Y=">AAACp3ichVHfS9xAEN6kttrYH9f2sS+Dh6X4cCRFaH0QBF8EKZxg9OCSHpu9yd3iZhN2J8IR8rf1f/DN/8bNmYdWRT8Y+Pb7dmZnZ7JKSUtheOv5rzZev9ncehtsv3v/4ePg0+cLW9ZGYCxKVZpJxi0qqTEmSQonlUFeZAovs6vjzr+8RmNlqc9pVWFa8IWWuRScnDQb/P0Nh4CzyIX+swfBISQKc5pCk2S4kLrhxvBV24gObRDCN+gimZdks/4QQZI87YSdk1x3Ume9SJ6vg3retwMtJEYulpTOBsNwFK4Bj0nUkyHrMZ4NblxlUReoSShu7TQKK0pdXZJCYRsktcWKiyu+wKmjmhdo02Y96BZ2nTKHvDQuNMFa/Tej4YW1qyJzNwtOS/vQ68SnvGlN+a+0kbqqCbW4fyivFVAJ3dZgLg0KUitHuDDS9QpiyQ0X5HYbuCFED7/8mMQ/Rgej6Gx/eLTfT2OLfWU77DuL2E92xE7YmMVMeLveqXfuxf6eP/Yv/Mn9Vd/rc76w/+DzOybDwwE=</latexit><latexit sha1_base64="BDwoBlvzuP9vRxmJB62/VzKKa7Y=">AAACp3ichVHfS9xAEN6kttrYH9f2sS+Dh6X4cCRFaH0QBF8EKZxg9OCSHpu9yd3iZhN2J8IR8rf1f/DN/8bNmYdWRT8Y+Pb7dmZnZ7JKSUtheOv5rzZev9ncehtsv3v/4ePg0+cLW9ZGYCxKVZpJxi0qqTEmSQonlUFeZAovs6vjzr+8RmNlqc9pVWFa8IWWuRScnDQb/P0Nh4CzyIX+swfBISQKc5pCk2S4kLrhxvBV24gObRDCN+gimZdks/4QQZI87YSdk1x3Ume9SJ6vg3retwMtJEYulpTOBsNwFK4Bj0nUkyHrMZ4NblxlUReoSShu7TQKK0pdXZJCYRsktcWKiyu+wKmjmhdo02Y96BZ2nTKHvDQuNMFa/Tej4YW1qyJzNwtOS/vQ68SnvGlN+a+0kbqqCbW4fyivFVAJ3dZgLg0KUitHuDDS9QpiyQ0X5HYbuCFED7/8mMQ/Rgej6Gx/eLTfT2OLfWU77DuL2E92xE7YmMVMeLveqXfuxf6eP/Yv/Mn9Vd/rc76w/+DzOybDwwE=</latexit><latexit sha1_base64="BDwoBlvzuP9vRxmJB62/VzKKa7Y=">AAACp3ichVHfS9xAEN6kttrYH9f2sS+Dh6X4cCRFaH0QBF8EKZxg9OCSHpu9yd3iZhN2J8IR8rf1f/DN/8bNmYdWRT8Y+Pb7dmZnZ7JKSUtheOv5rzZev9ncehtsv3v/4ePg0+cLW9ZGYCxKVZpJxi0qqTEmSQonlUFeZAovs6vjzr+8RmNlqc9pVWFa8IWWuRScnDQb/P0Nh4CzyIX+swfBISQKc5pCk2S4kLrhxvBV24gObRDCN+gimZdks/4QQZI87YSdk1x3Ume9SJ6vg3retwMtJEYulpTOBsNwFK4Bj0nUkyHrMZ4NblxlUReoSShu7TQKK0pdXZJCYRsktcWKiyu+wKmjmhdo02Y96BZ2nTKHvDQuNMFa/Tej4YW1qyJzNwtOS/vQ68SnvGlN+a+0kbqqCbW4fyivFVAJ3dZgLg0KUitHuDDS9QpiyQ0X5HYbuCFED7/8mMQ/Rgej6Gx/eLTfT2OLfWU77DuL2E92xE7YmMVMeLveqXfuxf6eP/Yv/Mn9Vd/rc76w/+DzOybDwwE=</latexit><latexit sha1_base64="BDwoBlvzuP9vRxmJB62/VzKKa7Y=">AAACp3ichVHfS9xAEN6kttrYH9f2sS+Dh6X4cCRFaH0QBF8EKZxg9OCSHpu9yd3iZhN2J8IR8rf1f/DN/8bNmYdWRT8Y+Pb7dmZnZ7JKSUtheOv5rzZev9ncehtsv3v/4ePg0+cLW9ZGYCxKVZpJxi0qqTEmSQonlUFeZAovs6vjzr+8RmNlqc9pVWFa8IWWuRScnDQb/P0Nh4CzyIX+swfBISQKc5pCk2S4kLrhxvBV24gObRDCN+gimZdks/4QQZI87YSdk1x3Ume9SJ6vg3retwMtJEYulpTOBsNwFK4Bj0nUkyHrMZ4NblxlUReoSShu7TQKK0pdXZJCYRsktcWKiyu+wKmjmhdo02Y96BZ2nTKHvDQuNMFa/Tej4YW1qyJzNwtOS/vQ68SnvGlN+a+0kbqqCbW4fyivFVAJ3dZgLg0KUitHuDDS9QpiyQ0X5HYbuCFED7/8mMQ/Rgej6Gx/eLTfT2OLfWU77DuL2E92xE7YmMVMeLveqXfuxf6eP/Yv/Mn9Vd/rc76w/+DzOybDwwE=</latexit>
  29. 29. THE INCOHERENCE ASSUMPTION • More generally, it is hard to recover if the singular vectors of the matrix M are similar to standard basis. • Because, information is highly concentrated on specific region. • Hence, the singular vectors need to be sufficiently spread. • This is why the paper introduce the incoherence assumption. • The EMCCO paper said that the random low-rank (orthogonal) matrices satisfy the incoherent assumption. !29
  30. 30. MAIN RESULTS • Required entries: |Ω| = O((k4.5 log k) n log n) • Required steps: O(log (1/ε)) • The EMCCO paper said that the optimum number of required entries is O(n log n) !30
  31. 31. COMPARISON TO NUCLEAR NORM MINIMIZATION • Alternating Minimization • Required entries: |Ω| = O((k4.5 log k) n log n) • Required steps: O(log (1/ε)) • Nuclear Norm Minimization (Convex Relaxation) • Required entries: |Ω| = O((k) n log n) • Required steps: O(ε-1/2) • Nuclear Norm Minimization requires smaller the number of samples. • Alternating Minimization converges faster than Nuclear Norm Minimization. • Theoretical bound is not tight in Nuclear Norm Minimization. !31
  32. 32. PROOF SKETCH !32
  33. 33. PRELIMINARY !33
  34. 34. BASE CASE !34
  35. 35. INDUCTION STEP (DISTANCE) !35
  36. 36. INDUCTION STEP (INCOHERENCE) !36
  37. 37. TABLE OF CONTENTS 1. Netflix Prize and Winner’s Algorithm 2. Funk SVD (2006) and Biased Matrix Factorization (IEEE 2009) 3. Probabilistic Matrix Factorization (NIPS 2008) 4. Why We Use Alternating Minimization? 5. Overcoming Local Minima Problem of Alternating Minimization 6. Why Random Initialization Works?
  38. 38. WHY RANDOM INITIALIZATION WORKS? • In prior paper, initialization steps matters to guarantee global optimality. • In practice, it is hard to determine hyper-parameters for clipping. • However, random initialization works well practically. • In this section, I’ll show you to why random initialization works. • It can be view as removing hyper-parameters. • Similar to motivation of WGAN-GP. • WGAN is really hard to determine hyper-parameter that performance is highly sensitive to. !38
  39. 39. PRELIMINARY: MATRIX SENSING PROBLEM • Quite similar to the matrix completion problem. • A Low-rank version of linear regression (y = xβ problem). • RIP condition: • Actually, the Jain et al’s work (2013) proved in low-rank matrix sensing problem first. • And then, they extended their work to low-rank matrix completion problem. !39
  40. 40. NO SPURIOUS LOCAL MINIMA • Rong et al (NIPS 2016), Matrix Completion has No Spurious Local Minimum • Matrix Completion problem with Semi-definite matrix assumption. • Showed why random initialization works with proper regularizer. • Srinadh et al (NIPS 2016), Global Optimality of Local Search for Low Rank Matrix Recovery • Matrix Sensing Problem. • Showed all local minima are very close to a global optimum with noisy measurements. • With a curvature bound (RIP condition), 
 a polynomial time global convergence is guaranteed by SGD from random initialization. • Rong et al (ArXiv 2017), No Spurious Local Minima in Non-convex Low Rank Problems: 
 A Unified Geometric Analysis • Extended Srinadh et al’s works from matrix sensing to matrix completion and robust PCA. • Showed no high-order saddle point exists if being with proper regularizer. !40
  41. 41. ANY QUESTION?
  42. 42. REFERENCES [1] Koren,Yehuda, Robert Bell, and ChrisVolinsky. "Matrix factorization techniques for recommender systems." Computer 8 (2009): 30-37. [2] Mnih,Andriy, and Ruslan R. Salakhutdinov. "Probabilistic matrix factorization." Advances in neural information processing systems. 2008. [3] Jain, Prateek, Praneeth Netrapalli, and Sujay Sanghavi. "Low-rank matrix completion using alternating minimization." Proceedings of the forty-fifth annual ACM symposium on Theory of computing.ACM, 2013. [4] Ge, Rong, Jason D. Lee, and Tengyu Ma. "Matrix completion has no spurious local minimum." Advances in Neural Information Processing Systems. 2016. [5] Bhojanapalli, Srinadh, Behnam Neyshabur, and Nati Srebro. "Global optimality of local search for low rank matrix recovery." Advances in Neural Information Processing Systems. 2016. [6] Ge, Rong, Chi Jin, andYi Zheng. "No spurious local minima in nonconvex low rank problems:A unified geometric analysis." arXiv preprint arXiv:1704.00708 (2017) [7] how does Netflix recommend movies? [8] Li, Dongsheng, et al. "Mixture-Rank Matrix Approximation for Collaborative Filtering." Advances in Neural Information Processing Systems. 2017. !42
  43. 43. REFERENCES [9] Candès, Emmanuel J., and Benjamin Recht. "Exact matrix completion via convex optimization." Foundations of Computational mathematics 9.6 (2009): 717. [10] Glendinning, R. H. "An evaluation of the ICM algorithm for image reconstruction." Journal of Statistical Computation and Simulation 31.3 (1989): 169-185. [11] http://sifter.org/~simon/journal/20061211.html [12] https://www.slideshare.net/ssuser62b35f/exact-matrix-completion-via-convex- optimization-slideppt [13] Lee, Joonseok, et al. "Local low-rank matrix approximation." International Conference on Machine Learning. 2013. [14] Sedhain, Suvash, et al. "Autorec:Autoencoders meet collaborative filtering." Proceedings of the 24th International Conference onWorldWideWeb.ACM, 2015. [15] Zheng,Yin, et al. "A neural autoregressive approach to collaborative filtering." arXiv preprint arXiv:1605.09477 (2016). [16] Fu, Mingsheng, et al. "Attention based collaborative filtering." Neurocomputing (2018). [17] Li, Dongsheng, et al. "Low-rank matrix approximation with stability." International Conference on Machine Learning. 2016. !43
  • weiyan29

    Apr. 8, 2019

This slide can answer why Biased Matrix Factorization works well?

Aufrufe

Aufrufe insgesamt

1.332

Auf Slideshare

0

Aus Einbettungen

0

Anzahl der Einbettungen

18

Befehle

Downloads

14

Geteilt

0

Kommentare

0

Likes

1

×