Diese Präsentation wurde erfolgreich gemeldet.

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
×

1 von 89 Anzeige

Singular value decomposition (SVD)

Slides for this video

Slides for this video

Anzeige
Anzeige

Weitere Verwandte Inhalte

Anzeige

Anzeige

Singular value decomposition (SVD)

1. 1. Singular Value Decomposition Luis Serrano
2. 2. Announcements
3. 3. github.com/luisguiserrano/singular_value_decomposition
4. 4. https://www.manning.com/books/grokking-machine-learning Discount code: serranoyt Grokking Machine Learning By Luis G. Serrano
5. 5. Transformations
6. 6. Transformations Stretch (or compress) horizontally
7. 7. Transformations Stretch (or compress) horizontally
8. 8. Transformations Stretch (or compress) horizontally
9. 9. Transformations Stretch (or compress) horizontally
10. 10. Transformations Stretch (or compress) vertically
11. 11. Transformations Stretch (or compress) vertically
12. 12. Transformations Stretch (or compress) vertically
13. 13. Transformations Stretch (or compress) vertically
14. 14. Transformations Rotate
15. 15. Puzzle (easy)
16. 16. Puzzle (easy)
17. 17. Puzzle (easy)
18. 18. Puzzle (easy)
19. 19. Puzzle (easy)
20. 20. Puzzle (hard)
21. 21. Puzzle (hard)
22. 22. Puzzle (hard)
23. 23. Puzzle (hard)
24. 24. Puzzle (hard)
25. 25. Solution
26. 26. Solution
27. 27. Solution
28. 28. Solution
29. 29. Solution
30. 30. Solution
31. 31. Solution
32. 32. Linear transformations
33. 33. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 What does this have to do with matrices? (p,q) (3p+0q, 4p+5q) (1,0) (3, 4) (0,1) (0, 5) (-1,0) (0,-1) (-3, -4) (0, -5) 3 0 4 5[ ]A =
34. 34. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation matrices cos(θ) −sin(θ) sin(θ) cos(θ)[ ] θ
35. 35. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Stretching matrices σ1 0 0 σ2 [ ] σ1 σ2
36. 36. Stretching matrices σ1 0 0 σ2 [ ] -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 σ1 σ2
37. 37. Stretching matrices σ1 0 0 σ2 [ ] -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 σ1 σ2
38. 38. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 What does this have to do with matrices? 3 0 4 5[ ]A = cos(θ) sin(θ) −sin(θ) cos(θ)[ ] cos(ϕ) sin(ϕ) −sin(ϕ) cos(ϕ)[ ] σ1 0 0 σ2 [ ]
39. 39. Singular value decomposition 3 0 4 5[ ] cos(θ) sin(θ) −sin(θ) cos(θ)[ ] cos(ϕ) sin(ϕ) −sin(ϕ) cos(ϕ)[ ] σ1 0 0 σ2 [ ]= A = UΣV†
40. 40. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 SVD A = UΣV† 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] 1/ 2 1/ 2 −1/ 2 1/ 2[ ] Rotation of θ = − π 4 = − 45o 3 0 4 5[ ] =
41. 41. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Rotation of θ = − π 4 = − 45o 3 0 4 5[ ] =
42. 42. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Horizontal scaling by 3 5 3 0 4 5[ ] =
43. 43. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Scaling 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Horizontal scaling by 3 5 Vertical scaling by 5 3 0 4 5[ ] =
44. 44. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Vertical scaling by 5 3 0 4 5[ ] =
45. 45. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Rotation of θ = arctan(3) = 71.72o 3 0 4 5[ ] =
46. 46. -7 -5 -3 -1 1 3 5 7 -7-6-5-4-3-2-1 0 1 2 3 4 5 6 7 Rotation 1/ 2 1/ 2 −1/ 2 1/ 2[ ] 3 5 0 0 5[ ] 1/ 10 −3/ 10 3/ 10 1/ 10[ ] 0.7071 0.7071 −0.7071 0.7071[ ] 6.708 0 0 2.236[ ] 0.316 −0.949 0.949 0.316[ ] A = UΣV† Rotation of θ = arctan(3) = 71.72o 3 0 4 5[ ] =
47. 47. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 A = UΣV† A Σ V† U
48. 48. Dimensionality reduction
49. 49. 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Difference between these two matrices?
50. 50. 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Difference between these two matrices? 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ?
51. 51. = 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Rank 1 matrices 16 numbers 8 numbers
52. 52. = 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ? Higher rank matrices 16 numbers
53. 53. Rank of a matrix = Rank 1 = Rank 2 = Rank 3 = Rank 4
54. 54. ∼ 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ? Approximation by a rank one matrix
55. 55. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4σ4 = U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 0 1 2 3 4 0 1 2 3 4 A
56. 56. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
57. 57. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
58. 58. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = + 0 1 2 3 4 0 1 2 3 4 A
59. 59. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4V1 V2 V3 V4σ1 σ2 σ3 σ4+ = + + Rank 1 Rank 1 Rank 1 Rank 1 TinySmallMediumLarge 0 1 2 3 4 0 1 2 3 4 A
60. 60. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2V1 V2σ1 σ2+ = Rank 1 Rank 1 MediumLarge 0 1 2 3 4 0 1 2 3 4 A
61. 61. 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 0.15+ Tiny 0.79 -0.29 -0.54 -0.04 -0.89 -0.23 0.15 0.36 0 1 2 3 4 0 1 2 3 4 21.2 6.4 4.9 0.15 = -0.55 -0.52 -0.49 -0.43 0.26 -0.4 0.65 -0.59 0.07 0.7 -0.22 -0.68 0.79 -0.29 -0.54 -0.04 -0.21 0.37 -0.13 -0.89 -0.52 -0.7 0.43 -0.23 -0.48 -0.21 -0.84 0.15 -0.67 0.57 0.31 0.36 4.9+ 0.07 0.7 -0.22 -0.68 -0.13 0.43 -0.84 0.31 Small 6.4+ 0.26 -0.4 0.65 -0.59 0.37 -0.7 -0.21 0.57 Medium 21.2 -0.55 -0.52 -0.49 -0.43 -0.21 -0.52 -0.48 -0.67 Large 2.51 2.37 2.22 1.97 6.07 5,72 5.37 4.77 5.63 5,31 4.99 4.43 7.88 7.43 6.98 6.19 3.15 1.41 3.79 0.56 4.87 7.53 2.44 7.43 5.28 5.85 4.12 5.22 8.85 5.96 9.36 4.03 3.1 0.96 3.93 0.99 5.03 8.99 1.98 6 4.98 3.01 5.01 8 8.96 7.02 9.03 3 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3
62. 62. Rank of a matrix = Rank 1 = Rank 2 = Rank 3 = Rank 4
63. 63. Dimensionality reduction
64. 64. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] A -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 A = UΣV† 0.316 −0.949 0.949 0.316[ ]U 6.71 0 0 0.44[ ]Σ 0.7071 0.7071 −0.7071 0.7071[ ] V† 6.71 0.44
65. 65. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 0 0 0.44[ ] 1.8 1.2 4.4 4.6[ ] A = UΣV† A Σ -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 0.44 0 0 0.7071 0.7071 −0.7071 0.7071[ ] V† 0.316 −0.949 0.949 0.316[ ]U
66. 66. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] A = UΣV† A Σ -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 [ 6.71 0 0 0.44 ] 0.44 0 0 0.316 −0.949 0.949 0.316[ ]U 0.7071 0.7071 −0.7071 0.7071[ ] V†
67. 67. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] A = UΣV† A Σ -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 6.71 [ 6.71 0 0 0.44 ] 0.44 0 0 1.5 1.5 4.5 4.5[ ] 0.316 −0.949 0.949 0.316[ ]U 0.7071 0.7071 −0.7071 0.7071[ ] V†
68. 68. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.5 1.5 4.5 4.5[ ] Rank 1
69. 69. -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.8 1.2 4.4 4.6[ ] Rank 2 -7 -5 -3 -1 1 3 5 7 -7 -5 -3 -1 1 3 5 7 1.5 1.5 4.5 4.5[ ] Rank 1
70. 70. 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Rank 1 matrices
71. 71. = 1 2 3 4 -1 -2 -3 -4 2 4 6 8 10 20 30 40 1 2 3 4 1 -1 2 10 Rank 1 matrices 16 numbers 8 numbers
72. 72. = 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 ? ? ? ? ? ? ? ? Higher rank matrices 16 numbers
73. 73. Approximation by rank one matrices = + + … 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3
74. 74. U 0 1 2 3 4 0 1 2 3 4 V† Σ= U1 U2 U3 U4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 0 1 2 3 4 0 1 2 3 4 A
75. 75. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4σ4 = U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 0 1 2 3 4 0 1 2 3 4 A
76. 76. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
77. 77. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = 0 1 2 3 4 0 1 2 3 4 A
78. 78. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4 V1 V2 V3 V4 σ1 σ2 σ3 σ4 + = + 0 1 2 3 4 0 1 2 3 4 A
79. 79. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2 U3 U4V1 V2 V3 V4σ1 σ2 σ3 σ4+ = + + Rank 1 Rank 1 Rank 1 Rank 1 TinySmallMediumLarge 0 1 2 3 4 0 1 2 3 4 A
80. 80. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4 0 1 2 3 4 0 1 2 3 4 V1 V2 V3 V4 0 1 2 3 4 0 1 2 3 4 σ1 σ2 σ3 σ4 U1 U2V1 V2σ1 σ2+ = Rank 1 Rank 1 MediumLarge 0 1 2 3 4 0 1 2 3 4 A
81. 81. Rectangular matrices
82. 82. 0 1 2 3 4 0 1 2 3 4 U1 U2 U3 U4= 0 1 2 3 4 0 1 2 3 4 5 6 A V1 V2 V3 V4 V5 V6 σ1 σ2 σ3 σ4 0 0 0 0 0 0 0 0 No square matrix? No problem!
83. 83. Image compression
84. 84. 0 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 0 0 0 1 1 1 0 0 0 0 0 1 0 0 0 Rank 4 www.github.com/luisguiserrano/singular_value_decomposition
85. 85. Thank you!
86. 86. Similar videos on dimensionality reduction Matrix Factorization Principal Component Analysis
87. 87. https://www.manning.com/books/grokking-machine-learning Discount code: serranoyt Grokking Machine Learning By Luis G. Serrano