1.
Singular Value Decomposition
Luis Serrano

2.
Announcements

3.
github.com/luisguiserrano/singular_value_decomposition

4.
https://www.manning.com/books/grokking-machine-learning
Discount code: serranoyt
Grokking Machine
Learning
By Luis G. Serrano

5.
Transformations

6.
Transformations
Stretch (or compress) horizontally

7.
Transformations
Stretch (or compress) horizontally

8.
Transformations
Stretch (or compress) horizontally

9.
Transformations
Stretch (or compress) horizontally

10.
Transformations
Stretch (or compress) vertically

11.
Transformations
Stretch (or compress) vertically

12.
Transformations
Stretch (or compress) vertically

13.
Transformations
Stretch (or compress) vertically

14.
Transformations
Rotate

15.
Puzzle (easy)

16.
Puzzle (easy)

17.
Puzzle (easy)

18.
Puzzle (easy)

19.
Puzzle (easy)

20.
Puzzle (hard)

21.
Puzzle (hard)

22.
Puzzle (hard)

23.
Puzzle (hard)

24.
Puzzle (hard)

25.
Solution

26.
Solution

27.
Solution

28.
Solution

29.
Solution

30.
Solution

31.
Solution

32.
Linear transformations

33.
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
What does this have to do with matrices?
(p,q) (3p+0q, 4p+5q)
(1,0) (3, 4)
(0,1) (0, 5)
(-1,0)
(0,-1)
(-3, -4)
(0, -5)
3 0
4 5[ ]A =

34.
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Rotation matrices
cos(θ) −sin(θ)
sin(θ) cos(θ)[ ]
θ

35.
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Stretching matrices
σ1 0
0 σ2
[ ]
σ1
σ2

36.
Stretching matrices
σ1 0
0 σ2
[ ]
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
σ1
σ2

37.
Stretching matrices
σ1 0
0 σ2
[ ]
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
σ1
σ2

38.
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
What does this have to do with matrices?
3 0
4 5[ ]A =
cos(θ) sin(θ)
−sin(θ) cos(θ)[ ]
cos(ϕ) sin(ϕ)
−sin(ϕ) cos(ϕ)[ ]
σ1 0
0 σ2
[ ]

39.
Singular value decomposition
3 0
4 5[ ]
cos(θ) sin(θ)
−sin(θ) cos(θ)[ ]
cos(ϕ) sin(ϕ)
−sin(ϕ) cos(ϕ)[ ]
σ1 0
0 σ2
[ ]=
A = UΣV†

40.
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
SVD A = UΣV†
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
Rotation of θ = −
π
4
= − 45o
3 0
4 5[ ] =

41.
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Rotation
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
A = UΣV†
Rotation of θ = −
π
4
= − 45o
3 0
4 5[ ] =

42.
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Rotation
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
A = UΣV†
Horizontal scaling by 3 5
3 0
4 5[ ] =

43.
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Scaling
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
A = UΣV†
Horizontal scaling by 3 5
Vertical scaling by 5
3 0
4 5[ ] =

44.
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Rotation
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
A = UΣV†
Vertical scaling by 5
3 0
4 5[ ] =

45.
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Rotation
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
A = UΣV†
Rotation of θ = arctan(3) = 71.72o
3 0
4 5[ ] =

46.
-7
-5
-3
-1
1
3
5
7
-7-6-5-4-3-2-1 0 1 2 3 4 5 6 7
Rotation
1/ 2 1/ 2
−1/ 2 1/ 2[ ]
3 5 0
0 5[ ]
1/ 10 −3/ 10
3/ 10 1/ 10[ ]
0.7071 0.7071
−0.7071 0.7071[ ]
6.708 0
0 2.236[ ]
0.316 −0.949
0.949 0.316[ ]
A = UΣV†
Rotation of θ = arctan(3) = 71.72o
3 0
4 5[ ] =

47.
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
-7
-5
-3
-1
1
3
5
7
-7
-5
-3
-1
1
3
5
7
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
A = UΣV†
A
Σ
V† U

48.
Dimensionality reduction

49.
1 2 3 4
-1 -2 -3 -4
2 4 6 8
10 20 30 40
1 2 3 4
1
-1
2
10
Difference between these two matrices?

50.
1 2 3 4
-1 -2 -3 -4
2 4 6 8
10 20 30 40
1 2 3 4
1
-1
2
10
Difference between these two matrices?
3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3
? ? ? ?
?
?
?
?

51.
=
1 2 3 4
-1 -2 -3 -4
2 4 6 8
10 20 30 40
1 2 3 4
1
-1
2
10
Rank 1 matrices
16 numbers 8 numbers

52.
=
3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3
? ? ? ?
?
?
?
?
Higher rank matrices
16 numbers

53.
Rank of a matrix
=
Rank 1
=
Rank 2
=
Rank 3
=
Rank 4

54.
∼
3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3
? ? ? ?
?
?
?
?
Approximation by a rank one matrix

55.
0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4σ4
= U1 U2 U3 U4
V1
V2
V3
V4
σ1
σ2
σ3
σ4
0
1
2
3
4
0 1 2 3 4
A

56.
0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1
U2 U3 U4
V1
V2
V3
V4
σ1
σ2
σ3
σ4
+
=
0
1
2
3
4
0 1 2 3 4
A

57.
0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2
U3 U4
V1 V2
V3
V4
σ1 σ2
σ3
σ4
+
=
0
1
2
3
4
0 1 2 3 4
A

58.
0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2 U3
U4
V1 V2 V3
V4
σ1 σ2 σ3
σ4
+
=
+
0
1
2
3
4
0 1 2 3 4
A

59.
0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2 U3 U4V1 V2 V3 V4σ1 σ2 σ3 σ4+
=
+ +
Rank 1 Rank 1 Rank 1 Rank 1
TinySmallMediumLarge
0
1
2
3
4
0 1 2 3 4
A

60.
0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2V1 V2σ1 σ2+
=
Rank 1 Rank 1
MediumLarge
0
1
2
3
4
0 1 2 3 4
A

61.
3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3
0.15+
Tiny
0.79 -0.29 -0.54 -0.04
-0.89
-0.23
0.15
0.36
0
1
2
3
4
0 1 2 3 4
21.2
6.4
4.9
0.15
=
-0.55 -0.52 -0.49 -0.43
0.26 -0.4 0.65 -0.59
0.07 0.7 -0.22 -0.68
0.79 -0.29 -0.54 -0.04
-0.21 0.37 -0.13 -0.89
-0.52 -0.7 0.43 -0.23
-0.48 -0.21 -0.84 0.15
-0.67 0.57 0.31 0.36
4.9+ 0.07 0.7 -0.22 -0.68
-0.13
0.43
-0.84
0.31
Small
6.4+ 0.26 -0.4 0.65 -0.59
0.37
-0.7
-0.21
0.57
Medium
21.2 -0.55 -0.52 -0.49 -0.43
-0.21
-0.52
-0.48
-0.67
Large
2.51 2.37 2.22 1.97
6.07 5,72 5.37 4.77
5.63 5,31 4.99 4.43
7.88 7.43 6.98 6.19
3.15 1.41 3.79 0.56
4.87 7.53 2.44 7.43
5.28 5.85 4.12 5.22
8.85 5.96 9.36 4.03
3.1 0.96 3.93 0.99
5.03 8.99 1.98 6
4.98 3.01 5.01 8
8.96 7.02 9.03 3
3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3

62.
Rank of a matrix
=
Rank 1
=
Rank 2
=
Rank 3
=
Rank 4

63.
Dimensionality reduction

64.
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
1.8 1.2
4.4 4.6[ ]
A
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
-7
-5
-3
-1
1
3
5
7
-7
-5
-3
-1
1
3
5
7
A = UΣV†
0.316 −0.949
0.949 0.316[ ]U
6.71 0
0 0.44[ ]Σ
0.7071 0.7071
−0.7071 0.7071[ ] V†
6.71
0.44

65.
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
-7
-5
-3
-1
1
3
5
7
-7
-5
-3
-1
1
3
5
7
6.71 0
0 0.44[ ]
1.8 1.2
4.4 4.6[ ]
A = UΣV†
A
Σ
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
6.71
0.44
0
0
0.7071 0.7071
−0.7071 0.7071[ ] V†
0.316 −0.949
0.949 0.316[ ]U

66.
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
-7
-5
-3
-1
1
3
5
7
-7
-5
-3
-1
1
3
5
7
1.8 1.2
4.4 4.6[ ]
A = UΣV†
A
Σ
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
6.71
[
6.71 0
0 0.44 ]
0.44
0
0
0.316 −0.949
0.949 0.316[ ]U
0.7071 0.7071
−0.7071 0.7071[ ] V†

67.
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
-7
-5
-3
-1
1
3
5
7
-7
-5
-3
-1
1
3
5
7
1.8 1.2
4.4 4.6[ ]
A = UΣV†
A
Σ
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
6.71
[
6.71 0
0 0.44 ]
0.44
0
0
1.5 1.5
4.5 4.5[ ]
0.316 −0.949
0.949 0.316[ ]U
0.7071 0.7071
−0.7071 0.7071[ ] V†

68.
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
1.5 1.5
4.5 4.5[ ]
Rank 1

69.
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
1.8 1.2
4.4 4.6[ ]
Rank 2
-7
-5
-3
-1
1
3
5
7
-7 -5 -3 -1 1 3 5 7
1.5 1.5
4.5 4.5[ ]
Rank 1

70.
1 2 3 4
-1 -2 -3 -4
2 4 6 8
10 20 30 40
1 2 3 4
1
-1
2
10
Rank 1 matrices

71.
=
1 2 3 4
-1 -2 -3 -4
2 4 6 8
10 20 30 40
1 2 3 4
1
-1
2
10
Rank 1 matrices
16 numbers 8 numbers

72.
=
3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3
? ? ? ?
?
?
?
?
Higher rank matrices
16 numbers

73.
Approximation by rank one matrices
= + + …
3 1 4 1
5 9 2 6
5 3 5 8
9 7 9 3

74.
U
0
1
2
3
4
0 1 2 3 4
V†
Σ= U1 U2 U3 U4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
0
1
2
3
4
0 1 2 3 4
A

75.
0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4σ4
= U1 U2 U3 U4
V1
V2
V3
V4
σ1
σ2
σ3
σ4
0
1
2
3
4
0 1 2 3 4
A

76.
0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1
U2 U3 U4
V1
V2
V3
V4
σ1
σ2
σ3
σ4
+
=
0
1
2
3
4
0 1 2 3 4
A

77.
0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2
U3 U4
V1 V2
V3
V4
σ1 σ2
σ3
σ4
+
=
0
1
2
3
4
0 1 2 3 4
A

78.
0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2 U3
U4
V1 V2 V3
V4
σ1 σ2 σ3
σ4
+
=
+
0
1
2
3
4
0 1 2 3 4
A

79.
0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2 U3 U4V1 V2 V3 V4σ1 σ2 σ3 σ4+
=
+ +
Rank 1 Rank 1 Rank 1 Rank 1
TinySmallMediumLarge
0
1
2
3
4
0 1 2 3 4
A

80.
0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4
0
1
2
3
4
0 1 2 3 4
V1
V2
V3
V4
0
1
2
3
4
0 1 2 3 4
σ1
σ2
σ3
σ4
U1 U2V1 V2σ1 σ2+
=
Rank 1 Rank 1
MediumLarge
0
1
2
3
4
0 1 2 3 4
A

81.
Rectangular matrices

82.
0
1
2
3
4
0 1 2 3 4
U1 U2 U3 U4=
0
1
2
3
4
0 1 2 3 4 5 6
A
V1
V2
V3
V4
V5
V6
σ1
σ2
σ3
σ4
0 0
0 0
0 0
0 0
No square matrix? No problem!

83.
Image compression

84.
0 1 1 0 1 1 0
1 1 1 1 1 1 1
1 1 1 1 1 1 1
0 1 1 1 1 1 0
0 0 1 1 1 0 0
0 0 0 1 0 0 0
Rank 4
www.github.com/luisguiserrano/singular_value_decomposition

85.
Thank you!

86.
Similar videos on dimensionality reduction
Matrix Factorization Principal Component Analysis

87.
https://www.manning.com/books/grokking-machine-learning
Discount code: serranoyt
Grokking Machine
Learning
By Luis G. Serrano

88.
Thank you!
@luis_likes_math
Subscribe, like,
share, comment!
youtube.com/c/LuisSerrano
http://serrano.academy