Slides for this video https://www.youtube.com/watch?v=jbluHIgBmBo

1. 1. Introduction SC

2. A Friendly Introduction to Logistic Regression
and the Perceptron Algorithm
Luis Serrano

3. Linear Regression Logistic Regression Support Vector Machines
Series of 3 videos
(Perceptron algorithm)

4. 3. Example and algorithm SC

5. What is logistic regression?
What is the perceptron algorithm?

6. E-mail spam classifier
Spam Non-spam (ham)
Hello grandson,
I made cookies.
Love, Grandma
Buy, l0ts of money,
now, che@p buy
buy free mon3y
‘buy’
spelling mistakes

7. 0 1 2 3
Appearances of the word “buy”
4 5 6
Rule 1: If #appearances of the word ‘buy’ > 2, then spam
spamham

8. 0 1 2 3
Spelling mistakes
4 5 6
Rule 2: If #spelling mistakes > 3, then spam
spamham

9. Buy: 4, Mistakes: 3
Buy: 1, Mistakes: 2
Buy: 0, Mistakes: 2
Buy: 2, Mistakes: 1
Buy: 4, Mistakes: 1
Buy: 0, Mistakes: 3
Buy: 2, Mistakes: 3
Buy: 0, Mistakes: 1
Buy: 2, Mistakes: 4
Buy: 3, Mistakes: 2
E-mails Buy + Mistakes
7
3
2
3
5
3
5
1
6
5
Rule 3: If #buy + #mistakes > 4, then spam

10. 1
2
3
4
1 2 3 4
Appearances of ‘buy’
Spellingmistakes
5
Rule 3: If #buy + #mistakes > 4, then spam
Buy: 2
Mistakes: 4
Buy: 2
Mistakes: 3
Buy: 4
Mistakes: 3
Buy: 3
Mistakes: 2
Buy: 4
Mistakes: 1
Buy: 3
Mistakes: 0
Buy: 2
Mistakes: 0
Buy: 2
Mistakes: 1
Buy: 1
Mistakes: 0
Buy: 1
Mistakes: 2

11. Classification goal: split data
Ham
Spam

12. Perceptron Algorithm

13. I’m good
Perceptron Algorithm

14. Get over
here!
Perceptron Algorithm

15. I’m good
Perceptron Algorithm

16. Perceptron Algorithm
Get over
here!

17. I’m good
Perceptron Algorithm

18. I’m good
Perceptron Algorithm

19. Get over
here!
Perceptron Algorithm

20. Get over
here!
Perceptron Algorithm

21. Get over
here!
Perceptron Algorithm

22. I’m good
Perceptron Algorithm

23. Perceptron Algorithm
Get over
here!

24. I’m good
I’m good
I’m good
I’m good
Perceptron Algorithm

25. Perceptron
algorithm
Step 3: (repeat 1000 times)
- Pick random point
- If point is correctly classified:
- Do nothing
- If point is incorrectly classified
- Move line towards point
Step 2: Pick a large number.
(number of repetitions, or epochs)
1000
Step 1: Start with a random line
with blue and red sides.
Step 4: Enjoy your line that
separates the data!
Get over
here!
I’m good

26. 4. How to move a line SC

27. How to move a line

28. How to move a line
Pivot

29. How to move a line
Pivot

30. Rotating and translating
y = 2x + 32x + 3y = 62x + 3y + (-6) = 0

31. Rotating and translating
2x + 3y + (-6) = 0

32. Rotating and translating
2x + 3y + (-6) = 0

33. Rotating and translating
6/2
6/3
x=0
y=6/3
y=0
x=6/2
2x + 3y + (-6) = 0

34. 2x + 3y + (-6) = 0
Rotating and translating
Pivot
6/2
6/3

35. 2x + 3y + (-6) = 0
Get over
here!
Learning rate: 0.1
-0.1-0.1-0.1
1.9x + 2.9y + (-6.1) = 0 Pivot
Pivot
Perceptron Trick

36. Algorithm
Step 3: Pick a small number.
(learning rate)
0.01
Step 4: (repeat 1000 times)
- Pick random point
- If point is correctly classified:
- Do nothing
- If point is incorrectly classified
- Move line towards point
Step 2: Pick a large number.
(number of repetitions, or epochs)
1000
Step 1: Start with a random line
of equation ax + by + c = 0
Step 5: Enjoy your fitted line!
- If point is correctly classified
- Do nothing
- If point is incorrectly classified
- Add + 0.01 to a
- Add + 0.01 to b
- Add + 0.01 to c
Figure out!

37. 5. Second algorithm SC

38. Algorithm
Step 3: Pick a small number.
(learning rate)
0.01
Step 4: (repeat 1000 times)
- Pick random point
- If point is correctly classified:
- Do nothing
- If point is incorrectly classified
- Move line towards point
Step 2: Pick a large number.
(number of repetitions, or epochs)
1000
Step 1: Start with a random line
of equation ax + by + c = 0
Step 5: Enjoy your fitted line!
- If point is correctly classified
- Do nothing
- If point is incorrectly classified
- Add + 0.01 to a
- Add + 0.01 to b
- Add + 0.01 to c
Figure out!
Perceptron
trick

39. Perceptron Trick
2x + 3y -6 = 0
Get over
here!
-6.1

40. 2x + 3y -6.1= 02x + 3y -6 = 0-6.1
Perceptron Trick

41. 2x + 3y -6.1= 01.8
Perceptron Trick

42. 2x + 3y -6.1= 01.8
Perceptron Trick

43. 2x + 3y -6.1= 01.8
Perceptron Trick

44. 2x + 3y -6.1= 01.81.4
Perceptron Trick

45. 2x + 3y -6.1= 01.4
Perceptron Trick

46. 2x + 3y -6.1= 01.42.2
Perceptron Trick

47. OK, let’s do it again

48. 2x + 3y -6.1= 02x + 3y -6 = 0-6.1
Perceptron Trick
1.8

49. 2x + 3y -6.1= 01.8
Perceptron Trick

50. 2x + 3y -6.1= 01.8
Perceptron Trick

51. 2x + 3y -6.1= 01.8
Perceptron Trick
1.4

52. 2x + 3y -6.1= 01.8
Perceptron Trick
1.4

53. 2x + 3y -6.1= 01.81.4
Perceptron Trick

54. 2x + 3y -6.1= 01.42.2
Perceptron Trick

55. 2x + 3y -6.1= 01.42.2
Perceptron Trick

56. Perceptron Trick
2x + 3y + -6 = 0
(4,5)
Learning rate: 0.010.01
4
5-0.04
0.010.01
1.96x + 2.95y + -6.01 = 0
-0.05 -0.01
Pivot
Pivot

57. Perceptron Trick
ax + by + c = 0
(p,q)
Learning rate: 0.01
p
q-0.01p
(a-0.01p)x + (b-0.01q)y + (c-0.01) = 0
-0.01q -0.01

58. Perceptron Trick
ax + by + c = 0
(p,q)
Learning rate: 0.01
p
q+0.01p +0.01q +0.01
(a+0.01p)x + (b+0.01q)y + (c+0.01) = 0

59. Perceptron Trick
-2x + -3y + 6 = 0
(4,5)
Learning rate: 0.01
4
5+0.04 +0.05 +0.01
(-1.96)x + (-2.95)y + (6.01) = 0

60. Correctly and incorrectly classified points
Correctly
classified
Correctly
classified
Incorrectly
classified
Incorrectly
classified

61. Perceptron
algorithm Step 3: Pick a small number.
(learning rate)
0.01
Step 4: (repeat 1000 times)
- Pick random point (p,q)
- If point is correctly classified:
- Do nothing
- If point is incorrectly classified
- Move line towards point
Step 2: Pick a large number.
(number of repetitions, or epochs)
1000
Step 1: Start with a random line
of equation ax + by + c = 0
Step 5: Enjoy your line!
- If point is correctly classified
- Do nothing
- If point is blue, in the red area
- Subtract 0.01p to a
- Subtract 0.01q to b
- Subtract 0.01 to c
- If point is red, in the blue area
- Add 0.01p to a
- Add 0.01q to b
- Add 0.01 to c

62. Correctly and incorrectly classified points
Correctly
classified
Correctly
classified
Incorrectly
classified
Incorrectly
classified

63. Positive and negative regions
2x + 3y - 6 = 0
(3,5)
(1,1)
2(3) + 3(5) - 6 = 15
2(1) + 3(1) - 6 = -1

64. Positive and negative regions
2x + 3y - 6 = 0
2x + 3y - 6 > 0
Positive region
2x + 3y - 6 < 0
Negative region

65. Positive and negative regions
-2x - 3y + 6 = 0
-2x - 3y + 6 < 0
-2x - 3y + 6 > 0
Negative region
Positive region

66. 1
2
3
4
1 2 3 4
Appearances of ‘buy’
Spellingmistakes
5
#buy + #mistakes = 4
#buy + #mistakes > 4
#buy + #mistakes < 4

67. Positive and negative regions
2x + 3y - 6 = 0
2x + 3y -6 > 0
2x + 3y - 6 < 0
Positive (red) region
Negative (blue) region

68. Correctly and incorrectly classified points
Correctly
classified
Correctly
classified
Incorrectly
classified
Incorrectly
classified

69. Perceptron
algorithm Step 3: Pick a small number.
(learning rate)
0.01
Step 4: (repeat 1000 times)
- Pick random point (p,q)
- If point is correctly classified:
- Do nothing
- If point is incorrectly classified
- Move line towards point
Step 2: Pick a large number.
(number of repetitions, or epochs)
1000
Step 1: Start with a random line
of equation ax + by + c = 0
Step 5: Enjoy your line!
- If point is correctly classified
- Do nothing
- If point is blue, on red area
- Add 0.01 p to a
- Add 0.01 q to b
- Add 0.01 to c
- If point is, red on blue area
- Subtract 0.01 p to a
- Subtract 0.01 q to b
- Subtract 0.01 to c
- If point is correctly classified
- Do nothing
- If point is blue, and ap+bq+c > 0
- Subtract 0.01p to a
- Subtract 0.01q to b
- Subtract 0.01 to c
- If point is, red and ap+bq+c < 0
- Add 0.01p to a
- Add 0.01q to b
- Add 0.01 to c

70. 7. Gradient descent SC

71. Challenge 1: Gradient descent

72. Perceptron Error
00
00
0
0

73. Perceptron Error
Error = 17
2x +
3y - 6
=
0
(4,5) 2(4) + 3(5) - 6 = 17
(1,1)
2(1) + 3(1) - 6 = -1
Error = 1

74. Gradient Descent
Same as the perceptron algorithm!
Minimize using calculus (gradient descent)
Bad line Good line
Large error Small error

75. Gradient Descent
Low error
High error
Perceptron trick!

76. 8. Log-loss error function SC

77. Challenge 2: Log-loss function

78. Continuous predictions

79. A friendly introduction to deep learning and neural networks

80. Continuous predictions
11
1
11
0
0
0
0
0
1 (spam)
0 (ham)

81. Continuous predictions
0.5
0.6
0.7
0.4
0.3
70% likely spam40% likely spam
0.8
0.2

82. Continuous predictions
2x+3y-6
=
0
2x+3y-6
=
1
2x+3y-6
=
2
2x+3y-6
=
3
2x+3y-6
=
-12x+3y-6
=
-2
2x+3y-6
=
-3

83. Activation function (sigmoid)
1
0
0.5
0.73
0.12
1-2

84. Continuous predictions
0
1
2
3
-1
-2
-3

85. Log-Loss Error
0.5
0.6
0.7
0.4
0.3
1 (spam)
0.7 likely spam
0.8
0.2

86. Log-Loss Error
0.5
0.6
0.7
0.4
0.3
1 (spam)
0.7 likely spam
0.8
0.2

87. Log-Loss Error
1 (spam)
0.8 likely spam
1 (spam)
0.1 likely spam
Error derivative: 1-0.8 = 0.2
Error derivative: 1-0.1 = 0.9

88. Log-Loss Error
0 (ham)
0.8 likely spam
Error derivative: 0-0.8 = -0.8
0 (ham)
0.1 likely spam
Error derivative: 0 - 0.1 = -0.1

89. I’m good
Perceptron Algorithm
Move
away!

90. Log-loss error
Large log-loss error Small log-loss error
Minimize using calculus (gradient descent)
Bad line Good line
Logistic regression algorithm

91. Logistic regression
algorithm
Step 3: Pick a small number.
(learning rate)
0.01
Step 4: (repeat 1000 times)
- Pick random point (p,q)
- If point is correctly classified:
- Move line away from point
- If point is incorrectly classified
- Move line towards point
Step 2: Pick a large number.
(number of repetitions, or epochs)
1000
Step 1: Start with a random line
of equation ax + by + c = 0
Step 5: Enjoy your fitted line!
- Add 0.01(y - y)p to a
- Add 0.01(y - y)q to b
- Add 0.01(y - y) to c
^
^
^

92. 9. Conclusion SC

93. Linear Regression Logistic Regression Support Vector Machines
Series of 3 videos

94. Thank you!
@luis_likes_math
Subscribe, like,
share, comment!
youtube.com/c/LuisSerrano