Binary Classification Naive Bayes Classifier Support Vector Machine

1. Programming for Data
Analysis
Week 11
Dr. Ferdin Joe John Joseph
Faculty of Information Technology
Thai – Nichi Institute of Technology, Bangkok

2. Today’s lesson
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
2
• Binary Classification
• Naïve Bayes Classifier
• Support Vector Machine

3. Naïve Bayes Classifier
• Conditional Probability Model of Classification
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
3

4. Conditional Probability Model of Classification
• The conditional probability can be calculated using the joint
probability, although it would be intractable.
• Bayes Theorem provides a principled way for calculating the
conditional probability.
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
4

5. Bayes Theorem
• P(A|B) = P(B|A) * P(A) / P(B)
• We can frame classification as a conditional classification problem
with Bayes Theorem as follows:
P(yi | x1, x2, …, xn) = P(x1, x2, …, xn | yi) * P(yi) / P(x1, x2, …, xn)
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
5

6. Naïve Bayes
• For simplifying the calculation
• The Bayes Theorem assumes that each input variable is dependent
upon all other variables.
• This is a cause of complexity in the calculation.
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
6

7. Calculation of Prior and Conditional
Probabilities
• P(yi) = examples with yi / total examples
• In the case of categorical variables, such as counts or labels, a
multinomial distribution can be used.
• If the variables are binary, such as yes/no or true/false, a binomial
distribution can be used.
• If a variable is numerical, such as a measurement, often a Gaussian
distribution is used.
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
7

8. Naïve Bayes Distribution
• Binomial Naïve Bayes: Naïve Bayes that uses a binomial distribution.
• Multinomial Naïve Bayes: Naïve Bayes that uses a multinomial
distribution.
• Gaussian Naïve Bayes: Naïve Bayes that uses a Gaussian distribution.
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
8

9. Preparation of dataset
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
9

10. Fitting Probability Distribution
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
10

11. Sorting Data
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
11

12. Calculate Priors
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
12

13. Fit Distribution
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
13

14. Summary
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
14

15. Independent Conditional Probability
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
15

16. Taking a sample
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
16

17. Probability Scores
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
17

18. Wrap Up
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
18

19. Gaussian Naïve Bayes
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
19

20. Lab Exercise
• Use this source code and make a classification report which gives
accuracy, precision, recall and F1-score
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
20

21. Support Vector Machine
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
21

22. SVM Objective
• A training set, S, for an SVM is comprised of m samples.
• The features, x, consist of real numbers and the classifications, y,
must be -1 or 1.
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
22

23. SVM Hyperplane
• The SVM hyperlane is defined by the weight vector, w, and the bias, b,
and is defined as:
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
23

24. Example for 2 Feature
• Hyperplane for two features can be written as:
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
24

25. Libraries
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
25

26. Data Clusters
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
26

27. Data Clusters
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
27

28. Prepare Datasets
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
28

29. Learning Rate
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
29

30. Learning Rate
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
30

31. Draw Decision Boundary
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
31

32. Plot Data
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
32

33. Plot Data
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
33

34. Test Classifier
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
34

35. Test Classifier
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
35

36. Output
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
36

37. Lab Exercise
• Create Confusion Matrix and classification report for SVM
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
37

38. DSA 207 – Binary Classifier
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
38