An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture
ā¢
275 gefƤllt mirā¢102,056 views
The document provides an introduction to supervised machine learning and pattern classification. It begins with an overview of the speaker's background and research interests. Key concepts covered include definitions of machine learning, examples of machine learning applications, and the differences between supervised, unsupervised, and reinforcement learning. The rest of the document outlines the typical workflow for a supervised learning problem, including data collection and preprocessing, model training and evaluation, and model selection. Common classification algorithms like decision trees, naive Bayes, and support vector machines are briefly explained. The presentation concludes with discussions around choosing the right algorithm and avoiding overfitting.
Empfohlen
Weitere Ƥhnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ćhnlich wie An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture
Ćhnlich wie An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture (20)
KĆ¼rzlich hochgeladen
KĆ¼rzlich hochgeladen (20)
An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture
- 1. Practical Data Science An Introduction to Supervised Machine Learning and Pattern Classiļ¬cation: The Big Picture Michigan State University NextGen Bioinformatics Seminars - 2015 Sebastian Raschka Feb. 11, 2015
- 2. A Little Bit About Myself ... Developing software & methods for - Protein ligand docking - Large scale drug/inhibitor discovery PhD candidate in Dr. L. Kuhnās Lab: and some other machine learning side-projects ā¦
- 3. What is Machine Learning? http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram "Field of study that gives computers the ability to learn without being explicitly programmed.ā (Arthur Samuel, 1959) By Phillip Taylor [CC BY 2.0]
- 4. http://commons.wikimedia.org/wiki/ File:American_book_company_1916._letter_envelope-2.JPG#ļ¬lelinks [public domain] https://ļ¬ic.kr/p/5BLW6G [CC BY 2.0] Text Recognition Spam Filtering Biology Examples of Machine Learning
- 5. Examples of Machine Learning http://googleresearch.blogspot.com/2014/11/a-picture-is-worth-thousand-coherent.html By Steve Jurvetson [CC BY 2.0] Self-driving cars Photo search and many, many more ... Recommendation systems http://commons.wikimedia.org/wiki/File:Netļ¬ix_logo.svg [public domain]
- 6. How many of you have used machine learning before?
- 7. Our Agenda ā¢ Concepts and the big picture ā¢ Workļ¬ow ā¢ Practical tips & good habits
- 8. Learning ā¢ Labeled data ā¢ Direct feedback ā¢ Predict outcome/future ā¢ Decision process ā¢ Reward system ā¢ Learn series of actions ā¢ No labels ā¢ No feedback ā¢ āFind hidden structureā Unsupervised Supervised Reinforcement
- 9. Unsupervised learning Supervised learning Clustering: [DBSCAN on a toy dataset] Classiļ¬cation: [SVM on 2 classes of the Wine dataset] Regression: [Soccer Fantasy Score prediction] Todayās topic Supervised LearningUnsupervised Learning
- 10. Instances (samples, observations) Features (attributes, dimensions) Classes (targets) Nomenclature sepal_length sepal_width petal_length petal_width class 1 5.1 3.5 1.4 0.2 setosa 2 4.9 3.0 1.4 0.2 setosa ā¦ ā¦ ā¦ ā¦ ā¦ ā¦ 50 6.4 3.2 4.5 1.5 veriscolor ā¦ ā¦ ā¦ ā¦ ā¦ ā¦ 150 5.9 3.0 5.1 1.8 virginica https://archive.ics.uci.edu/ml/datasets/Iris IRIS
- 11. Classiļ¬cation x1 x2 class1 class2 1) Learn from training data 2) Map unseen (new) data ?
- 12. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classiļ¬cation/ Regression Model New DataPre-Processing Reļ¬nement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
- 13. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classiļ¬cation/ Regression Model New DataPre-Processing Reļ¬nement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
- 14. A Few Common Classiļ¬ers Decision Tree Perceptron Naive Bayes Ensemble Methods: Random Forest, Bagging, AdaBoost Support Vector Machine K-Nearest Neighbor Logistic Regression Artiļ¬cial Neural Network / Deep Learning
- 15. Discriminative Algorithms Generative Algorithms ā¢ Models a more general problem: how the data was generated. ā¢ I.e., the distribution of the class; joint probability distribution p(x,y). ā¢ Naive Bayes, Bayesian Belief Network classiļ¬er, Restricted Boltzmann Machine ā¦ ā¢ Map x ā y directly. ā¢ E.g., distinguish between people speaking different languages without learning the languages. ā¢ Logistic Regression, SVM, Neural Networks ā¦
- 16. Examples of Discriminative Classiļ¬ers: Perceptron xi1 xi2 w1 w2 Ī£ yi y = wTx = w0 + w1x1 + w2x2 1 F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957. x1 x2 y ā {-1,1} w0 wj = weight xi = training sample yi = desired output yi = actual output t = iteration step Ī· = learning rate Īø = threshold (here 0) update rule: wj(t+1) = wj(t) + Ī·(yi - yi)xi 1 if wTxi ā„ Īø -1 otherwise ^ ^ ^ ^ yi ^ until t+1 = max iter or error = 0
- 17. Discriminative Classiļ¬ers: Perceptron F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957. - Binary classiļ¬er (one vs all, OVA) - Convergence problems (set n iterations) - Modiļ¬cation: stochastic gradient descent - āModernā perceptron: Support Vector Machine (maximize margin) - Multilayer perceptron (MLP) xi1 xi2 w1 w2 Ī£ yi 1 y ā {-1,1} w0 ^ x1 x2
- 18. Generative Classiļ¬ers: Naive Bayes Bayes Theorem: P(Ļj | xi) = P(xi | Ļj) P(Ļj) P(xi) Posterior probability = Likelihood x Prior probability Evidence Iris example: P(āSetosa"| xi), xi = [4.5 cm, 7.4 cm]
- 19. Generative Classiļ¬ers: Naive Bayes Decision Rule: Bayes Theorem: P(Ļj | xi) = P(xi | Ļj) P(Ļj) P(xi) pred. class label Ļj argmax P(Ļj | xi) i = 1, ā¦, m e.g., j ā {Setosa, Versicolor, Virginica}
- 20. Class-conditional probability (here Gaussian kernel): Generative Classiļ¬ers: Naive Bayes Prior probability: Evidence: (cancels out) (class frequency) P(Ļj | xi) = P(xi | Ļj) P(Ļj) P(xi) P(Ļj) = NĻj Nc P(xik |Ļj) = 1 ā (2 Ļ ĻĻj 2) exp ( )- (xik - Ī¼Ļj)2 2ĻĻj 2 P(xi |Ļj) P(xik |Ļj)
- 21. Generative Classiļ¬ers: Naive Bayes - Naive conditional independence assumption typically violated - Works well for small datasets - Multinomial model still quite popular for text classiļ¬cation (e.g., spam ļ¬lter)
- 22. Non-Parametric Classiļ¬ers: K-Nearest Neighbor - Simple! - Lazy learner - Very susceptible to curse of dimensionality k=3 e.g., k=1
- 23. Iris Example Setosa Virginica Versicolor k = 3 mahalanobis dist. uniform weights C = 3 depth = 2
- 24. Decision Tree Entropy = depth = 4 petal length <= 2.45? petal length <= 4.75? Setosa Virginica Versicolor Yes No e.g., 2 (- 0.5 log2(0.5)) = 1 āāpi logk pi i Information Gain = entropy(parent) ā [avg entropy(children)] NoYes depth = 2
- 25. "No Free Lunch" :( Roughly speaking: āNo one model works best for all possible situations.ā Our model is a simpliļ¬cation of reality Simpliļ¬cation is based on assumptions (model bias) Assumptions fail in certain situations D. H. Wolpert. The supervised learning no-free-lunch theorems. In Soft Computing and Industry, pages 25ā42. Springer, 2002.
- 26. Which Algorithm? ā¢ What is the size and dimensionality of my training set? ā¢ Is the data linearly separable? ā¢ How much do I care about computational efļ¬ciency? - Model building vs. real-time prediction time - Eager vs. lazy learning / on-line vs. batch learning - prediction performance vs. speed ā¢ Do I care about interpretability or should it "just work well?" ā¢ ...
- 27. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classiļ¬cation/ Regression Model New DataPre-Processing Reļ¬nement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
- 28. Missing Values: - Remove features (columns) - Remove samples (rows) - Imputation (mean, nearest neighbor, ā¦) Sampling: - Random split into training and validation sets - Typically 60/40, 70/30, 80/20 - Donāt use validation set until the very end! (overļ¬tting) Feature Scaling: e.g., standardization: - Faster convergence (gradient descent) - Distances on same scale (k-NN with Euclidean distance) - Mean centering for free - Normal distributed data - Numerical stability by avoiding small weights z = xik - Ī¼k Ļk (use same parameters for the test/new data!)
- 29. Categorical Variables color size prize class label 0 green M 10.1 class1 1 red L 13.5 class2 2 blue XL 15.3 class1 ordinalnominal green ā (1,0,0) red ā (0,1,0) blue ā (0,0,1) class label color=blue color=green color=red prize size 0 0 0 1 0 10.1 1 1 1 0 0 1 13.5 2 2 0 1 0 0 15.3 3 M ā 1 L ā 2 XL ā 3
- 30. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classiļ¬cation/ Regression Model New DataPre-Processing Reļ¬nement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
- 31. Generalization Error and Overļ¬tting How well does the model perform on unseen data?
- 32. Generalization Error and Overļ¬tting
- 33. Error Metrics: Confusion Matrix TP [Linear SVM on sepal/petal lengths] TN FN FP here: āsetosaā = āpositiveā
- 34. Error Metrics TP [Linear SVM on sepal/petal lengths] TN FN FP here: āsetosaā = āpositiveā TP + TN FP +FN +TP +TN Accuracy = = 1 - Error FP N TP P False Positive Rate = TP TP + FP Precision = True Positive Rate = (Recall) āmicroā and āmacroā averaging for multi-class
- 35. Receiver Operating Characteristic (ROC) Curves
- 36. Test set Training dataset Test dataset Complete dataset Test set Test set Test set 1st iteration calc. error calc. error calc. error calc. error calculate avg. error k-fold cross-validation (k=4): 2nd iteration 3rd iteration 4th iteration fold 1 fold 2 fold 3 fold 4 Model Selection
- 37. k-fold CV and ROC
- 38. Feature Selection - Domain knowledge - Variance threshold - Exhaustive search - Decision trees - ā¦ IMPORTANT! (Noise, overļ¬tting, curse of dimensionality, efļ¬ciency) X = [x1, x2, x3, x4]start: stop: (if d = k) X = [x1, x3, x4] X = [x1, x3] Simplest example: Greedy Backward Selection
- 39. Dimensionality Reduction ā¢ Transformation onto a new feature subspace ā¢ e.g., Principal Component Analysis (PCA) ā¢ Find directions of maximum variance ā¢ Retain most of the information
- 40. 0. Standardize data 1. Compute covariance matrix z = xik - Ī¼k Ļik = ā (xij - Āµj) (xik - Āµk) Ļk 1 in -1 Ļ2 1 Ļ12 Ļ13 Ļ14 Ļ21 Ļ2 2 Ļ23 Ļ24 Ļ31 Ļ32 Ļ2 3 Ļ34 Ļ41 Ļ42 Ļ43 Ļ2 4 ā = PCA in 3 Steps
- 41. 2. Eigendecomposition and sorting eigenvalues PCA in 3 Steps X v = Ī» v Eigenvectors [[ 0.52237162 -0.37231836 -0.72101681 0.26199559] [-0.26335492 -0.92555649 0.24203288 -0.12413481] [ 0.58125401 -0.02109478 0.14089226 -0.80115427] [ 0.56561105 -0.06541577 0.6338014 0.52354627]] Eigenvalues [ 2.93035378 0.92740362 0.14834223 0.02074601] (from high to low)
- 42. 3. Select top k eigenvectors and transform data PCA in 3 Steps Eigenvectors [[ 0.52237162 -0.37231836 -0.72101681 0.26199559] [-0.26335492 -0.92555649 0.24203288 -0.12413481] [ 0.58125401 -0.02109478 0.14089226 -0.80115427] [ 0.56561105 -0.06541577 0.6338014 0.52354627]] Eigenvalues [ 2.93035378 0.92740362 0.14834223 0.02074601] [First 2 PCs of Iris]
- 43. Hyperparameter Optimization: GridSearch in scikit-learn
- 44. Non-Linear Problems - XOR gate k=11 uniform weights C=1 C=1000, gamma=0.1 depth=4
- 45. Kernel Trick Kernel function Kernel Map onto high-dimensional space (non-linear combinations)
- 46. Kernel Trick Trick: No explicit dot product! Radius Basis Function (RBF) Kernel:
- 47. Kernel PCA PC1, linear PCA PC1, kernel PCA
- 48. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classiļ¬cation/ Regression Model New DataPre-Processing Reļ¬nement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
- 51. Inspiring Literature P. N. Klein. Coding the Matrix: Linear Algebra Through Computer Science Applications. Newtonian Press, 2013. R. Schutt and C. OāNeil. Doing Data Science: Straight Talk from the Frontline. OāReilly Media, Inc., 2013. S. Gutierrez. Data Scientists at Work. Apress, 2014. R. O. Duda, P. E. Hart, and D. G. Stork. Pattern classiļ¬cation. 2nd. Edition. New York, 2001.
- 54. class1 class2 Which one to pick?
- 55. class1 class2 The problem of overļ¬tting Generalization error!