1. Introduction to Machine Learning Isabelle Guyon isabelle @ clopinet .com

5. Applications inputs training examples 10 10 2 10 3 10 4 10 5 Bioinformatics Ecology OCR HWR Market Analysis Text Categorization Machine Vision System diagnosis 10 10 2 10 3 10 4 10 5

9. Challenges inputs training examples 10 10 2 10 3 10 4 10 5 Arcene, Dorothea, Hiva Sylva Gisette Gina Ada Dexter, Nova Madelon 10 10 2 10 3 10 4 10 5 NIPS 2003 & WCCI 2006

10. Ten Classification Tasks 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0 50 100 150 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0 50 100 150 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0 50 100 150 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0 50 100 150 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0 50 100 150 ADA GINA HIVA NOVA SYLVA 0 5 10 15 20 25 30 35 40 45 50 0 20 40 ARCENE 0 5 10 15 20 25 30 35 40 45 50 0 20 40 DEXTER 0 5 10 15 20 25 30 35 40 45 50 0 20 40 DOROTHEA 0 5 10 15 20 25 30 35 40 45 50 0 20 40 GISETTE 0 5 10 15 20 25 30 35 40 45 50 0 20 40 MADELON Test BER (%)

11. Challenge Winning Methods BER/<BER>

12. Conventions X={x ij } n m x i y ={y j }  w

13. Learning problem Colon cancer, Alon et al 1999 Unsupervised learning Is there structure in data? Supervised learning Predict an outcome y . Data matrix: X m lines = patterns (data points, examples): samples, patients, documents, images, … n columns = features: (attributes, input variables): genes, proteins, words, pixels, …

15. Artificial Neurons f( x ) = w  x + b Axon Synapses Activation of other neurons Dendrites Cell potential Activation function McCulloch and Pitts, 1943 x 1 x 2 x n 1  f( x ) w 1 w 2 w n b

16. Linear Decision Boundary x 1 x 2 x 3 hyperplane x 1 x 2

17. Perceptron Rosenblatt, 1957 f( x ) f( x ) = w   ( x) + b  1 ( x ) 1 x 1 x 2 x n  2 ( x )  N ( x )  w 1 w 2 w N b

18. NL Decision Boundary x 1 x 2 x 1 x 2 x 3

19. Kernel Method Potential functions, Aizerman et al 1964 f( x ) =  i  i k ( x i , x ) + b k( x 1 ,x ) 1 x 1 x 2 x n   1  2  m b k( x 2 ,x ) k( x m ,x ) k(. ,. ) is a similarity measure or “kernel”.

24. Multi-Layer Perceptron Back-propagation, Rumelhart et al, 1986  x j   “hidden units” internal “latent” variables

25. Chessboard Problem

27. Iris Data (Fisher, 1936) Linear discriminant Tree classifier Gaussian mixture Kernel method (SVM) setosa virginica versicolor Figure from Norbert Jankowski and Krzysztof Grabczewski

28. Fit / Robustness Tradeoff x 1 x 2 15 x 1 x 2

29. Performance evaluation x 1 x 2 f( x ) = 0 f( x ) > 0 f( x ) < 0 f( x ) = 0 f( x ) > 0 f( x ) < 0 x 1 x 2

30. Performance evaluation x 1 x 2 f( x ) = -1 f( x ) > -1 f( x ) < -1 f( x ) = -1 f( x ) > -1 f( x ) < -1 x 1 x 2

31. Performance evaluation x 1 x 2 f( x ) = 1 f( x ) > 1 f( x ) < 1 f( x ) = 1 f( x ) > 1 f( x ) < 1 x 1 x 2

32. ROC Curve 100% 100% For a given threshold on f(x), you get a point on the ROC curve. Actual ROC 0 Positive class success rate (hit rate, sensitivity) 1 - negative class success rate (false alarm rate, 1-specificity) Random ROC Ideal ROC curve

33. ROC Curve Ideal ROC curve (AUC=1) 100% 100% 0  AUC  1 Actual ROC Random ROC (AUC=0.5) 0 For a given threshold on f(x), you get a point on the ROC curve. Positive class success rate (hit rate, sensitivity) 1 - negative class success rate (false alarm rate, 1-specificity)

34. Lift Curve O M Fraction of customers selected Hit rate = Frac. good customers select . Random lift Ideal Lift 100% 100% Customers ranked according to f(x); selection of the top ranking customers. Gini=2 AUC-1 0  Gini  1 Actual Lift 0