1. Machine Learning
&
Decision Trees
Nithum Thain
January 12th, 2013

2. Overview
• The Data Science Value Chain
• Common Uses for Machine Learning
• The Art of Prediction & Classification
• Introduction to Decision Trees
• The Data Ninja Methodology

3. The Data Science Value Chain
Visualization Strategy,
Storage & Marketing,
Collection &
Maintenance Product,
Analysis Operations
Machine Learning
lives here

4. Overview
• The Data Science Value Chain
• Common Uses for Machine Learning
• The Art of Prediction & Classification
• Introduction to Decision Trees
• The Data Ninja Methodology

5. Machine Learning vs. Artificial Intelligence
• Artificial Intelligence is a set of tools that allow machines to
perform higher order functions. These include natural language
processing, robotics, knowledge representation, etc.
• Machine Learning is a subset of artificial intelligence. It is a set of
(usually statistical) tools that allow machines to detect and extract
patterns from data.

6. Subdomains of Machine Learning
Unsupervised Learning
• Clustering
• Optimization
• Recommendation Systems
Supervised Learning
• Prediction & Classification
Reinforcement Learning

7. Clustering

8. Optimization

9. Recommendation Systems

10. Recommendation Systems

11. Recommendation Systems

12. Reinforcement Learning

13. Reinforcement Learning

14. Prediction & Classification

15. Overview
• The Data Science Value Chain
• Common Uses for Machine Learning
• The Art of Prediction & Classification
• Introduction to Decision Trees
• The Data Ninja Methodology

16. What is a Prediction Problem?
• A set of known input variables.
• An unknown output variable.
• A training set of data for which both the inputs and
outputs are known.

17. A Useful Formulation
Output Variable
Training Set
Test Set

18. The Algorithms Are Many
• Regression
• Decision Trees
• Neural Networks Each has it’s own
strengths and
• Support Vector Machines weaknesses.
• Random Forests
• Naive Bayes Classifier

19. Prediction vs. Classification
1.618033988
7498948482
0458683436
.....

20. Break Time!

21. Overview
• The Data Science Value Chain
• Common Uses for Machine Learning
• The Art of Prediction & Classification
• Introduction to Decision Trees
• The Data Ninja Methodology

22. What is a Decision Tree?

23. Why Not Automate It?

24. I Did!
Internet Friends?
Video Games? XBOX 360
Friends? Friends?
PS3 Wii PC PS3

25. How Our Algorithm Works
1. Start with the “root” node.
2. Check if the data all has the same output variable. If so, then
you are done.
3. Check how every possible output variable splits the data.
4. Choose the one that splits the data MOST
- The one which reduces the variance in the output variable
in the resulting sets.
5. Repeat the process for the resulting “true” node and “false”
node.

26. A Picture
Friends?

27. A Picture
Internet Friends?
Friends?

28. A Picture
Internet Friends?
Friends?
XBOX 360

29. A Picture
Internet Friends?
Friends?
Video Games? XBOX 360

30. A Picture
Internet Friends?
Friends?
Video Games? XBOX 360
Friends?

31. A Picture
Internet Friends?
Friends?
Video Games? XBOX 360
Friends? Friends?
PS3 Wii PC PS3

32. Coding Time

33. The Classes and Functions We Will Build:
Classes
• decisionnode: The basic building block of our tree
Functions
• divideset: Splits the tree into two sets based on a variable
• variance: Calculates the variance of the output variable in a set
• buildtree: Builds the tree according to the algorithm described
• classify: For any new data points, uses the tree to predict their value
• printree: Prints a text-based version of the full decision tree

34. The decisionnode Class
if variable >= value

35. The decisionnode Class
if variable >= value
or result

36. Overview
• The Data Science Value Chain
• Common Uses for Machine Learning
• The Art of Prediction & Classification
• Introduction to Decision Trees
• The Data Ninja Methodology

37. The Data Ninja Methodology
1. Find the appropriate data
2. Play with the data (plot, sort, examine)
3. Clean the data
4. Choose the appropriate tool for analysis
5. Apply the tool
6. Repeat steps 2-6 until something works
7. ...
8. Profit!

38. Let’s Try Predicting Housing Prices!

39. Beware Overfitting!

40. How Can We Improve Our Results?

41. Appendix

42. Neural Network