Bring your own device and join us for an interactive session and hands-on experience with Pentaho. We’ll provide a hosted virtual machine for the first 30 people in the room so you can click along with your instructor while you solve common data challenges. This session addresses how data scientists strive to find the right balance of complexity, performance and accuracy in their predictive models. Unfortunately, up to 80% of a data scientist’s time is spent on data preparation instead of analyzing the data and applying advanced algorithms. With integrations for languages like R and Python, and for machine learning packages like Weka and Spark MLlib; Pentaho streamlines the process of training, tuning, building and testing models. In this session, you will learn how to use Pentaho Data Integration to efficiently prepare data, engineer features, test models, and deploy and operationalize the model. Caio Moreno Professional Services Senior Consultant Pentaho, a Hitachi Group Company James O'Reilly Training & Development Specialist Pentaho, a Hitachi Group Company

1. Hands-on Training: Tackling Data
Science Challenges with PDI
Caio Moreno de Souza
Pentaho Senior Consultant, EMEA & APAC Services Team, Hitachi Vantara
James O’Reilly
Pentaho Training & Development Specialist, Hitachi Vantara

2. Topics: Tackling Data Science Challenges with PDI
• Overview of Machine Learning
• 4 Steps of Machine Learning with Pentaho
• Demonstration
– Credit Card Fraud
• Summary
• Q&A

3. Machine Learning

4. Tackling Data Science Challenges with PDI
Overview
• Machine Learning (ML) uses algorithms to find patterns in data, and then uses a
model that recognizes those patterns to make predictions on new data.

5. Tackling Data Science Challenges with PDI
Overview
• Supervised learning
algorithms use
labelled data
• Unsupervised
learning algorithms
find patterns in
unlabelled data.

6. Tackling Data Science Challenges with PDI
Supervised Learning – Predictive Analysis
• Supervised algorithms use labelled data in which both the input and target
outcome, or label, are provided to the algorithm.

7. Tackling Data Science Challenges with PDI
Examples

8. Machine Learning
with Pentaho

9. Tackling Data Science Challenges with PDI
CRISP-DM
• CRoss Industry Standard
Process for Data Mining
• The sequence of the phases is
not strict and moving back and
forth between different phases
is always required.
Business
Understanding
Data
Understanding
Data
Preparation
Modeling
Evaluation
Deployment
Data

10. Tackling Data Science Challenges with PDI
Examples
• Business Understanding
– This initial phase focuses on understanding the project objectives and requirements
from a business perspective, and then converting this knowledge into a data mining
problem definition, and a preliminary plan designed to achieve the objectives.
• Data Preparation
– The data preparation phase covers all activities to construct the final dataset (data
that will be fed into the modelling tool(s)) from the initial raw data.
• Modelling
– In this phase, various modelling techniques are selected and applied, and their
parameters are calibrated to optimal values.
• Evaluation
• Deployment

11. Tackling Data Science Challenges with PDI
Examples
• Evaluation
– At this stage in the project you have built a model (or models) that appears to have
high quality, from a data analysis perspective. Before proceeding to final deployment
of the model, it is important to more thoroughly evaluate the model, and review the
steps executed to construct the model, to be certain it properly achieves the business
objectives.
• Deployment
– Creation of the model is generally not the end of the project. Even if the purpose of
the model is to increase knowledge of the data, the knowledge gained will need to
be organized and presented in a way that is useful to the customer.

12. Machine Learning
Demonstration

13. Tackling Data Science Challenges with PDI
Data Modelling Software
• Python is an interpreted, object-oriented, high-level programming language
with dynamic semantics.
• RStudio is an integrated development environment (IDE) for R. It includes a
console, syntax-highlighting editor that supports direct code execution, as well
as tools for plotting, history, debugging and workspace management.
• Anaconda is a freemium open source distribution of the Python and R
programming languages for large-scale data processing, predictive analytics, and
scientific computing, that aims to simplify package management and
deployment

14. Tackling Data Science Challenges with PDI
Lab: 1-1 Python using Spyder

15. Tackling Data Science Challenges with PDI
Lab 1-2: R Studio

16. Tackling Data Science Challenges with PDI
Workshop 1-1: Shell script
• Execute using Shell Script Job Entry

17. Machine Learning
Demonstration

18. Tackling Data Science Challenges with PDI
Demonstration: Detect credit card fraud
• A direct retailer wants to reduce losses due to orders involving fraudulent use of
credit cards. Orders are accepted via phone and their web site and goods ship
directly to the customer.
• A relational database stores basic customer details such as:
– customer name
– date of birth
– billing address
– preferred shipping address.
• As orders come in, they are stored in a database.
• A report of historical instances of fraud is contained in a CSV spreadsheet.

19. Machine Learning with Pentaho
4 steps to Machine Learning

20. Data Preparation
Feature Engineering

21. Tackling Data Science Challenges with PDI
Data Preparation
• With the goal of preparing a dataset for ML, we can use PDI to combine these
disparate data sources and engineer some features for learning from it.
• To begin with customer data is joined from several relational database tables,
and then blended with transactional data from a database and historical fraud
occurrences contained in a CSV file.
• Following this, there are steps for deriving additional fields that might be useful
for predictive modelling. These include:
– computing the customer's age,
– extracting the hour of the day the order was placed,
– and setting a flag to indicate whether the shipping and billing addresses have the same
zip code.

22. Tackling Data Science Challenges with PDI
Data Preparation

23. Tackling Data Science Challenges with PDI
Data Preparation
• This process culminates with output of flattened (a Data Scientist’s
preferred data shape) data in both CSV and ARFF (Attribute Relational
File Format) data, the latter being the native file format used by PDM
(Pentaho Data Mining, AKA WEKA).
• We end up with 100,000 examples (rows).
• From this list, for the purposes of predictive modelling, we can drop the
customer name, ID fields, email addresses, phone numbers and physical
addresses. These fields are unlikely to be useful for learning purposes
and, in fact, can be detrimental due to the large number of distinct
values they contain.

24. Train, Tune, Test
Machine Learning
Models

25. Machine Learning with Pentaho
Train, Tune, Test Machine Learning Model
• So, what does the data scientist do at this point?
• Typically, they will want to get a feel for the data by examining simple summary
statistics and visualizations, followed by applying quick techniques for
assessing the relationship between individual attributes (fields) and the target
of interest which, in this example, is the "reported_as_fraud_historic" field.

26. Tackling Data Science Challenges with PDI
Train, Tune, Test Machine Learning Model: ktr_ml_r_train_model_fraud_v1.ktr
• To build a model directly in R (rather than via WEKA's wrapper classifiers), we
can simply add a R Script Executor step to the transformation.
• PDI materializes incoming batches of rows as a data frame in the R environment.
• The following figure shows using this step to execute code that builds and saves
a Random Forest regression model.

27. Tackling Data Science Challenges with PDI
Train, Tune, Test Machine Learning Model: ktr_ml_r_train_model_fraud_v1.ktr

28. Tackling Data Science Challenges with PDI
Train, Tune, Test Machine Learning Model: ktr_ml_r_train_model_fraud_v1.ktr

29. Tackling Data Science Challenges with PDI
Train Machine Learning Model: ktr_ml_r_train_model_fraud_v1.ktr
• You’re now going to train the Random Forest model..!
• Open the transformation: ktr_ml_r_train_model_fraud_v1.ktr
• Copy / Paste the train_model.txt script from the ..scripts folder..

30. Deploy Predictive
Models

31. Tackling Data Science Challenges with PDI
Predict / Deploy Machine Learning Model: ktr_ml_r_predict_fraud_v1.ktr
• You’re now going to predict / deploy the Random Forest model..!
• Open the transformation: ktr_ml_r_predict_fraud_v1.ktr
• Copy / Paste the predict_model.txt script from the ..scripts folder..

32. Tackling Data Science Challenges with PDI
Deploy / Predict Model:ktr_ml_r_predict_fraud_v1.ktr

33. Tackling Data Science Challenges with PDI
Deploy / Predict Model:ktr_ml_r_predict_fraud_v1.ktr

34. Tackling Data Science Challenges with PDI
Results

35. Summary
What we covered today:
• Overview of Machine Learning
• 4 Steps of Machine Learning with Pentaho
• Demonstration
– Credit Card Fraud

36. Next Steps
Want to learn more?
• Talk to us during Pentaho World 2017 or send us an e-mail
caio.moreno@HitachiVantara.com / james.oreilly@hitachivantara.com
• Meet-the-Experts:
– https://www.pentahoworld.com/meet-the-experts

37. Any Questions?

39. Appendix A
Random Forest
• Random forests or random decision forests are an ensemble learning method
for classification, regression and other tasks, that operate by constructing a
multitude of decision trees at training time and outputting the class that is the
mode of the classes (classification) or mean prediction (regression) of the
individual trees.
• Random decision forests correct for decision trees' habit of overfitting to their
training set.

40. Appendix A
Unsupervised
• Unsupervised learning, also sometimes called descriptive analytics, does not
have labelled data provided in advance. These algorithms discover similarities,
or regularities in the input data. An example of unsupervised learning is
grouping similar customers, based on purchase data.