Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (18)
Ähnlich wie Spark-Zeppelin-ML on HWX
Ähnlich wie Spark-Zeppelin-ML on HWX (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Spark-Zeppelin-ML on HWX
- 1. Data Science at Scale Spark – Zeppelin - ML Kirk Haslbeck, Sr. Solution Engineer HWX
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Kirk Haslbeck - Hortonworks Sr. Solution Engineer @ Hortonworks Lead Architect for Trade Surveillance @ Morgan Stanley Masters in Data Mining @UMBC Computer Science Degree @ Mount Saint Mary’s University github.com/kirkhas/zeppelin-notebooks
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Spark – Apache Open Source Project
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Why do we need Spark? Distributed – Multi-threading is hard to do in Java but even if you get it right it isn’t distributed. It is limited to a single JVM Horizontal – Spark can take advantage of a modern data architecture. Scales out as a function of hardware. Data Science – Language R, Python both growing in popularity and great for statistical workloads but suffer from single threaded nature. Need for a top level computing language – SQL is great and provides a lot of what we need but not everything. Tradeoffs occur when SQL is better for some operations and a full programming language for others. Spark satisfies both!
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Spark API Languages
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Spark - Functional + Distributed = Concise and Powerful Spark Map Function Java Thread Pool Objective: we have a list of tasks and we want to pad each project timeline with 20% time buffer
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Why Spark chose Scala? Functional – Map, Filter, Fold, GroupBy – 5-10X code reduction Immutable – No state management, less headache, each operation is fully encapsulated. Thread Safety is the Biggest Challenge
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved RDDs, DataFrames and DataSets Resilient Distributed Dataset – Good for schema – case class Trade (sym: String, price: Double) DataFrame – SQL like operations, higher level object – aggregations, ordering Interoperability – Finally interop between Tables, Classes, and Vectors for Data Science. Borrowing the best from R, Scala and SQL. Impedance mismatch solved, no need for Domain Layer, Data Access Layer
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved RDD (low level) vs. DataFrames (new API)
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Spark 101 – Execution Model Spark Driver –Client side application that creates Spark Context Spark Context –Talks to Spark Driver, Cluster Manager to Launch Spark Executors Cluster Manager – E.g YARN, Spark Standalone, MESOS Executors – Spark worker bees
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Spark Engine in the HDP Stack Spark is first-class citizen of Hadoop
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Demo Show me the Code!
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Model Inputs Data Gathering Custom Logic Process Flow Evaluate Results
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What About Machine Learning?
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Machine Learning and Big Data Machine learning has advanced to the point where it more or less goes hand- in-hand with Big Data. Indeed, so popular is the technology that over a third of developers – some 36 percent – who are working on Big Data or advanced analytics projects use elements of machine learning, says a new study by Evans Data Corp. Machine Learning involves creating and improving complex algorithms that are able to analyze data automatically and identify patterns or predict outcomes based on the knowledge they have “learned”. As such, it has great potential for helping companies to better understand what their data is telling them.
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Where Can We Use Data Science? Healthcare • Predict diagnosis • Prioritize screenings • Reduce re-admittance rates Financial services • Fraud Detection/prevention • Predict underwriting risk • New account risk screens Public Sector • Analyze public sentiment • Optimize resource allocation • Law enforcement & security Retail • Product recommendation • Inventory management • Price optimization Telco/mobile • Predict customer churn • Predict equipment failure • Customer behavior analysis Oil & Gas • Predictive maintenance • Seismic data management • Predict well production levels
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Customer Use Cases with Spark Web Analytics - WebTrends Web Analytics for Marketing • Ingesting 13 Billion events/Day • Use Spark Streaming & Samza for Data Ingest • Extremely low latency: 40 milliseconds • Need more metrics for Spark Streaming • Wants 2 way SSL for Kafka Spark receiver Bank/Credit Card Real time monitoring and Fraud Detection • Monitor ATM with NiFi • Start with Log Aggregation • Tackle fraud detection next Railroad Company Real time view of state of track • Optimize the train maintenance • Large volume of track data, down to feel granularity • GeoSpatial analytics is critical Cable Company Optimize Advertising • Monitor channel changes with Spark Streaming • Correlate changes with Ads/Programming • Allocate Ads real time: Show ads to user who are watching a show and will stay for > over 20 seconds • How to optimize Spark App development
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Example: Credit Card Fraud Detection
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Building a Model Show of hands, how many have built a “Model”? What are some limitations? – Conditional based logic: if/else binary decisions If you need a lot of data to build a good model, what tools can you use? – Data volumes can eliminate the possibility of desktop tools Sampling? – Well… we better get an even distribution of true and false positives in each sample, but wait that requires data munging, back to what tools can we use. Security Concerns? – Extracting data from it’s secure resting place and pushing it into other environments, often times unsecure files or desktops where Matlab or R can be installed. Collaboration – Push processing to the data using modern distributed tooling.
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved “All models are wrong, some are useful” George E. P. Box Most limiting factor is the data, with modern systems we are now able to capture more data and hopefully produce better insights
- 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Credit Card Fraud Requirement: Detect fraudulent transactions. Goal: Save the card company money and build trust amongst card users. Cut down on fraudulent crime Functional Requirement: Detect fraud in under 2 seconds at point of sale. Learn, adapt and make smarter decisions over time. Design – Distance: How far can one travel over a period of time before it is fraudulent? – Category: How can we detect a purchase that a customer wouldn’t likely make? – Frequency: How can we detect purchasing patterns that do not resemble the card holder? Ideas? – White board some conditional logic, egregiousness vs binary – Back test the data – Build a model per card holder?
- 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Rules, Statistics, Machine Learning Rule Based Logic – Great for checking conditions that can prove to be 100% accurate. Easy to build and no reason to over engineer. – Example: Spending Limit. Card holder limit = $2,000 • If (currentPurchaseAmount + balance > 2,000) then deny transaction Statistics – Mean, median, mode, variance, deviation – Anomaly detection. Outliers. (i.e. womens retail example) Machine Learning – Supervised – Unsupervised – Trainable – Adapt over time
- 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Discovery Gathered all Credit Card Transactions – Problem is they didn’t make sense – No identifiable patterns, no log normal curves – Gas $45, Chipotle $8.50, Steak dinner $88, Amazon shoes $55 Classification
- 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Outlier Detection: identify abnormal patterns Example: identify anomalies Features: - Time frequency - Category - Amount - Distance
- 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Page 26 Hortonworks Data Flow
- 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Page 27 Hortonworks Data Flow
- 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Machine Learning Continued
- 29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Classification: predicting a category Some techniques: - Naïve Bayes - Decision Tree - Logistic Regression - SGD - Support Vector Machines - Neural Network - Ensembles
- 30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Regression: predict a continuous value Some techniques: - Linear Regression / GLM - Decision Trees - Support vector regression - SGD - Ensembles
- 31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Unsupervised Learning: detect natural patterns Age State Annual Income Marital status 25 CA $80,000 M 45 NY $150,000 D 55 WA $100,500 M 18 TX $85,000 S … … … … No labels Model Naturally occurri (hidden) structur
- 32. 32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Clustering: detect similar instance groupings Some techniques: - k-means - Spectral clustering - DB-scan - Hierarchical clustering
- 33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Getting the Proper Fit Over-fitting: Model over-fits training set, but does not generalize well to new inputs Under-fitting: Model can’t predict accurately
- 34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Business Intelligence vs Data Science R and Matplotlib now available
- 35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved R and Matlab Visuals in Zeppelin
- 36. 36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Matplotlib with Python
- 37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Appendix – Links to content Github https://github.com/kirkhas/zeppelin-notebooks Credit Card Fraud (real-time ML) https://community.hortonworks.com/articles/38457/credit-fraud-prevention-demo-a-guided-tour.html Monte Carlo / VaR https://community.hortonworks.com/articles/39096/predicting-stock-portfolio-gains-using-monte-carlo.html Stock Variance https://community.hortonworks.com/repos/32713/stock-variance-using-zeppelin.html
- 38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hinweis der Redaktion
- We have other engines out there, plenty of them and we have SQL / Java
- Language trends, thirst for answers.
- Example each task takes a certain amount of time but we want to buffer the time and leave room for Murphy’s law that something will go wrong or take a bit longer. 20% more time. Each unit is independent.
- Concise, declarartive but also provides greater description to CPU on how to handle the problem.
- Anywhere is the real answer. Here’s a few examples that have been bubbling up among our customers. Speaker: pick one of these and describe, briefly, an example that you know about in 4-5 sentences. One ‘close’ to the audience of course is the best. There’s separate use case deck by industry in the workshop wiki page that you can use for ideas.
- Example from movie: “IDENTITY THEFT” Some common applications of outlier detection include: Fraud detection: Purchasing behavior of a credit card owner usually changes when the purchasing behavior of a credit card owner usually changes when the card is stolen and the abnormal buying patterns can indicate fraud. Medicine: Unusual test results may indicate an underlying health issue Sports: Exceptional players may appear as outliers in particular parameters and placed in positions where the team can most benefit
- Gas, Convenience, retail,
- Before we can detect an outlier, we have to define it. The most intuitive definition I’ve seen is from a 1980 book Identification of Outliers (Hawkins): “An outlier is an observation which deviates so much from the other observations as to arouse suspicions that it was generated by a different mechanism” Speaker: It’s worth repeating that all of these topics a deep enough to have entire university library shelves filled with books on the topic. We’re just skimming the surface. Anomaly detection is related to clustering, but almost its inverse. If points that are similar to each other cluster together (representing “normal” behavior or patterns), we want to find the points that are NOT in any cluster.
- What: Identify what group a new observation belong in. This is a simplified visual example, taken from Andrew Ng’s ML class. Plotted are only 2 dimensions (features) from the full feature vector – age and tumor size. The instances provided are “labeled” – in this case marked in red/x vs. blue/circle. Note that some red instances are inside the blue cluster, which represents the fact that learning sets sometimes introduce noise, making the learning task not trivial. Similarly some blue points are inside the red cluster.
- Regression is supervised learning where instead of predicting a category (like malignant or benign from the previous examples) we predict a “value” – a number. In this example (again from Andrew Ng’s class) we are trying to predict the “price of a house” given a single variable: size in feet. Clearly more complex models in multiple dimensions would be better; for example, we can use other features like “age of house” or “number of previous owners”, “geographic location” or “score for closest public school”
- With unsupervised learning, we again have as input a feature matrix with rows as instances and columns as variables, but NO LABELS. Now the goal is to find a label (cluster number) for each instance, but we are not learning a given function to match, rather trying to figure out the natural way instances may be grouped together. Note that we are usually NOT given the number of desired cluster (often called “K”), and may need to determine this on our own.
- Over-fitting means the model performs very well on the training set but does not generalize well so results on unseen data are poor. As shown in the diagram, this means the model learned the specific granular details of the training set and not the generic function it was meant to learn. This is why we “evaluate” on the validation set (and not the training set), because if we measured error on the training set we may get a false sense of performance if the model is over-fitting. Under-fittingmeans the model doesn’t have enough degrees of freedom to learn the needed model, and usually has a high bias. Underfitting is often a result of an excessively simple model. In practice you won’t encounter underfitting very often. Data sets that are used for predictive modelling nowadays often come with too many predictors, not too few. Nonetheless, when building any predictive model, you should use validation or cross-validation to assess predictive accuracy and avoid these problems. Here we may have many observations, but too few features (matrix is tall and narrow).