Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (19)
Andere mochten auch
Andere mochten auch (20)
Ähnlich wie Machine Learning + Analytics
Ähnlich wie Machine Learning + Analytics (20)
Mehr von Splunk
Mehr von Splunk (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Machine Learning + Analytics
- 4. 4 Agenda • Machine Learning 101 & Splunk • Demo of the Machine Learning Toolkit & Showcase • How to be successful with ML + Splunk
- 5. 5 Why do we need Machine Learning? - Improve Decision Making, Improve Future Actions - Forecast or Predict KPIs, Alert on Deviation - Uncover hidden trends or relationships All of this requires Diverse Data from across Many Silos. Lots of Unstructured, Real Time Data.
- 6. 6 Run The Business in Real-time Data From the Past Real-time Data Statistical Forecast T – a few days T + a few days Security Operations Center IT Operations Center Business Operations Center Predictive (Models) Descriptive (BI Tools, Data Lakes) Grey space
- 7. 7 What is “Learning”? [Prediction] • When we see thick clouds and an overcast sky, we predict that it’s likely going to rain [Estimation/ Regression] • Estimate how much an apartment costs based on its location, condition and prices of properties in that neighborhood [Classification/ Clustering] • Determine the gender of a person based on her/his features, hair style and the way s/he dresses [Anomaly Detection] • Identify the odd one out [Reinforcement Learning] • If I made a mistake this time, can I do better next time? All of us have had some experience in learning. But… what’s behind our experience? How do we translate that knowledge to code?
- 9. 9 ML 101: Existing Applications Recall: EXPLORE > FIT > APPLY > VALIDATE > REPEAT • Face detection: find faces in images • Spam filtering: identify SPAM messages • Shopping Recommendations: predict what customers would like to buy • Fraud detection: identify credit card transactions which may be fraudulent in nature • Weather forecast: predict whether or not it will rain tomorrow; estimate daily max/min
- 10. 10 Three Types of Machine Learning 1. Supervised Learning: generalizing from labeled data OR? Gather data: • Dimensions • Stem Length • Color • Etc.
- 11. 11 Three Types of Machine Learning 2. Unsupervised Learning: generalizing from unlabeled data Will my home sell? Gather data: • Square feet • Levels • Parks nearby • Schools • Zipcode • Etc.
- 16. 16 Splunk Machine Learning Toolkit Assistants: Guide model building, testing, & deployment for common objectives Showcases: Interactive examples for over 25 typical IT, security, business, IoT use cases Algorithms: 25+ standard algorithms available prepackaged with the toolkit SPL ML Commands: New commands to fit, test and operationalize models Python for Scientific Computing Library: 300+ open source algorithms available for use Build custom analytics for any use case Extends Splunk platform functions and provides a guided modeling environment
- 17. 17 What’s New in 2.0? 1 7 • New name and abbreviation • No event limits (removal of 50K limit on fitting models) • Configurable resource caps via mlspl.conf • Search head clustering support • Distributed / streaming apply • Scheduled fit • New algorithms (see Slide 7) – Feature engineering and selection – Stochastic gradient descent (e.g.) – ARIMA • Multi-algorithm support across Assistants • Scatterplot matrix viz • Alerting • Tooltips • In-app tours • Cluster Numeric Events assistant
- 18. 18 Splunk ML Algorithms(v2.0, .Conf2016) • ARIMA • SGDClassifier • SGDRegressor • DecisionTreeClassifier • DecisionTreeRegressor • AdaBoostRegressor • BernoulliNB • Birch • DBSCAN • ElasticNet • FieldSelector • GaussianNB • KMeans • KernelPCA • KernelRidge • Lasso • LinearRegression • LogisticRegression • OneClassSVM • PCA • RandomForestClassifier • RandomForestRegressor • Ridge • SVM • SpectralClustering • TFIDF • StandardScaler
- 22. 22 Summary: The ML Process 1. Get all relevant data to problem 2. Explore data, and fit predictive models on past / real-time data 3. Apply & validate models until predictions are accurate 4. Forecast KPIs & notable events associated to use case 5. Surface incidents to X Ops, who INVESTIGATES & ACTS Problem: <Stuff in the world> causes big time & money expense. Value Hypothesis Solution: Build ML model to forecast <possible incidents>, act pre-emptively & learn Operationalize
- 25. 25 Fit, Apply & Validate Models ML SPL – New grammar for doing ML in Splunk fit – fit models based on training data – [training data] | fit LinearRegression costly_KPI from feature1 feature2 feature3 into my_model apply – apply models on testing and production data – [testing/production data] | apply my_model Validate Your Model (The Hard Part) – Why hard? Because statistics is hard! Also: model error ≠ real world risk. – Analyze residuals, mean-square error, goodness of fit, cross-validate, etc. – Take Splunk’s Analytics & Data Science Education course
- 26. 26 ML Commands in Core SPL Cluster – groups events together based on how textually similar they are to each other. Anomalies – finds events or field values that are unusual or unexpected Predict - forecasts values for one or more sets of time-series data Kmeans - Kmeans clustering on events. Anomalousvalue - anomaly score for each field of each event, relative to the values of this field across other events. Anomalydetection - identifies anomalous events by computing a probability for each event and then detecting unusually small probabilities. • X11 - exposes seasonal trend in your time series. Associate – Change in entropy between two fields. Findkeywords - Given a set of numbered groups (from say cluster) calculates the common words found in each cluster. Analyzefields - What is the ability of a set of fields to predict a single field. Univariate analysis. And lots more Reference Docs -> http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/ListOfSearchCommands
- 27. 27 Don’t Forget: 80% of Data Science is Data Munging Trendline – Moving Averages of fields Erex- Use the erex command to extract data from a field when you do not know the regular expression to use Correlation – Co-occurence NOT correlation as per Pearson et., don’t mix this up. Autoregress – Copies one or more previous values for a field into an event. Contingency - Frequency distribution matrix. Cofilter - find how many times field1 and field2 values occurred together. Reference Docs -> http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/ListOfSearchCommands Stats, Eventstats, Streamstats, Timechart, Chart – stats reporting Eval - evaluate new fields And lots more
- 28. 28 What Now? 2 8 • Get the Machine Learning Toolkit from Splunkbase • Go watch Machine Learning Videos on Splunk Youtube Channel http://tiny.cc/splunkmlvideos • Go to Machine Learnings talks: https://conf.splunk.com/ – Advanced Machine Learning in SPL with the Machine Learning Toolkit by Jacob Leverich – Extending SPL with Custom Search Commands and the Splunk SDK for Python by Jacob Leverich • Several Customers and Partner Talks – Cisco, Scianta Analytics, Asian Telco, etc. • Early Adopter And Customer Advisory Program : mlprogram@splunk.com • Product Manager: Manish Sainani ms@splunk.com • Field Expert: Andrew Stein astein@splunk.com http://tiny.cc/splunkmlapp
- 35. 35 Reliable website updates Proactive website monitoring leads to reduced downtime 3 5 “Splunk ML helps us rapidly improve end-user experience by ranking issue severity which helps us determine root causes faster thus reducing MTTR and improving SLA” • Very frequent code and config updates (1000+ daily) can cause site issues • Find errors in server pools, then prioritize actions and predict root cause • Custom outlier detection built using ML Toolkit Outlier assistant • Built by Splunk Architect with no Data Science background ML Use Case Technical overview
- 39. 39 Energy Data 3 9 Process Data 86348;24.03.15 23:59:59; 140808,297; 140746,031;140919,500; 24-03-2015 01:00:59;EPIP02-03-A;SB;PPR;PR; PRODUCTION;PR;aRTC: accounted transaction (equip02_evnt_job_unit01);;;;;;;0,014;753,000 correlation over time (join) Production statusEnergy consumption - Transparency of equipment on shop floor level - Discover process weaknesses - Condition based and predictive maintenance - Optimization of energy efficiency of equipment - Optimization of energy purchasing process (forecast / predictive) … etc …. Increased efficiency Saved energy Saved $$$ Use cases