Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (15)
Ähnlich wie Building Predictive Analytics on Big Data Platforms
Ähnlich wie Building Predictive Analytics on Big Data Platforms (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Building Predictive Analytics on Big Data Platforms
- 1. A Day of Empowerment Building Predictive Analytics on Big Data Platforms
- 2. 1. Opportunity: Big Data 2. Demystifying Predictive Analytics 3. Taking advantage of combined power
- 3. Striving for an “unfair” competitive advantage
- 5. Old Days
- 6. New Days
- 7. Big Data could be looking like rubbish
- 9. “Data are becoming the new raw material of business” - Craig Mundie, head of research and strategy, Microsoft
- 10. Modeling true risk Network data analysis to predict failure Customer churn analysis Threat analysis Recommendations Feature Usage analysis Ad targeting …
- 11. Collect and Store • Complex data (text files, audio, video, images, …) • Multiple sources • Lots of data Process • Batch processing • Parallel execution • Cluster solution Analyze • • • • • Simple visualization (reports, dashboard) Text mining Sentiment analysis Prediction models Collaborative filtering
- 13. Event sources (Log files, Windows Event Log, WMI, SNMP, database, etc.) Event Storage Event Aggregation and Transformation Event Transport Event Serialization and Archiving Event Processing and Analytics Presentation Query Engine Interactive Search User Full-text Search engine Event DB Rules Engine Reports and Dashboards Full-text Index Predictive Analytics Alerts Visualization E-mail, SMS, SNMP, etc. Operational Management Tools Event Ingestion
- 14. Event sources (Log files, Windows Event Log, WMI, SNMP, database, etc.) Event Storage Event Transport Event Aggregation and Apache Flume Transformation Event Serialization and Archiving Protobuf, Avro, Thrif t, MessagePack Event Processing and Analytics Presentation Query Engine Impala Interactive Search Custom User Full-text Solr, ElasticSe Search engine arch Full-text Event DB HDFS, Hbase, Cas Index sandra Rules Engine Drools Reports and JasperSoft, Dashboards Tableau Predictive Analytics R Alerts Visualization Custom E-mail, SMS, SNMP, etc. Operational Management Tools Event Ingestion Cloudera Manager, Apache Ambari
- 15. “The idea that the future is unpredictable is undermined every day by the ease with which the past is explained” ― Daniel Kahneman, Thinking, Fast and Slow
- 16. More data is available for companies Storage technologies allow to store and operate it Advanced analytics could be applied to this new data to achieve competitive advantage
- 17. Descriptive Diagnostic Predictive Prescriptive What happened? Why did it happen? What is going to happen? What should we do about that? Hindsight Insight Foresight
- 18. Senior (Executive) Management Ambiguity The goals to be achieved or the problem to be solved is unclear Alternatives are difficult to define Information about outcomes is unavailable. Uncertainty Middle Management Managers know which goals they wish to achieve. Information about alternatives and future events is incomplete. Risk Junior (Line) Management A decision has clear goals and good information is available, but the future outcomes associated with each alternative are subject to chance. Certainty All of the information the decision maker needs is fully available
- 19. Define objective • Increase customer satisfaction level • Identify prospective customers • Identify crossselling opportunities • Decrease time to market • Decrease costs of marketing campaigns Identify data sets Design the model • Historical data on • Classification model for Internet customers from users defining CRM system what one is • Geographical interested in location data • Smartphone data • Adaptive control models for • Social network managing IT and data network • Text data from the infrastructure Internet pages • Probabilistic • Image data from model for defining the medical credit worthiness sources Design the solution • Data storage type • Logical database design • Availability and scalability of the solution • Integration into corporate information environment • Solution deployment model Implement the solution • Add new functionality to the existing corporate BI platform • Implement new BI solution • Enrich existing business system (CRM, ERP) with the predictive analytics functionality
- 20. Business Tasks Model Family Algorithms • Define prospective customers • Define traffic jams in the city • Recommend restaurants and menus • Adjust UI to the particular user • Classify body part on X-Ray image • Define market niche • Define influencers in the social networks • Define similar customers or projects in portfolio • Define informal groups in the organization • Define fraud bank transaction • Define network intrusion attempts • Provide automatic aircraft engine testing • Provide automatic IT infrastructure monitoring • Provide clinical test analysis • Define the best price for the goods or services to maximize profits • Define best working schedule for the store • Define best amount of production • Define best business rules Classification Clustering Anomaly Detection Optimization • Naïve Bayes • Logistic regression • Support Vector Machines • Neural Networks • K-Means • K nearest neighbor • Self-organized maps • Mixture of Gaussians • Mixture of Gaussians • Self-learning anomaly detection • • • • • Gradient descent Simplex method Newton’s method Normal equations Genetic algorithms
- 21. Google to Buy Waze for $1.3 Billion Xerox plans to clear traffic on I-10 The promise of better data has MetLife investing $300M in new tech Gracenote did a whole business on recommending music Obama’s data scientists built a volunteer army on Facebook
- 23. Description: Cloud-based service for providing more accurate estimates of the credit worthiness (loan scoring) using publicly available data from social networks. Service is oriented to be used by banks. Technologies: Amazon EC2 MySQL SAP HANA R JAVA Credit Score
- 24. Facebook Twitter LinkedIn API Processing Preprocessing MySQL (data filtering, data cleansing) SAP HANA Credit scoring API (scoring model)
- 25. Description: Computer aid diagnostic system that can recognize human body part on X-Ray image and detect broken or fractured bones X-Ray Image Technologies: Matlab/Octave Python PyBrain NumPy SciPy Analytical Engine This is a hand. Broken bone detected
- 27. Big Data and NoSQL Data Warehouse Data Integration BI Platforms
- 28. Big Data Analytics Predictive Analytics Data Science Service Data Integration Data Warehousing Data Visualization and Analysis