The next generation of data scientists will be asked to build predictive models that can extract inferences from very large datasets which are unobservable at the surface, even to the best domain experts. Microsoft has access to some truly large data sets, web and search data from the Bing search engine and social data through collaborations with Twitter. In this talk, we show you how a small team of data scientists used this data to build the Bing Predicts engine — a collection of machine learnt predictive models that is beating industry experts at predicting the outcome of events like the Super Bowl, the Oscars, elections and referendums and even breakthroughs in health sciences. The talk will also give a preview of how organizations can adopt a big data mindset to generate and experiment with large data sets and to make amazing predictions using their own data.

1. Predicting the Future: Surprising
revelations from truly Big Data
Pushpraj Shukla
Pushpraj.Shukla@Microsoft.com

5. Bing Predicts

11. Knowledge
Understanding what the data means
Information
Indexing, aggregating, categorizing
the data
Data
Unstructured and unorganized

13. What can we predict?

14. *

15. *Data from http://www.americanidolbettingodds.com/

17. Did we get lucky?

22. But wait, there’s more

24. Data science skepticism
“The Unreasonable Effectiveness of Mathematics in the Natural
Sciences”, Eugene Wigner, 1960.
• Isn’t it remarkable how well we can predict physical phenomena using
simple equations like F = ma or PV = nRT?
“The Unreasonable Effectiveness of Data”, Alon Halevy, Peter Norvig, &
Fernando Pereira, 2009.
• Isn’t it remarkable how well we can predict human phenomena using
simple statistical models and massive quantities of training data?

26. Task: Classify events into "tau tau decay of a Higgs boson"
versus "background

27. Data Science and Machine Learning
@Microsoft
‘Big’ Tasks – Predicting human behavior and preferences, Speech,
Translation, Image and Video recognition
‘Big’ Data = SCALE
- Huge data. More data = better accuracy
- Huge models. More features = better accuracy
- Huge human feedback
‘Big’ Infrastructure and Experimental Ecosystems

28. Machine learning for everyone
Big Tasks:
• Be curios, make bold hypotheses
• Ask challenging questions
• Disrupt polls and traditional data collection
Big Data
• Create data: high volume public/private data streams
• Join multiple data sources: Weather, Public Health, Traffic, Crime etc.
• Utilize the Crowd: Mechanical Turk, Kaggle
Smart Infrastructure
• Re-use solutions: public APIs in Text Analytics, Speech,Image
• Cloud-based ML solutions

30. Collaboration and learning
Existing templates and APIs for:
Stream Analytics
Anomaly Detection
Sentiment Analysis
Churn prediction
Sales Forecasting
Face recognition
Speech Recognition
.. And many more
Azure ML : ML Studio pictured below

31. Questions?
Some References:
Bing Predicts: http://www.bing.com/explore/predicts?setmkt=en-us
AzureML : http://gallery.cortanaanalytics.com/
Reach me at Pushpraj.Shukla@Microsoft.com