Data Science deals with the extraction of valuable insights from an incredible number of sources in an endless number of formats. This session will go through a typical workflow using practical tools and tricks. This will give you a basic understanding of Data Science in the Cloud. The examples will show the steps that are needed to build and deploy a model to predict traffic collisions with weather data.

1. Data Science
in the Cloud
Margriet Groenendijk
IP EXPO Nordic
Stockholm - 20 September 2017

2. What is…
Data Science?
@MargrietGr

3. Data Science?What is Data Science?
@MargrietGr http://visual.ly/exports-and-imports-scotland

4. Data Science?What is Data Science?
@MargrietGr

5. Data Science?What is Data Science?
@MargrietGr

6. Data Science is…
Big Data?
@MargrietGr

7. Data Science is…
Extracting new
insights from data
@MargrietGr

8. Data Science is…
Using the
Scientific Method
@MargrietGr

9. PredictionsHypothesis TestsObservations
@MargrietGr
Scientific Method

10. @MargrietGr
Discover
Data
Use Data Publish
Data
Socialize
Data
The ideal workflow

11. @MargrietGr
In reality…

12. @MargrietGr
In reality…
https://medium.com/towards-data-science/the-ten-fallacies-of-data-science-9b2af78a1862

13. Data Science
in the Cloud
@MargrietGr

14. Normal Day
325TB per day
2.4 Billion API requests every
15 minutes
50,000 videos played
@MargrietGr
The Weather Company
Hurricane Harvey
500TB per day
3.4 Billion API requests every
15 minutes
750,000 videos played

15. Example
@MargrietGr
Traffic
Collisions
and Weather
www.pexels.com/photo/blur-cars-dew-drops-125510/

16. Collect Data Store Data Explore Data Publish Results
@MargrietGr

17. Collect Data Explore Data Publish Results
@MargrietGr
Store Data

18. NYPD Traffic Collisions
@MargrietGr
812,526
Collisions since April
2014
https://data.cityofnewyork.us/Public-Safety/NYPD-
Motor-Vehicle-Collisions/h9gi-nx95

19. Weather
@MargrietGr
Historic
Weather
https://business.weather.com/products/
weather-data-packages

20. Collect Data
Traffic
Collisions
Historic
Weather
Store Data Explore Data Publish Results
@MargrietGr

21. Store Data Object Store Relational Database NoSQL Database
@MargrietGr

22. Store Data Object Store
Static data
Unstructured data
Relational Database NoSQL Database
@MargrietGr

23. Store Data Object Store
Static data
Unstructured data
Relational Database
Tables with a fixed
schema
NoSQL Database
@MargrietGr

24. Store Data Object Store
Static data
Unstructured data
Relational Database
Tables with a fixed
schema
NoSQL Database
Everything else
@MargrietGr

25. Collect DataCollect Data
Traffic
Collisions
Historic
Weather
Store Data
Object-Store
Explore Data Publish Results
@MargrietGr

26. @MargrietGr https://stackoverflow.blog/2017/09/06/incredible-growth-python/

27. @MargrietGr https://stackoverflow.blog/2017/09/14/python-growing-quickly/

28. Explore data in Jupyter notebooks
@MargrietGr

29. Notebooks in the Cloud
@MargrietGr
https://datascience.ibm.com/

30. @MargrietGr
Organized in Collaborative Projects

31. Explore data in notebooks in the Cloud
@MargrietGr

32. ibm.co/pixiedust
@MargrietGr

33. Collisions Data
@MargrietGr
Load into Pandas DataFrame

34. Collisions Data
@MargrietGr
Load into Pandas DataFrame

35. Explore Data with PixieDust
@MargrietGr
Load into Pandas DataFrame

36. Explore Data with PixieDust
@MargrietGr

37. @MargrietGr
Explore Weather Data with PixieDust

38. Hypothesis
The number of traffic
collisions is influenced by
the weather
@MargrietGr

39. Combine collisions and weather data
@MargrietGr

40. Combine collisions and weather data
@MargrietGr

41. Analysis with scikit-learn
@MargrietGr

42. Get the data ready for analysis
@MargrietGr

43. @MargrietGr

44. Other models
@MargrietGr
Dimensionality Reduction (PCA)
Stepwise Linear Regression
Decision Trees
Random Forest
Kmeans clustering

45. Collect DataCollect Data
Traffic
Collisions
Historic
Weather
Store Data
Object-Store
Explore Data
New hypothesis
Add more data
Aggregate and
summarize in
different ways
Publish Results
@MargrietGr

46. Collect DataCollect Data
Traffic
Collisions
Historic
Weather
Store Data
Object-Store
Explore Data
New hypothesis
Add more data
Aggregate and
summarize in
different ways
Publish Results
@MargrietGr

47. Publish Results
@MargrietGr
Use Weather forecast Data
Model as a RESTful API - Flask app deployed in
Bluemix
Watson Machine Learning Service

48. Publish Results – PixieApp in a notebook
@MargrietGr
PixieApps
Dashboards within a notebook

49. Why Data Science to the Cloud?
@MargrietGr
Scales up – unlimited resources
All tools connected
Github integration
Local development, easy to move
Collaboration!

50. Thank you!
Dr. Margriet Groenendijk
Developer Advocate
mgroenen@uk.ibm.com
@MargrietGr
Slides
https://www.slideshare.net/MargrietGroenen
dijk/presentations
Blog
https://medium.com/ibm-watson-data-lab
@MargrietGr
IBM Data Science Experience
https://datascience.ibm.com
PixieDust
https://ibm-cds-labs.github.io/pixiedust/
Notebooks
https://github.com/ibm-cds-labs/python-
notebooks
Weather Data
https://business.weather.com/products/weather-
data-packages
IBM Bluemix
https://console.ng.bluemix.net/

51. @MargrietGr