MongoDB World 2019: MongoDB in Data Science: How to Build a Scalable Product Using MongoDB
•
1 like•253 views
You have made a successful Proof of Concept by using Pandas for data manipulation and analysis. So, how are you going to productionize it? Come to learn how to transform your POC to a scalable product with MongoDB. Learn about pitfalls and drawbacks of Pandas and benefits of using MongoDB in the early stages.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to MongoDB World 2019: MongoDB in Data Science: How to Build a Scalable Product Using MongoDB
Similar to MongoDB World 2019: MongoDB in Data Science: How to Build a Scalable Product Using MongoDB (20)
More from MongoDB
More from MongoDB (20)
Recently uploaded
Recently uploaded (20)
MongoDB World 2019: MongoDB in Data Science: How to Build a Scalable Product Using MongoDB
- 1. MongoDB in Data Science How to convert a Pandas Proof-of-Concept to a scalable product and why MongoDB is the key to success !
- 2. Who I am Software Engineer Compiler Engineer Compiler Engineer LLVM contributor Software Engineer R/D Lead ML Engineer Backend Infrastructure Sr. ML Engineer
- 3. What will we learn ? ● Understand existing tools for delivering Data Science projects and when to use them. ● Why MongoDB could be crucial for your product and business ● How to easily productionize a Pandas Proof-of-Concept ● How to use MongoDB while being open to other technologies.
- 4. Motivation
- 5. Speed of inference Speed of development Key factors
- 7. Research Data Scientist Productionization Data/ML Engineer Speed of development Key factors
- 8. What is Pandas? Most popular Python framework for data manipulation and data wrangling in Data Science community.
- 9. What is Pandas? Most popular Python framework for data manipulation and data wrangling in Data Science community. Source: numpy.org, scipy.org, matplotlib.org, scikit-learn.org, pandas.pydata.org
- 10. Source: Stackoverflow post by David Robinson
- 11. Why use Pandas Dataframes ?
- 12. Why use Pandas Dataframes ?
- 13. Why use Pandas Dataframes ?
- 14. Why use Pandas Dataframes ?
- 15. Why use Pandas Dataframes ?
- 16. Drawbacks of Pandas ● Doesn’t have persistence layer ● Doesn’t support primary and secondary indexes ○ As a result, not efficient for querying ● Doesn’t support multi-threading
- 18. Productionization options Real time service Batch Job Slow Inference
- 19. Productionization options Real time service Batch Job Slow Inference Fast Inference
- 20. Real time service demo (recommendation) Event Store
- 21. Real time service demo (recommendation) Event Store Model Training Job
- 22. Real time service demo (recommendation) Event Store Model Training Job Model store
- 23. Real time service demo (recommendation) Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 24. Real time service demo (recommendation) Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 25. Real time service demo (recommendation) Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 26. Real time service demo (recommendation) Event Store Feature Aggregation Model Inference Inference Service request respond
- 27. Real time service demo (recommendation)
- 28. Real time service demo (recommendation)
- 29. Real time service demo (recommendation)
- 30. Things to avoid ● Don’t forget to put indexes on your collection ● Don’t put indexes on every field ● Don’t read and write from the same replica
- 31. But… we generate a tons of user events! Is this solution going to work for us?
- 32. user events Consumer 1 Consumer 2 Consumer N MongoDB Postgres DFS Typical data pipeline
- 33. user events Consumer 1 Consumer 2 Consumer N MongoDB Postgres DFS Typical data pipeline
- 34. MongoDB TTL index Filters event_type ... Consumer Shrink down the amount of data
- 35. Real time service demo (recommendation) Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 36. Training Job Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 39. MongoDB Connector Event Store Inference Job Inference as a batch job
- 41. Batch Job versus Real Time Service Real Time Service Batch Job Pros On demand (scales as needed) Easier to develop and maintain Cons Harder to develop and maintain Constantly utilizing resources
- 42. Benefits of MongoDB ● Schema-Less ● Horizontally scalable ● Available as PaaS from many vendors. ● Has a huge community ● Easier to hire people
- 43. Summary ● Allows to provide a real time experience ● Could help save expensive computational resources ● Provides a way to do real time as well as batch inference
- 44. We are hiring !!! careers.shopbonsai.ca
- 45. References ● https://stackoverflow.blog/2017/09/14/python-growing-quickly/ ● https://www.mongodb.com/products/spark-connector ● https://pandas.pydata.org/ ● https://scikit-learn.org/ ● https://matplotlib.org/ ● https://www.scipy.org/ ● https://www.numpy.org/ ● https://iconscout.com/icon/device-management-mobile-computer-seo-tool-analyze-7
- 46. Thanks !!!