Machine Learning at Scale with MLflow and Apache Spark
•
0 gefällt mir•914 views
This document summarizes the challenges faced by SocGen, a large French bank, in implementing machine learning at scale using Spark and MLflow. Some key challenges included: 1) Keeping data and models local for regulatory reasons while performing training and prediction, 2) Ensuring reliability when moving models between prototyping and production phases, 3) Managing different Python package dependencies, 4) Tracking and managing many models, and 5) Ensuring high availability of the tracking server. The presentation provided a concrete example of using Spark, MLflow, and Kafka to periodically retrain a model for scoring news articles and handling user feedback in a scalable and reliable way.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Machine Learning at Scale with MLflow and Apache Spark
Ähnlich wie Machine Learning at Scale with MLflow and Apache Spark (20)
Mehr von Databricks
Mehr von Databricks (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Machine Learning at Scale with MLflow and Apache Spark
- 1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
- 2. Chongguang LIU, Société Générale Machine learning at scale with MLflow and Spark #UnifiedDataAnalytics #SparkAISummit
- 3. #UnifiedDataAnalytics #SparkAISummit About me 3 • Studied computer science and engineering • Data engineer at SocGen • Using Spark and MLflow at work • Skiing and diving during vacations
- 4. Data is strategic at SocGen • SocGen is French multinational bank. • We have 80+ data pipelines in production in our data lake. • More than 200 data scientists working across the globe. • Data allows us to create new products, improve customer experience and be more efficient. • Relevant use cases such as anti-money laundering, fraud detection, automatic document analysis etc. 4
- 5. But also a lot of pain points ... 5 Business Data scientist Data engineer Manually copy training data Code rewrite in another programming language No automated data flow Manually deploy models Difficult to use ML models Suboptimal predictions Models rarely updated Limited value for business !
- 6. Finally we realised that ... 6 ML Code Hidden technical debt in machine learning systems, 2015, Google
- 7. Challenge 1: data locality • A central Hadoop cluster • Client data, transaction data, accounting data etc. • Automated data pipelines • Banking industry is highly regulated, sensitive data is kept in the data lake for security reasons. 7 training and prediction inside the data lake
- 8. Challenge 2: application reliability 8 prototyping phase
- 9. Challenge 2: application reliability 9 Yarn Data node Data node Data node production phase
- 10. Challenge 3: variant python packages 10 python code python code + conda env python code + conda env
- 11. Challenge 4: model management 11
- 12. 12 MLflow tracking server HDFS Data nodeData nodeData node ML modelsML meta data Challenge 4: model management
- 13. Challenge 5: tracking server reliability 13 Tracking server Tracking server HA proxy PostgreSQLData node
- 14. Challenge 6: model serving 14 MLflow HDFS Knox API server Kafka Spark streaming
- 15. A concrete example 15 Web app DB ML model server news HDFS feedback feedback Kafka score MLflow Spark + Sklearn feedback new model periodic model retraining score news score Spark streaming model score
- 16. Moving forward • Model drift monitoring • A/B testing • pandas_udf • koalas 16
- 17. Thank you! 17
- 18. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT