Overton, Apple Flavored ML
•
1 gefällt mir•313 views
In early September, Apple released a paper describing Overton, the framework they built to create, monitor, and improve production-based ML systems. After presenting the main lines that define this framework, we will take a closer look at the heart of Overton: slice-based learning.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Ähnlich wie Overton, Apple Flavored ML
Ähnlich wie Overton, Apple Flavored ML (20)
Mehr von source{d}
Mehr von source{d} (15)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Overton, Apple Flavored ML
- 1. OvertonOverton Apple- avored MLApple- avored ML 2019/10/07 Nantes Machine Learning Meetup2019/10/07 Nantes Machine Learning Meetup
- 3. OverviewOverview What / When / WhereWhat / When / Where Publishing date Early September Conference NeurIPS Goal ML software lifecycle management Maturity Production
- 4. OverviewOverview Challenges to overcomeChallenges to overcome Precise monitoring Complex pipelines Ef cient feedback loop
- 5. OverviewOverview Architectural choicesArchitectural choices Code-less deep learning Multi-task learning Weak supervision
- 7. Code-less deep learningCode-less deep learning PrinciplesPrinciples Models & training code = experts “black box” by engineers
- 8. Code-less deep learningCode-less deep learning ModularityModularity
- 9. Code-less deep learningCode-less deep learning Con gurationCon guration
- 10. Fine grained MLFine grained ML & multi-tasks& multi-tasks
- 11. Fine grained ML & multi-tasksFine grained ML & multi-tasks ProblemsProblems Lots of subtasks (implicit & explicit) Need to evaluate & monitor them Need to improve on them
- 12. Fine grained ML & multi-tasksFine grained ML & multi-tasks ApproachApproach Slice-based learning De nition of data subsets Augmentation of model capacity Dedicated metrics
- 13. Fine grained ML & multi-tasksFine grained ML & multi-tasks Slice-based learningSlice-based learning
- 14. Fine grained ML & multi-tasksFine grained ML & multi-tasks De nition of critical dataDe nition of critical data subsetssubsets With “slice functions”: def sf_bike(x): return "bike" in object_detector(x) def sf_night(x): return avg(X.pixels.intensity) < 0.3
- 15. Fine grained ML & multi-tasksFine grained ML & multi-tasks Slice expertsSlice experts For each slice, an expert: should add capacity to the model has to know when to trigger has to have dedicated metrics
- 16. Fine grained ML & multi-tasksFine grained ML & multi-tasks Hard partsHard parts Noise : slices are de ned with heuristics Scale 1 : when the number of slices goes up, does the model still run fast enough? Scale 2 : when the number of slices goes up, is the model still good enough?
- 17. Fine grained ML & multi-tasksFine grained ML & multi-tasks Proposed solutionProposed solution
- 18. ResourcesResources Snorkel blog “Slice-based Learning” NeurIPS paper “Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices”
- 20. Weak supervisionWeak supervision Problem statementProblem statement We need data, but: It's expensive, not available, yadda yadda Especially for interesting corner cases It's noisy
- 21. Weak supervisionWeak supervision SolutionSolution Use heuristics @labeling_function() def lf_regex_check_out(x): """Spam comments say 'check out my video', 'check it out', et return SPAM if re.search(r"check.*out", x.text, flags=re.I) e
- 22. Weak supervisionWeak supervision SolutionSolution Then correct the loss accounting for their correlation
- 23. Weak supervisionWeak supervision ImplementationImplementation Overton uses a modi ed version of the “Label Model” from Snorkel.
- 24. ResourcesResources Snorkel blog « Introducing the New Snorkel » ICML paper “Learning Dependency Structures for Weak Supervision Models”
- 25. ConclusionConclusion Modularity (alike AllenNLP, tensor2tensor) Dedicated metrics & multi-tasks Improvements on critical data subsets Arti cal data with theoretical corrections