IEEE Software 2020

1. An Exploratory Study on the
Machine Learning Model Stores
Zhen Ming (Jack) Jiang

2. Machine Learning (ML) &
Software Engineering (SE)
Software
Engineering
Machine
Learning

3. Machine Learning (ML) &
Software Engineering (SE)
Software
Engineering
Machine
Learning
..
Bug Prediction
Test Case Prioritization
Automated Bug Triage
.

4. Machine Learning (ML) &
Software Engineering (SE)
Software
Engineering
Machine
Learning
..
Bug Prediction
Test Case Prioritization
Automated Bug Triage
.
Test ML Applications

5. Machine Learning (ML) &
Software Engineering (SE)
Software
Engineering
Machine
Learning
..
Bug Prediction
Test Case Prioritization
Automated Bug Triage
.
Test ML Applications
Anything else???

10. ML Model Stores

11. ML Model Stores
January 2018
June 2018
November 2018
November 2018
April 2019 (beta)

12. ML Model Stores
January 2018
June 2018
November 2018
November 2018
April 2019 (beta)
What are the unique SE
practices/challenges for
building ML applications?

13. Structure
Contents

14. Structure

15. What kind of information elements
do ML model stores provide?
January 2018
June 2018
November 2018
November 2018
April 2019 (beta)

16. What kind of information elements
do ML model stores provide?
January 2018
June 2018
November 2018
April 2019 (beta)

17. What kind of information elements
do ML model stores provide?
January 2018
June 2018
November 2018

18. What kind of information elements
do ML model stores provide?
January 2018
June 2018
November 2018
October 2008
July 2008

19. Identifying Information Elements

20. Identifying Information Elements
Owner

21. Identifying Information Elements
ML Algorithm
Owner

22. Identifying Information Elements
Framework
ML Algorithm
Owner

23. Identifying Information Elements
Framework Training set
ML Algorithm
Owner

24. Identifying Information Elements
LicenseFramework Training set
ML Algorithm
Size Usage Statistics
User feedback
User Manual
Owner

25. Identifying Information Elements
LicenseFramework Training set
ML Algorithm
Size Usage Statistics
User feedback
User Manual
Owner

26. Identifying Information Elements
LicenseFramework Training set
ML Algorithm
Size Usage Statistics
User feedback
User Manual
Owner

27. vs.ML
Engineers
End
Users

29. To package an ML application for release
- Source code
-Trained ML model(s)
-Training set info
Integrating such a model …
- API calls, re-train on new data?

30. To package an ML application for release
- Source code
-Trained ML model(s)
-Training set info
Integrating such a model …
- API calls, re-train on new data?

32. Quality representation
- Statistics vs. QoS measures (reliability)
- Requirements for different ML applications

34. [Pricing] Subscription-based
- Pricing scheme vs. actual usage context
Capacity planning for ML applications

35. Structure
Contents

36. How unique are the ML models
provided by each model store?
Very few cross-store ML applications
- Vendor lock-in
- Need for cross-platform frameworks

37. Same ML Models Different Usage Context

38. Same ML Models Different Usage Context

39. Minor Releases Different Product Pages

40. Minor Releases Different Product Pages

41. Minor Releases Different Product Pages
A new version for an ML application is:
- same ML model re-trained on another dataset? or
- same dataset with different ML implementations?