Building a machine learning model is an iterative process. A data scientist will build many tens to hundreds of models before arriving at one that meets some acceptance criteria. However, the current style of model building is ad-hoc and there is no practical way for a data scientist to manage models that are built over time. In addition, there are no means to run complex queries on models and related data. In this talk, we present ModelDB, a novel end-to-end system for managing machine learning (ML) models. Using client libraries, ModelDB automatically tracks and versions ML models in their native environments (e.g. spark.ml, scikit-learn). A common set of abstractions enable ModelDB to capture models and pipelines built across different languages and environments. The structured representation of models and metadata then provides a platform for users to issue complex queries across various modeling artifacts. Our rich web frontend provides a way to query ModelDB at varying levels of granularity. ModelDB has been open-sourced at https://github.com/mitdbg/modeldb.

1. ModelDB: A system
to manage machine
learning models
Manasi Vartak
PhD Student, MIT DB Group

2. People
Manasi Vartak
PhD student, MIT
Srinidhi Viswanathan
MEng, MIT
Samuel Madden
Faculty, MIT
Matei Zaharia
Faculty, Stanford
Harihar Subramanyam
MEng, MIT
Wei-En Lee
MEng student, MIT

3. Building a credit
recommendation algorithm
Profession Credit History Risk of Default
Politician Reasonable 0.3
Struggling
artist
Poor 0.7
Investor
Has more
money than our
company
0.0
… … … …
Barack
Obama
Lindsay
Lohan
Warren
Buffet

4. Model 1

5. Accuracy: 62%
Model 1

6. Accuracy: 58%
Model 2
RandomForestClassifier
val udf1: (Int => Int) = (delayed..)
df.withColumn(“timesDelayed”, udf1)

7. Accuracy: 63%
RandomForestClassifier
df.withColumn(“timesDelayed”, udf1)
.withColumn(“percentPaid”, udf2)
val lrGrid = new ParamGridBuilder()
.addGrid(rf.maxDepth, Array(5, 10, 15))
.addGrid(rf.numTrees, Array(50, 100))
Model 5
credit-default-clean.csv

8. df.withColumn(“timesDelayed”, udf1)
.withColumn(“percentPaid”, udf2)
.withColumn(“creditUsed”, udf3)
…
val lrGrid = new ParamGridBuilder()
.addGrid(lr.elasticNetParam, Array(0.01, 0.1, 0.5, 0.7))
val scaler = new StandardScaler()
.setInputCol(“features”)
…
val labelIndexer1 = new LabelIndexer()
val labelIndexer2 = new LabelIndexer()
…
Model 50
val udf1: (Int => Int) = (delayed..)
val udf2: (String, Int) = …
credit-default-clean.csv

9. Why is this a problem?

10. Why is this a problem?
• No record of model history
Did my colleague do that
already?

11. Why is this a problem?
• No record of model history
• Insights lost along the way
Did my colleague do that
already?
How did normalization
affect my ROC?

12. Why is this a problem?
• No record of model history
• Insights lost along the way
• Difficult to reproduce results
Did my colleague do that
already?
How did normalization
affect my ROC?
What params did I use?

13. Why is this a problem?
• No record of model history
• Insights lost along the way
• Difficult to reproduce results
• Cannot search for or query models
Did my colleague do that
already?
How did normalization
affect my ROC?
Where’s the LR
model I tried last
week with featureX?
What params did I use?

14. Why is this a problem?
• No record of model history
• Insights lost along the way
• Difficult to reproduce results
• Cannot search for or query models
• Difficult to collaborate
Did my colleague do that
already?
How did normalization
affect my ROC?
How does someone review
your model?
Where’s the LR
model I tried last
week with featureX?
What params did I use?

15. Requirements from model
management tool

16. Requirements from model
management tool
• Experiment tracking

17. Requirements from model
management tool
• Experiment tracking
• Versioning

18. Requirements from model
management tool
• Experiment tracking
• Versioning
• Reproducibility

19. Requirements from model
management tool
• Experiment tracking
• Versioning
• Reproducibility
• Comparisons, queries, search

20. Requirements from model
management tool
• Experiment tracking
• Versioning
• Reproducibility
• Comparisons, queries, search
• Collaboration

21. Requirements from model
management tool
• Experiment tracking
• Versioning
• Reproducibility
• Comparisons, queries, search
• Collaboration
*With minimal effort

22. ModelDB: a system to
manage machine
learning models
https://github.com/mitdbg/modeldb
http://modeldb.csail.mit.edu

23. ModelDB: model
management system

24. ModelDB: model
management system
Ingest models,
metadata

25. ModelDB: model
management system
Model, Pipeline
Storage
Versioning
Ingest models,
metadata

26. ModelDB: model
management system
Model, Pipeline
Storage
Versioning
Query
Ingest models,
metadata

27. ModelDB: model
management system
Model, Pipeline
Storage
Versioning
Query
Ingest models,
metadata
Collaboration,
Reproducibility

28. User quotes
“I should have had this in my self-driving cars class; it
would have made things so much easier”
“…it can really help with reproducibility … and
collaboration in multi-person teams…”
“I used it to track models for a research project; it
was so simple”

29. Demo

30. ModelDB Architecture &
Design Decisions

31. ModelDB Architecture &
Design Decisions

32. ModelDB Architecture &
Design Decisions
1. Support for diverse
languages and environments

33. ModelDB Architecture &
Design Decisions
1. Support for diverse
languages and environments
2. Minimal changes to
existing workflows

34. ModelDB Architecture &
Design Decisions
1. Support for diverse
languages and environments
2. Minimal changes to
existing workflows
3. Rich visual interface

35. ModelDB Architecture &
Design Decisions
1. Support for diverse
languages and environments
2. Minimal changes to
existing workflows
3. Rich visual interface
4. Support for complex
queries

36. “Oh, but why not git?”
• All code treated equal
• Some elements are special: data sources,
parameters, metrics, models
• Difficult to tease that out
• No semantics, so can’t run interesting queries

37. Requirements from model
management tool

38. Requirements from model
management tool
• Experiment tracking

39. Requirements from model
management tool
• Experiment tracking
• Versioning

40. Requirements from model
management tool
• Experiment tracking
• Versioning
• Reproducibility

41. Requirements from model
management tool
• Experiment tracking
• Versioning
• Reproducibility
• Comparisons, queries, search

42. Requirements from model
management tool
• Experiment tracking
• Versioning
• Reproducibility
• Comparisons, queries, search
• Collaboration

43. Requirements from model
management tool
• Experiment tracking
• Versioning
• Reproducibility
• Comparisons, queries, search
• Collaboration
*With minimal effort

44. ModelDB Features
(currently available)
• Experiment tracking
• Versioning
• Reproducibility
• Comparisons, queries, search
• Collaboration

45. ModelDB Features
(currently available)
• Experiment tracking
• Versioning
• Reproducibility
• Comparisons, queries, search
• Collaboration
Log models, params, pipelines
etc. via ModelDB API

46. ModelDB Features
(currently available)
• Experiment tracking
• Versioning
• Reproducibility
• Comparisons, queries, search
• Collaboration
Log models, params, pipelines
etc. via ModelDB API
Every modeling run = version

47. ModelDB Features
(currently available)
• Experiment tracking
• Versioning
• Reproducibility
• Comparisons, queries, search
• Collaboration
Log models, params, pipelines
etc. via ModelDB API
All pipeline details, params
logged
Every modeling run = version

48. ModelDB Features
(currently available)
• Experiment tracking
• Versioning
• Reproducibility
• Comparisons, queries, search
• Collaboration
Log models, params, pipelines
etc. via ModelDB API
Model search, query,
comparison via frontend
All pipeline details, params
logged
Every modeling run = version

49. ModelDB Features
(currently available)
• Experiment tracking
• Versioning
• Reproducibility
• Comparisons, queries, search
• Collaboration
Log models, params, pipelines
etc. via ModelDB API
Model search, query,
comparison via frontend
Central repository of models
Review models, annotate
All pipeline details, params
logged
Every modeling run = version

50. ModelDB Features
(ongoing)

51. ModelDB Features
(ongoing)
• Unified Querying of Modeling Artifacts

52. ModelDB Features
(ongoing)
• Unified Querying of Modeling Artifacts
Base data, intermediates,
models, predictions, metadata

53. ModelDB Features
(ongoing)
• Unified Querying of Modeling Artifacts
Base data, intermediates,
models, predictions, metadata
“How did the GBDTs do on married customers who
are interested in gardening?”

54. ModelDB Features
(ongoing)
• Unified Querying of Modeling Artifacts
Base data, intermediates,
models, predictions, metadata
“How did the GBDTs do on married customers who
are interested in gardening?”
Base
Data
is_married=T

55. ModelDB Features
(ongoing)
• Unified Querying of Modeling Artifacts
Base data, intermediates,
models, predictions, metadata
“How did the GBDTs do on married customers who
are interested in gardening?”
Base
Data
is_married=T
Intermediates
gardening=T

56. ModelDB Features
(ongoing)
• Unified Querying of Modeling Artifacts
Base data, intermediates,
models, predictions, metadata
“How did the GBDTs do on married customers who
are interested in gardening?”
Base
Data
is_married=T
Intermediates
gardening=T
Metadata
type=
GBDT
Models
ids={..}

57. ModelDB Features
(ongoing)
• Unified Querying of Modeling Artifacts
Base data, intermediates,
models, predictions, metadata
“How did the GBDTs do on married customers who
are interested in gardening?”
Base
Data
is_married=T
Intermediates
gardening=T
Predictions
accuracy(…)
Metadata
type=
GBDT
Models
ids={..}

58. ModelDB Features
(ongoing)
• Unified Querying of Modeling Artifacts
Base data, intermediates,
models, predictions, metadata
“How did the GBDTs do on married customers who
are interested in gardening?”
Base
Data
is_married=T
Intermediates
gardening=T
Predictions
accuracy(…)
Metadata
type=
GBDT
Models
ids={..}
What query language?
How to persist data?

59. ModelDB Features
(ongoing)

60. ModelDB Features
(ongoing)
• Mining data in ModelDB

61. ModelDB Features
(ongoing)
• Mining data in ModelDB
Model Features Params Metric
M13 X3,X9... l1=0.3 0.63
M22 X1,X4,X7 l2=0.7 0.8
M34 X11,X13 l1=0.7 0.55
… … … …

62. ModelDB Features
(ongoing)
• Mining data in ModelDB
Given model history, what
should we try next?
Bayesian Modeling/AutoML
Model Features Params Metric
M13 X3,X9... l1=0.3 0.63
M22 X1,X4,X7 l2=0.7 0.8
M34 X11,X13 l1=0.7 0.55
… … … …

63. ModelDB Features
(ongoing)
• Mining data in ModelDB
• Full model lifecycle management
Given model history, what
should we try next?
Bayesian Modeling/AutoML
Model Features Params Metric
M13 X3,X9... l1=0.3 0.63
M22 X1,X4,X7 l2=0.7 0.8
M34 X11,X13 l1=0.7 0.55
… … … …

64. ModelDB Features
(ongoing)
• Mining data in ModelDB
• Full model lifecycle management
Given model history, what
should we try next?
Bayesian Modeling/AutoML
Model Features Params Metric
M13 X3,X9... l1=0.3 0.63
M22 X1,X4,X7 l2=0.7 0.8
M34 X11,X13 l1=0.7 0.55
… … … …

65. ModelDB Features
(ongoing)
• Mining data in ModelDB
• Full model lifecycle management
Given model history, what
should we try next?
Bayesian Modeling/AutoML
Model performance degrades
Retrain model over time
Model Features Params Metric
M13 X3,X9... l1=0.3 0.63
M22 X1,X4,X7 l2=0.7 0.8
M34 X11,X13 l1=0.7 0.55
… … … …

66. ModelDB available now!
https://github.com/mitdbg/modeldb
*MIT License

67. ModelDB available now!

68. ModelDB available now!
• Download, try it out!

69. ModelDB available now!
• Download, try it out!

70. ModelDB available now!
• Download, try it out!
• Tell us what you think; what can we do better?

71. ModelDB available now!
• Download, try it out!
• Tell us what you think; what can we do better?
• Contribute! (see Issues on repo for some ideas)

72. ModelDB: a system to
manage machine
learning models
mvartak@csail.mit.edu | @DataCereal
http://modeldb.csail.mit.edu
*Icons from FlatIcon