One of the main missing concepts in a data-driven business is a stable platform for production models. Model factory provides a framework where production models can be easily maintained and monitored. Because the platform was build with a data scientist in mind, it removes the hurdles which commonly occur while moving model from development to production. The platform is build using open source software (e.g., GIT, Jenkins, R) . We explain how to combine these technologies to build your own model factory. Talk given during the meetup: https://www.meetup.com/The-Amsterdam-Applied-Machine-Learning-Meetup-Group/events/234463593/

1. Model Factory
October 2016
Advanced Analytics
Maria Vechtomova
Chris Molanus

2. About KPN

3. About KPN
2014 20162015 2017 2018 2019 2020
BM
CM
€ bn
Total TelCo market
1. Revenu declining
2. NPS increasing
3. Cost reduction
4. Simplification
Trend traditional TelCo market: continuous decline

4. About KPN
Data & Analytics
Analytics
Data &
Information
Advanced
Analytics
Customer
interaction
CEO
CCO COO CFO

5. Model development & production before
Data discovery
Test model
Production model
Scheduling
4 months

6. Main components of the model factory before
Orchestrator Compute engine Storage

7. Main components of the model factory now
Orchestrator Compute engine StorageVersion control

8. Model development & production now
Data discovery
Test model
Production model
Scheduling
Data discovery
Test model
Production model
Scheduling
4 months
1,5 months
Before
Now

9. Development process
Fork base model Fill in template
modelFactory

10. Development process
Create connection to database/ compute engine using
credentials stored on user’s machine

11. Development process
Generate a session ID and add it to run_history + fill in the
start time; end time stays empty till the end of the session

12. Development process
Load data from the database

13. Development process
Create summary statistics and store it in the database

14. Development process

15. Development process
Create variable importance information and store
it in the database

16. Development process

17. Development process
Create model performance statistics and store it in the
database

18. Development process
Store model scores in the database

19. Development process
Update end_time of the session in run_history table

20. Development & production
Fork base model Fill in template
modelFactory
Test model in
development
Test model in
production

21. Reporting and monitoring

22. Reporting and monitoring

23. Reporting and monitoring

24. Reporting and monitoring

25. Workflow
Analysts

26. Workflow
Development
Master
Production
V1

27. Workflow
Feature 1
Feature 3
Development
Master
Production
V1

28. Workflow
Feature 1
Feature 3
Development
Master
Production
V1 V2

29. Workflow
Feature 1
Feature 3
Development
Master
Production
Feature 2
V1 V2

30. Workflow
Feature 1
Feature 3
Development
Master
Production
Hot Fix
Feature 2
V1 V2 V2.1
V2.1

31. Workflow
Feature 1
Feature 3
Development
Master
Production
Hot Fix
Feature 2
V1 V2 V2.1
V2.1

32. Workflow
Feature 1
Feature 3
elopment
Hot Fix
Feature 2
V1 V2 V2.1
V2.1

33. Workflow
Feature 1
Feature 3
Hot Fix
Feature 2
V2 V2.1 V3
V2.1

34. Workflow
Feature 1
Feature 3
Hot Fix
Feature 2
Feature x
V2 V2.1 V3
V2.1

35. Big picture
Decrease deployment time
Unified model metrics
Unified way of working
More modeling flexibility
Faster response time
Less FTE per model
Better performing models
Model Factory attributes Business value

36. Big picture
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
+13%
After SMOTEBefore SMOTERandom
Churn model example: estimated saving of €120k per year

37. Thank you!
Questions/ Comments:
Email: mariia.vechtomova@kpn.com
chris.molanus@kpn.com