Excitement about AI and Machine Learning is tremendous. But how many mainstream products with AI functions do you know? This talk discusses the importance to have a technical strategy for AI projects to avoid pitfalls.

1. 3.1 How to approach Machine Learning for innovation projects? 1
CONFIDENTIAL Template Innovation Day 2018CONFIDENTIAL
MACHINE LEARNING:
PROJECT APPROACH
Jochem Grietens
System Engineer – A.I. passionate
Jochem.Grietens@verhaert.com
TRACK 3 - ROBOTICS FOR A BETTER WORLD

2. 3.1 How to approach Machine Learning for innovation projects? 2
CONFIDENTIAL
Introduction

3. 3.1 How to approach Machine Learning for innovation projects? 3
CONFIDENTIAL
Goal

4. 3.1 How to approach Machine Learning for innovation projects? 4
CONFIDENTIAL
AI and ML ?

5. 3.1 How to approach Machine Learning for innovation projects? 5
CONFIDENTIAL
AI and ML ?

6. 3.1 How to approach Machine Learning for innovation projects? 6
CONFIDENTIAL
AI and ML ?
ML
• Unstructured data  structured data
• Learn from data
• Pattern recognition/detection
• …
AI
• Reasoning systems
• Planning, searching, …
• Expert systems
• Understanding ?
• …

7. 3.1 How to approach Machine Learning for innovation projects? 7
CONFIDENTIAL
Biggest drivers
1. Data availability
2. Computational performance.
Why now

8. 3.1 How to approach Machine Learning for innovation projects? 8
CONFIDENTIAL
Framing the A.I. roadmap

9. 3.1 How to approach Machine Learning for innovation projects? 9
CONFIDENTIAL
“Let’s collect as much data as possible and apply A.I. later”
“We have a bunch of data laying around … “
“A.I. as a solution to everything…”
Framing the A.I. roadmap

10. 3.1 How to approach Machine Learning for innovation projects? 10
CONFIDENTIAL
“Goal oriented”
Vs
“Explorative”
Framing the A.I. roadmap
dzdz

11. 3.1 How to approach Machine Learning for innovation projects? 11
CONFIDENTIAL
ML, where do we begin ?

12. 3.1 How to approach Machine Learning for innovation projects? 12
CONFIDENTIAL
Machine learning process
TRAINING THE MODEL
USING THE MODEL

13. 3.1 How to approach Machine Learning for innovation projects? 13
CONFIDENTIAL
Unfamiliar technology
 Risk project failure
 Having the right team with the right skills
 Careful execution
Machine learning – The new kid on the block

14. 3.1 How to approach Machine Learning for innovation projects? 14
CONFIDENTIAL
Development cycle

15. 3.1 How to approach Machine Learning for innovation projects? 15
CONFIDENTIAL
Machine learning project steps
Verhaert project flow:
based on CRISP-DM

16. 3.1 How to approach Machine Learning for innovation projects? 16
CONFIDENTIAL
1. Business understanding

17. 3.1 How to approach Machine Learning for innovation projects? 17
CONFIDENTIAL
• “Tech driven” vs “business driven”
• Analyse the opportunity first before investing resources
and infrastructure.
Business understanding

18. 3.1 How to approach Machine Learning for innovation projects? 18
CONFIDENTIAL
Rules of ML

19. 3.1 How to approach Machine Learning for innovation projects? 19
CONFIDENTIAL
Concept on product level & Model level
Algorithms don’t live in a box

20. 3.1 How to approach Machine Learning for innovation projects? 20
CONFIDENTIAL
“Sometimes A.I. isn’t up for the job”
• Problem too complex
• Never seen something like this before
Human in the loop (HITL)
HITL + Active (incremental) Learning
A.I. failure and human interaction

21. 3.1 How to approach Machine Learning for innovation projects? 21
CONFIDENTIAL
Human interaction

22. 3.1 How to approach Machine Learning for innovation projects? 22
CONFIDENTIAL
Explainable AI (XAI )
• Decision trees
• Bayesian reasoning systems
• Expert systems
Reverse engineering
• Internal state reflection
• Parameter variation
Divide and conquer
Lack of transparency

23. 3.1 How to approach Machine Learning for innovation projects? 23
CONFIDENTIAL
2. Concept generation

24. 3.1 How to approach Machine Learning for innovation projects? 24
CONFIDENTIAL
Before understanding  explore the problem
Automated car:
Traffic sign detection, Free Road detection, Traffic lights, object recognition,
Traffic rules, Distance, ….
Input data:
Visual, Lidar, HD maps, traffic rules, gps, …
Output data: steering, breaking and driving, …
 Understanding of the problem
Explore the problem

25. 3.1 How to approach Machine Learning for innovation projects? 25
CONFIDENTIAL
Concept generation

26. 3.1 How to approach Machine Learning for innovation projects? 26
CONFIDENTIAL
3. Data understanding

27. 3.1 How to approach Machine Learning for innovation projects? 27
CONFIDENTIAL
Before understanding  explore the problem
Automated car:
Traffic sign detection, Free Road detection, Traffic lights, object recognition,
Traffic rules, Distance, ….
Traffic sign detection:
Lighting, surroundings, color, angle, image sharpness , ….
 Understanding of the problem AND data
Explore the problem

28. 3.1 How to approach Machine Learning for innovation projects? 28
CONFIDENTIAL
Creating data = expensive
Gathering low cost data & learn
• Create toy datasets
• Use existing (online) databases
• Augment limited databases
Learn as much as you can this way.
• What's missing in the database ?
• Which are the hard cases ?
…
Data Gathering – First Runs

29. 3.1 How to approach Machine Learning for innovation projects? 29
CONFIDENTIAL
So you need more data…
Questions:
• Is the needed information present in the data provided ?
• What data input could provide this information
 Recruit expert knowledge.
 This requires engineering/system/physics insight.
Data gathering – AI engineering

30. 3.1 How to approach Machine Learning for innovation projects? 30
CONFIDENTIAL
“Learning from a dataset that is skewed will predict skewed”
Fixes:
• Over sampling
• Under sampling
• Create new data
• Weighting techniques
• …
Training data balance
0
1
2
3
4
5
Traffic agent identification

31. 3.1 How to approach Machine Learning for innovation projects? 31
CONFIDENTIAL
Data Veracity = Accuracy
Data fidelity
• Quality in context
• Unintended Bias
Data understanding
Quality in the context it’s being used Unintended Bias

32. 3.1 How to approach Machine Learning for innovation projects? 32
CONFIDENTIAL
Learning by heart vs. learning general features
Algorithm should have learned general features :
• Recognizing 4 wheels vs 2 wheels
• Shape of person
• size vs environment
Instead it learned:
• Red/blue = car
• Tree in the background
Explaining overtraining

33. 3.1 How to approach Machine Learning for innovation projects? 33
CONFIDENTIAL
4. Data Preparation

34. 3.1 How to approach Machine Learning for innovation projects? 34
CONFIDENTIAL
Feature extraction vs raw data usage

35. 3.1 How to approach Machine Learning for innovation projects? 35
CONFIDENTIAL
Labeling is expensive!
Crowd sourcing: HUMAN INTELLIGENCE
Data labeling

36. 3.1 How to approach Machine Learning for innovation projects? 36
CONFIDENTIAL
4 C = Clean, Correct, Consistent, Complete
1. Data formatting
2. Data cleaning (usually 90% of the job),
3. Data anonymization (i.e. when working with healthcare and banking data)
4. Data augmentation
5. Feature engineering
6. …
Data preparation

37. 3.1 How to approach Machine Learning for innovation projects? 37
CONFIDENTIAL
Data augmentation

38. 3.1 How to approach Machine Learning for innovation projects? 38
CONFIDENTIAL
Artificial data synthesis
“Pick up box A ”
Factory noise

39. 3.1 How to approach Machine Learning for innovation projects? 39
CONFIDENTIAL
5. Modeling

40. 3.1 How to approach Machine Learning for innovation projects? 40
CONFIDENTIAL
1. Determine ML algorithm performance metrics
2. Create a Data pipeline
3. Build a simple model first
4. Compare models
Build a simple model

41. 3.1 How to approach Machine Learning for innovation projects? 41
CONFIDENTIAL
Off the shelf A.I. software
• From open source to pay-per-use
• Vendor support
• Clean and supply data
• Capabilities match
• Beware: Ideas with a website
AI PLATFORMS
• IBM, Amazon, Google
• off the shelf  AI building tools for researchers.
Custom build
• Only when needed
• Great flexibility
BUILD vs. BUY vs. PLATFORM

42. 3.1 How to approach Machine Learning for innovation projects? 42
CONFIDENTIAL
End-to-end vs Ensembling

43. 3.1 How to approach Machine Learning for innovation projects? 43
CONFIDENTIAL
End-To-End vs Ensemble learning
Acc.
Break
Steering

44. 3.1 How to approach Machine Learning for innovation projects? 44
CONFIDENTIAL
Autonomous vehicles
Subdivide the problem in to many subcomponents
End-To-End vs Ensemble learning
Acc
Break
Steering

45. 3.1 How to approach Machine Learning for innovation projects? 45
CONFIDENTIAL
Perception
Acc
Break
Steering

46. 3.1 How to approach Machine Learning for innovation projects? 46
CONFIDENTIAL
Localization
Acc.
Break
Steering

47. 3.1 How to approach Machine Learning for innovation projects? 47
CONFIDENTIAL
Planning
Acc.
Break
Steering

48. 3.1 How to approach Machine Learning for innovation projects? 48
CONFIDENTIAL
End-to-end
- Let the machine figure the
problem out…
- More data needed
- Can improve performance
End-to-End vs Ensemble methods
Person
Ensemble methods
- Systems can be trained separately
- Less data needed
- Algorithm insight
- Levels of abstraction allow freedom:

49. 3.1 How to approach Machine Learning for innovation projects? 49
CONFIDENTIAL
5. Evaluation

50. 3.1 How to approach Machine Learning for innovation projects? 50
CONFIDENTIAL
Defining evaluation metrics

51. 3.1 How to approach Machine Learning for innovation projects? 51
CONFIDENTIAL
Define success metrics @ 2 levels
Business Level
i.e. Decrease # interventions of driver in assisted driving.
Model (ML) Level
Comparing models
i.e. Accuracy object detection, Accuracy lane detection
Guard relation between the two !
Success metrics

52. 3.1 How to approach Machine Learning for innovation projects? 52
CONFIDENTIAL
Find a metric that represents your goal & easy to compare
Multiple metrics
Single metric: Accuracy = 50 %
Single metric combo
Model level success metric

53. 3.1 How to approach Machine Learning for innovation projects? 53
CONFIDENTIAL
When ML automates things humans do well.
1. Humans are good labelers
2. Error analysis can draw on human intuition
3. Use human-level performance is often a good first goal.
 Automating things humans don’t do well don’t have these advantages
to draw from – Creativity needed !
Comparing to human level performance
“This recipe calls for a pear of apples”
mistaking “pair” for “pear.”

54. 3.1 How to approach Machine Learning for innovation projects? 54
CONFIDENTIAL
Acceptance level of a solution ?
• Rational threshold ?
• Consumer acceptance threshold ?
• Legal context ?
Guard relation between the two !
Business Level
Model
success
Business
success

55. 3.1 How to approach Machine Learning for innovation projects? 55
CONFIDENTIAL
Recommendations

56. 3.1 How to approach Machine Learning for innovation projects? 56
CONFIDENTIAL
One group, five brands
Our services are marketed through 5 brands each
addressing specific missions in product development.
INTEGRATED PRODUCT DEVELOPMENT
ON-SITE
PRODUCT
DEVELOPMENT
DIGITAL
PRODUCTS
DEVELOPMENT
OPTICAL
PRODUCTS
DEVELOPMENT
VENTURING