For the full video of this presentation, please visit: https://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/may-2019-embedded-vision-summit-julian For more information about embedded vision, please visit: http://www.embedded-vision.com David Julian, CTO and Founder of Netradyne, presents the "Addressing Corner Cases in Embedded Computer Vision Applications" tutorial at the May 2019 Embedded Vision Summit. Many embedded vision applications require solutions that are robust in the face of very diverse real-world inputs. For example, in automotive applications, vision-based safety systems may encounter unusual configurations of road signs, or unfamiliar temporary barriers around construction sites. In this talk, Julian presents the approach that his company uses to address these “corner cases” in the development of Netradyne’s intelligent driver-safety monitoring system (IDMS). The essence of the company' approach is establishing a virtuous cycle which begins with running analytics at the edge and identifying scenarios of interest and corner cases on the embedded edge device. Data from these cases is then uploaded to the cloud, where it is labeled and then utilized for training new deep learning and analytics models. These new models are then deployed to the embedded device to enable improved performance and begin the cycle anew. Juian also looks at how his company uses this virtuous cycle to develop and deploy new features. Additionally, he shows how his company leverages customer expertise to help identify corner cases at scale.

1. © 2019 Netradyne
Addressing Corner Cases in
Embedded Computer Vision
Applications
David Julian
Netradyne
May 2019

2. © 2019 Netradyne
Problem Statement
• Many embedded CV
applications in the wild
encounter a wide range of
variations and corner
cases
• The systems need to be
robust to a wide range of
variations and have the
ability to improve over
time.
• Case Study using IDMS
2
1. By A.Savin (Wikimedia Commons · WikiPhotoSpace) - Own work, FAL, https://commons.wikimedia.org/w/index.php?curid=58430612
2. By Ian Maddox (Creative Commons Attribution), https://en.wikipedia.org/wiki/File:Tesla_Autopilot_Engaged_in_Model_X.jpg
3. By Jurvetson (Creative Commons Attribution), http://commons.wikimedia.org/wiki/File:Jurvetson_Google_driverless_car.jpg
4. By www.kansascity.com https://commons.wikimedia.org/w/index.php?curid=49391780
1. 2.
3. 4.

3. © 2019 Netradyne
The Intelligent Driver Monitoring System (IDMS)
Challenge:
3
❑ [US] 35K deaths & ~$800B
financial loss due to road
accidents in 2015*
❑ [Globally] est. 1.25M deaths
annually**
❑ 94% of accidents are due to
driver related reasons***
Road Accidents are the leading
cause of loss of life & property
*https://crashstats.nhtsa.dot.gov
** WHO
***NHTSA, “Traffic Safety Facts”, Feb, 2015

4. © 2019 Netradyne
Autonomous Vehicles will take a long time to replace
the large installed base of ‘human driven’ vehicles
4
Need to invest in our drivers now to have meaningful impact on safety.
Can we leverage the same technology to improve driver behavior?
Assuming this
prediction is true..
< 10% of all vehicles on
the road will be fully
autonomous by 2030.

5. © 2019 Netradyne
Driveri™ Vision-Based Driver Safety System
5
Quad HD
Cameras
360 Degree,
120 dB HDR
NVIDIA TX1
Deep Learning Processor
Inertial Sensors
9 Axis Accelerometer,
Gyro, and Magneto
sensors
Communication Channels
4G LTE / Wi-Fi / BT / GPS
Integrated with CAN Bus
(J1939/OBD II)
Storage
Up to 100 Hours of
Video on device
DriveriTM uses Edge Computing to analyze
every second of driving
Mobile Coach App
Fleet Safety Dashboard

6. © 2019 Netradyne
Road Environment Has A Lot Of Variations
6

7. © 2019 Netradyne
Stop - Except Right Turn Signs
7

8. © 2019 Netradyne
Unusual Events – Road/Lane Closures
8

9. © 2019 Netradyne
Geographical Variations
9

10. © 2019 Netradyne
Weather Variations – Rain
10

11. © 2019 Netradyne
Weather Variations – Snow
11

12. © 2019 Netradyne
Unusual Attitudes
12

13. © 2019 Netradyne
Wide Variety of Objects & Signage
13

14. © 2019 Netradyne
Problem Restatement
How can you hope to build and
deploy an embedded CV system
robust to the real-world
variations?
14

15. © 2019 Netradyne
Netradyne Virtuous Cycle
15
Gather Data
Identify Clips of Interest
Label
Train and Test
Algorithms
OTA Update
Push out new
models/features
Leverage data at scale across embedded systems

16. © 2019 Netradyne
Driveri™ – Building a massive collection of Rich,
Vision-Based Driving Data
16
Ride Sharing
Commercial Fleets
Rich, Vision Based Driving Data
Current: 25M+ miles/month
2019: 100M+ miles/month
2021: 1B+ miles/month
Create
Dynamic 3D
HD Maps22M US, 150M globally
Improving
IDMS
Analytics
Netradyne Monthly Footprint

17. © 2019 Netradyne
Valid vs Not For Me Stop Signs
17
Valid:
Not For Me:

18. © 2019 Netradyne
Gathering Data
Boot strapping
• Create initial simple models from hand curated data and deploy in pilot
mode hidden from customer
• Collect data, use to improve, and push out OTA improved model
• Keep cycle going, natural online hard example mining (OHEM)
18
Gather Data
Identify Clips of Interest
Label
Train and Test
Algorithms
OTA Update
Push out new
models/features

19. © 2019 Netradyne
Gather Data
Identify Clips of Interest
Label
Train and Test
Algorithms
OTA Update
Push out new
models/features
Gathering Data
Interesting Event Mining
• Semantic representation of each driving minute sent to cloud
• Data efficient
• Interesting event module in cloud scans for interesting events
• Automatically pull event videos and add to labeling queue
19

20. © 2019 Netradyne
Leverage Expert Customer Feedback
Alert Rejection/Reclassification
Fleet Safety Managers and
drivers can reject alerts and
provide feedback
20
Leverage to improve models and features Gather Data
Identify Clips of Interest
Label
Train and Test
Algorithms
OTA Update
Push out new
models/features

21. © 2019 Netradyne
Leverage Expert Customer Feedback
Event Access – What are customers interested in?
21
Leverage to improve models and features

22. © 2019 Netradyne
Push Out New Models/Features
• OTA Updates are Key
• Feature Progression for Customer Exposure:
• Pilot – Only visible internally
• Alpha – Customer visible, doesn’t impact score
• Regular – Impacts Driver Green-Zone™ Score
22
Gather Data
Identify Clips of Interest
Label
Train and Test
Algorithms
OTA Update
Push out new
models/features

23. © 2019 Netradyne
Applies across features
23
Netradyne Analytics
Driver
Drowsiness
Collision Warning
Lane Departure
Warning System
Talking on Phone

24. © 2019 Netradyne
Summary
• IoT with edge based computing allows access to vast amounts of data
• Virtuous training cycle enables building a robust system to handle corner
cases
• OTA upgradability of DL models and efficient gathering of interesting
data is important to enable this virtuous cycle
• Design in mechanisms to leverage expert customer feedback
24

25. © 2019 Netradyne
Resources
25
Software Frameworks and Toolsets for Deep
Learning-based Vision Processing
https://www.embedded-
vision.com/platinum-members/embedded-
vision-alliance/embedded-vision-
training/documents/pages/deep-learning-
software
The Internet of Things That See:
Opportunities, Techniques and Challenges
https://www.embedded-
vision.com/platinum-members/embedded-
vision-alliance/embedded-vision-
training/documents/pages/embedded-
world-2017
Netradyne
www.Netradyne.com
A Trucking Firm Installs Cameras To Promote
Driver Safety And The Drivers Like It?
Preposterous!
https://www.forbes.com/sites/stevebanker/
2018/10/12/a-trucking-firm-installs-
cameras-to-promote-driver-safety-and-the-
drivers-like-it-preposterous/#4483968147f7