For the full video of this presentation, please visit: https://www.embedded-vision.com/platinum-members/videantis/embedded-vision-training/videos/pages/may-2017-embedded-vision-summit For more information about embedded vision, please visit: http://www.embedded-vision.com Marco Jacobs, VP of Marketing at videantis, presents the "Computer-vision-based 360-degree Video Systems: Architectures, Algorithms and Trade-offs" tutorial at the May 2017 Embedded Vision Summit. 360-degree video systems use multiple cameras to capture a complete view of their surroundings. These systems are being adopted in cars, drones, virtual reality, and online streaming systems. At first glance, these systems wouldn’t seem require computer vision since they’re simply presenting images that the cameras capture. But even relatively simple 360-degree video systems require computer vision techniques to geometrically align the cameras – both in the factory and while in use. Additionally, differences in illumination between the cameras cause color and brightness mismatches, which must be addressed when combining images from different cameras. Computer vision also comes into play when rendering the captured 360-degree video. For example, some simple automotive systems simply provide a top-down view, but more sophisticated systems enable the driver to select the desired viewpoint. In this talk, Jacobs explores the challenges, trade-offs and lessons learned while developing 360-degree video systems, with a focus on the crucial role that computer vision plays in these systems.

1. Copyright © 2017 videantis GmbH 1
Marco Jacobs
May 2017
Computer-vision-based 360-degree Video
Systems: Architectures, Algorithms and
Trade-offs

2. Copyright © 2017 videantis GmbH 2
About videantis
#1 vision
processor
100% vision
company
automotive
since 2008
Algorithm partner

3. Copyright © 2017 videantis GmbH 3
360-degree video systems
Automotive
Consumer
Capture Vision Render
Calibration to
align images
Blindness
detection
3D from 2D
Obstacle
detection

4. Copyright © 2017 videantis GmbH 4
Extend visibility, warn driver and autonomy
Problem:
Limited
visibility
Solution:
Surround view
& automation

5. Copyright © 2017 videantis GmbH 5
360-video processing pipeline
Display path
Vision path
Capture Render and Act
Display
Calibration
Render
Camera parameters
Color & intensity
Warp
Color correct
Stitch
+Graphics
Vision
2-4 Cameras
Wide angle lens
Image sensor
ISP
Video codec
Analytics
Blindness detect
Crossing traffic alert
Pedestrian detection
Parking assist
Blind spot assist
Lane recognition ETC

6. Copyright © 2017 videantis GmbH 6
Manufacturing
tolerances
Mechanical
shifts
Vehicle load Tire pressure
Calibration – camera parameters change
Accurate camera parameters
needed to generate
surround view

7. Copyright © 2017 videantis GmbH 7
Calibration – required for aligning the images
Camera manufacturing Assembly line Vehicle power on While driving
Intrinsics Extrinsics
Photometric alignment
Markers Markerless
focal length
sensor resolution
principal point
lens distortion
camera's location in the world:
translation (x, y, z)
rotation (pitch, yaw, roll)
use patterns that
can easily be found
1. Matching image features
2. Using vanishing point
Match color and intensity
Changes based on sensor gain
and scene illumination
Different calibration techniques
for production and in field

8. Copyright © 2017 videantis GmbH 8
Input information
• Static:
initial extrinsic parameters
• Dynamic: CAN data
(speed and steering angle)
Computing the vanishing point
• Sparse optical flow to find
motion vectors
• Vanishing point where flow
vectors intersect
Benefits
• Single camera, not multiple
• Less compute compared to
feature matching
• Estimates ground plane location
Results
• ~1 minute at 5 – 120 km/h
• <1 degree accuracy
Calibration – Extrinsics using vanishing point
dy (Pitch)
dx (Yaw)
dr
(Roll)

9. Copyright © 2017 videantis GmbH 9
Rendering stage uses output from calibration
+3D model
Calibration
Mesh geometry
Color correction data
Alpha blending data
2D / 3D model params
Render
Warp
Color correct
Blend & stitch
2D / 3D Graphics

10. Copyright © 2017 videantis GmbH 10
Projection onto the ground plane
Projection onto ground plane flattens & stretches objects
Image formed in cameraLight from surface Projected pixel

11. Copyright © 2017 videantis GmbH 11
Projection onto a bowl shape
Image formed in cameraLight from surface Projected pixel
Projection onto bowl gives more accurate images

12. Copyright © 2017 videantis GmbH 12
• Warping for surround view systems:
• Projections onto surface implemented with
generic image warp
• Uses camera calibration results
• Wide angle lens correction single camera view:
• Accounts for camera (radial) lens distortion,
translation, scale, and yaw, pitch, roll
• Single camera view often shown in addition
• Generic mesh-based implementation:
• Mesh of points defines geometric transformation
• Similar to OpenCV remap() function
Warping algorithm (1/2)

13. Copyright © 2017 videantis GmbH 13
Output from intrinsic & extrinsic calibration:
• Surround view mesh per camera
• Alpha blending per node for camera overlap
• Color correction per mesh node to account for
illumination
Parameters:
• 8-bit or 16-bit color (for wide dynamic range)
• Bilinear or bicubic interpolation
Special cases:
• Aliasing when scaling down – low pass filter
• Blurring when scaling up – use higher order
interpolation
Warping algorithm (2/2)

14. Copyright © 2017 videantis GmbH 14
Trend to incorporate more ADAS functions:
• Pedestrian detection for backover protection
• Autonomous parking (free space detection,
parking marker detection, path planning)
• Lane detection for lane departure warning, blind
spot assist
We will highlight a few:
• Blindness detection
• Crossing traffic alert
• Obstacle detection using structure from motion
ADAS functions in 360-video systems

15. Copyright © 2017 videantis GmbH 15
Algorithm:
• Separate image into tiles
• Classify each tile using image quality metrics
• Sharpness, Laplacian, gradients, Local Binary Patterns
• Support vector machine to find most important features
• Temporally filter grid to avoid state changes due to noise
Challenges:
• Homogeneous areas like sky and ground:
• Smooth gradient values, while dirt usually does not
• Low light performance. Detection may be turned off in dark
Blindness detection
Blindness
detect
camera
clean?

16. Copyright © 2017 videantis GmbH 16
Structure from motion:
• 3D point cloud from monocular camera
on moving vehicle, static objects
Point cloud analysis in order to:
• Warn for objects close to car
• Automated parking
• Adjust surround view generation
Enables removal of ultrasound sensors
Structure from motion (1/2)
Structure
from motion
3D point
cloud
car speed
& angle
Point cloud
analysis

17. Copyright © 2017 videantis GmbH 17
Algorithm:
• Vehicle speed received from CAN data
• Uses sparse optical flow, clustering,
and 3D reconstruction
• Operates on fisheye images
Challenges:
• Need accurate intrinsic and extrinsic
camera calibration. Small errors affect
height and distance estimations greatly
• Areas in images that don’t have
features (sky, walls, road)
• Moving obstacles
Structure from motion (2/2)
(R,T)
c0
c1
x1
x2
P P
Find camera motion
Estimate 3D location
Detect
feature
points
Sparse
optical
flow
Estimate
3D
location

18. Copyright © 2017 videantis GmbH 18
Crossing traffic alert (1/2)
Exiting parking space Exiting driveway
Crossing
traffic alert
time to
collisioncar speed

19. Copyright © 2017 videantis GmbH 19
Overview of method:
• Single monocular camera
• Vehicle speed received from CAN data
• Uses sparse optical flow, clustering, and
structure from motion
• After clustering, only track features
inside region of interest
Working conditions:
• No need to know type of vehicle, based
on movement of obstacles
• Low vehicle speed, up to 5 km/hr
Crossing traffic alert (2/2)
Sparse optical flow (LK)
Subtract vehicle egomotion
Cluster motion vectors
Estimate direction & velocity
Time to collision < threshold
Driver warning or apply brakes
Camera images
Vehicle
speed
Feature detection

20. Copyright © 2017 videantis GmbH 20
System architecture options
DisplayHead unit
Camera
Lens Sensor
ISP
Camera
Lens SensorCamera
Lens SensorCamera
Lens Sensor
ISP
ISP
ISP
360 view gen
Vision
Central head unit does it all

21. Copyright © 2017 videantis GmbH 21
System architecture options
DisplaySurround ECU
Camera
Lens Sensor
ISP
Camera
Lens SensorCamera
Lens SensorCamera
Lens Sensor
ISP
ISP
ISP
360 view gen
Head
unit
Vision
Surround view ECU sends rendered images to head unit

22. Copyright © 2017 videantis GmbH 22
System architecture options
DisplaySurround ECU
360 view gen
Head
unit
Camera
Lens Sensor ISP
Camera
Lens Sensor ISP
Camera
Lens Sensor ISP
Camera
Lens Sensor ISP
Vision Vision
Computer vision in cameras and ECU

23. Copyright © 2017 videantis GmbH 23
• Even simple 360-video requires computer vision for calibration
• During manufacturing and in the field
• Automotive 360-video rapidly adding more computer vision features
• Cameras are there, relatively inexpensive to add embedded vision
• Cameras are replacing other sensors like ultrasound
• Integration other vision systems in the car (e.g., front camera)
• System architectures still evolving
Summary
Huge opportunity for 360-vision/video, both in
automotive and consumer applications

24. Copyright © 2017 videantis GmbH 24
• Feature-based alignment chapter in Computer Vision: Algorithms and
Applications by Richard Szeliski
• http://szeliski.org/Book/drafts/SzeliskiBook_20100903_draft.pdf
• Structure from Motion tutorial from EV Summit 2015:
• https://www.youtube.com/watch?v=3Wy7zvUbPSM
• www.videantis.com
• www.adasens.com
Resources

25. Copyright © 2017 videantis GmbH 25
• Demos at booth 804
• marco.jacobs@videantis.com
Thank You!