Autonomous vehicles primary need is to have a robust and precise perception of the surrounding environment. To achieve accurate results in autonomous driving a combination of several different sensors is used. This technique is commonly known as sensor fusion. However, a critical aspect of autonomous driving perception is also the reactivity: we do not only need to perform sensor fusion properly, but we also have to respect real time requirements. The latter are fundamental to guarantee predictability of the system, avoid anomalous situations and prevent hazards. Performing fusion online is not trivial because sensors returns a lot of data and process them may be time consuming. The solution we present is an alignment of the sensors, which combines information from LiDAR and cameras, that is performed once in preprocessing and allow us to exploit the precomputed matching in real time.

1. Sensor fusion for autonomous
driving perception
Bagni Fabio
228594@studenti.unimore .it
San Francisco - 15 May 2019
1

2. Fabio Bagni
San Francisco - 15 May 2019
Autonomous driving perception
Precision and reactivity
Risks of accidents, traffic laws respect.
2

3. Fabio Bagni
San Francisco - 15 May 2019
Embedded platforms
3

4. Fabio Bagni
San Francisco - 15 May 2019
Goals
Real time 3D detection
● High performances
○ required 10 Hz frequency
● High precision
● Real time on embedded platforms
4

5. Fabio Bagni
San Francisco - 15 May 2019
UNIMORE HiPeRT prototype
5

6. Fabio Bagni
San Francisco - 15 May 2019
Sensors
● LiDAR
○ 3D point cloud
● Cameras
○ images
Heterogeneous outputs
6

7. Fabio Bagni
San Francisco - 15 May 2019
Sensor fusion
What is sensor fusion:
● Matching of different sensors
views
Why sensor fusion:
● Heterogeneous information of
environment
7

8. Fabio Bagni
San Francisco - 15 May 2019
Fields of view
LiDAR: 360 degrees around the vehicle
Camera: wide angle, disposed do cover 360
degrees.
8
Cameras
LiDAR

9. Fabio Bagni
San Francisco - 15 May 2019
Deep learning approach
Camera:
● Bird’s Eye View estimation
● Object detection
LiDAR
● Bird’s Eye View calculation
● Object detection
9
Image
Point
cloud
3D
detection
3D
detection
Fusion

10. Fabio Bagni
San Francisco - 15 May 2019
Deep learning approach time
Image stitching for multiple camera
fusion
10
➔ Under10 Hz
Notenoughresponsive
1. Ming Liang, Bin Yang, Shenlong Wang and Raquel Urtasun: Deep Continuous
Fusion for Multi-Sensor3D ObjectDetection.In: ECCV (2018).

11. Fabio Bagni
San Francisco - 15 May 2019
Static alignment sensor fusion
Alignment of LiDAR with each camera
individually.
11
Calibration
Preprocessing

12. Fabio Bagni
San Francisco - 15 May 2019
Features extraction
Matching is made by a calibration with a
perforated panel.
Holes are detected by all sensors.
Holes’ centers are used as features to be
matched.
12

13. Fabio Bagni
San Francisco - 15 May 2019
Cylindrical projection
Fields of view matching is made projecting sensors’ outputs on the surface of a
cylinder.
13

14. Fabio Bagni
San Francisco - 15 May 2019
Projections complexity
LiDAR cylindrical projection: Depthmap
O(n) : one operation per point
14
Camera cylindrical projection
O(n) : one operation per pixel
PARALLELIZABLE

15. Fabio Bagni
San Francisco - 15 May 2019
Calibration results
15

16. Fabio Bagni
San Francisco - 15 May 2019
Colored points
Alignment of each point with the
corresponding pixel on the image.
16

17. Fabio Bagni
San Francisco - 15 May 2019
Object detection
Neural network for object detection on
camera frame
Clustering of LiDAR points
Bounding boxes matching
17

18. Fabio Bagni
San Francisco - 15 May 2019
Object detection results
18

19. Fabio Bagni
San Francisco - 15 May 2019
Object detection
average time
19
LiDAR output : 10 Hz
Time limit : 100 ms

20. Thanks for your attention
Fabio Bagni - 228594@studenti.unimore.it
20
Fabio Bagni
San Francisco - 15 May 2019

21. 21

22. Fabio Bagni
San Francisco - 15 May 2019
Homography
Homography allow to translate camera projection plane on the LiDAR projection plane.
This method make cylindrical surfaces match.
22