Presentation of Ralf Herrtwich about autonomous driving and the approach to this by Mercedes Benz

1. Autonomous Driving to Advance Road Safety and Comfort
The S500 INTELLIGENT DRIVE
Ralf Herrtwich
Driver Assistance and Chassis Systems - Group Research and Advanced Engineering

2. Project Goals
Highly autonomous driving on the original Bertha Benz Route from
Mannheim to Pforzheim 125 years after the world's first overland drive
with as few driver interventions as possible
• floating within regular traffic
• with no special guidance in front
• with no alterations of given infrastructure
•
Goals:
Gain experience with autonomous driving beyond highways
• Experiment with a vehicle setup with regular production or
close-to-production sensors (only more)
• Cameras (mono and stereo)
• Radars
• Identify technical issues and examine different solutions
for future autonomous functions
• Prepare for innovation beyond 2020
• Push the envelope
•

3. Why We Wanted to Do It

4. Mercedes-Benz Intelligent Drive
Accident-free driving
Autonomous driving
Prevent and avoid accidents altogether
or at least mitigate their effects
to an amount of minimum harm and damage
Assist drivers with maneuvers if they want it,
but only when and where it is technically possible
without taking imponderable risks
Example:
PRE-SAFE BRAKE
Example:
DISTRONIC PLUS
Safety
Comfort
Key to the Mercedes-Benz Brand

5. Autonomous Driving in the New S- and E-Class
Mercedes-Benz Intelligent Drive
Stop & Go Pilot:
Vehicle moves autonomously
at low speeds in traffic jams
• Longitudinal and lateral control
incl. lane keeping in curves
• Lane mark detection
• Observation of vehicle ahead
(swarm mode)
• Intelligent hands-on detection
• Driver is called back into the loop
when needed
• Every 10-15 seconds
as soon as vehicle drives
30 km/h or more
•

6. Autonomous?
αὐτο "self" + νόμος "law"
Autonomy (as seen by Immanuel Kant):
The capacity of an individual or entity to make an informed, un-coerced decision within principles of reason.
Observations
Rules of behaviour
Actions

7. Stages of Autonomy (NHTSA Logic)
To here:
From here:
Level 0
No automation
Driver in charge
Level 1
Function-specific
automation
Single control
functions such as
speed selection,
braking or lane
keeping are
automated
Level 2
Combined function
automation
More than one control
function is automated
Driver expected to be
available for control
at all times and
on short notice
Level 3
Limited self-driving
automation
Vehicle takes control
most of the time
Driver expected to be
available for
occasional control
with comfortable
transition times
Level 4
Full self-driving
automation
Vehicle takes control
all of the time
Driver not expected to
be available for
control at any time

8. Complexity of Automation
Low ego velocity
High ego velocity
Structured
traffic
environment
Traffic Jam
Highway
Step1
Step 3
Chaotic
traffic
environment
Parking
Off-Highway
Step 2
Bertha Benz
Step 4
Drive

9. Complexity of Automation
Highway
Off-highway
Lanes
Wide
Narrow and not necessarily exclusive
Direction
Same direction for all
Vehicles from and in all directions
•
Maneuvers
Lane Change
•
•
Passing of in-road obstacles (e.g.
parked cars) incl. coordination with
oncoming traffic
Intersection navigation incl. turning
Roundabouts
Traffic signs
Speed limits
Traffic lights and priority rules
Road users
Vehicles only
Pedestrians and cyclists all around
Field of view
Wide
Obstructed

10. Where We Did It

11. The Historic Bertha Benz Route
In August 1888, Bertha Benz
took her husband's Patentmotorwagen
at dawn and, together with her two sons,
drove to Pforzheim to visit her family.
She arrived shortly before midnight
after what later became known as
the world's first overland drive.

12. Route Difficulty (Original Estimation)
Pforzheim
Mannheim
N

13. Route Difficulty (Actual Difficulty)
Pforzheim
Heidelberg Passage
Nussloch Passage
Pfinztal Passage
Ladenburg Passage
Bruchsal Passage
Mannheim Passage
Mannheim
Weingarten Passage
N

17. How We Did It

18. Vehicle Base
Regular S 500 with all
emergency braking systems
enabled as underlying
protection

19. Sensors

20. Actuators
Steering
Acceleration / Braking
Special software version for
Electrical Power Steering
to allow for larger steering angles
at higher speeds
Special software version for
DISTRONIC
to limit initial acceleration and
braking deceleration
(212 4x4 ZF EPS)
(222 MB DISTRONIC)

21. System Structure
Sensors
Artificial Intelligence
Actuators
Feature Map
Feature Loc Camera
ESP
Planning Map
Feature Localization
Localization
Vehicle Loc Sensors
Planning
EPS
Lane Map
Lane Localization
Stereo Camera
RDU Emulation
Object Detection
Traffic Light Map
Traffic Light Camera
Traffic Light Detection
360° Radars
Radar Processing
DISTRONIC Radars
ESP Sensor Cluster
Visualization

22. What the Car Sees

23. Stereo Vision
Left Image Disparity (Distance) Image Right Image
Color encoded distance:
close ….. far

24. Object Recognition: Pedestrians
Input
Hypothesis
Classification
Pedestrian examples
Tracking
Non-pedestrian examples
Image
Depth

25. Traffic Light Recognition
Traffic light recognition is not as easy as one may think since
When stopping in front of the traffic light, it must be in the field of view
• When approaching a traffic light on rural roads, it must be visible at large distances
• At intersections, we have to find “our” traffic light
•

26. Traffic Light Recognition
Easy
Medium
Hard
Impossible

27. What the Car Knows

28. What's in a Map?
Vehicle
Stop line
Traffic light
Trajectory plan: color = confidence

29. Map Architecture
Different map layers serve different purposes.
Navi map
(GDF)
Planning layer
Localization layers

30. Map Accuracy
Speed limit
+/- 10 m
Relative accuracy
of map features of 10 cm
desirable
Curb +/- 5 cm
Map data
Lane marking
+/- 5 cm
Traffic light
+/- 10 cm
Camera image of
road surface

31. Vehicle Localization
[
[
[
1.
Landmarks
for localization
are defined.
]
]
]
2.
A map is created
with such
localization landmarks.
3.
A camera detects landmarks.
Those are matched
with landmark maps
to compute the actual pose
of the vehicle.

32. Landmark Robustness and Diversity
Different environments require different types of landmarks.
Rural:
"Lane Loc"
Curbs
Lane Markings
Urban:
"Feature Loc"
Edge Detector
Collaboration
with KIT

33. Map Generation
To explore map generation for
autonomous driving, Daimler has
entered into a cooperation with
Nokia HERE as one of the world's
leading map suppliers.
Exploration areas:
Content needs (and accuracy needs)
for autonomous driving
• Localization support via landmarks
in maps
• Scalable and reliable data capturing
methods
• Online map updating
• Online map insufficiency reporting
incl. crowd sourcing concepts
• "Path clearance" concepts
•
Source: HERE

34. What the Car Thinks

35. Maneuvering Tasks
Behaviour generation
Object classification and prediction
Trajectory planning

36. Object Classification and Prediction
Hypothesis generation for movements
of dynamic objects based on
Prediction Path A (unlikely)
Object attributes
• Driving lanes
•
Probabilistic evaluation of hypotheses
Prediction w/o lane information
Prediction Path B (likely)

37. Behaviour Generation
Hierarchical concurrent state machine to determine vehicle behaviour in individual situations
Collaboration with KIT (former AnnieWay technology from Urban Challenge)

38. Trajectory Planning Examples

39. How the Car Behaves

40. Overland
Few road users, unobstructed view, dedicated lanes

41. Inner-City
Denser traffic, obstructed view, dedicated lanes

42. Parked Cars
Driving around static obstacles

43. Cyclists
Driving around dynamic obstacles

44. Pedestrians
Stopping for pedestrians at crosswalks

45. Intersections
Stopping at traffic lights at intersections

46. Turning
Observing cross and oncoming traffic
(possible manual confirm)

47. Roundabouts
Waiting for "empty slot" from a safe position

48. What Is Next

49. Complexity of Automation
Low ego velocity
High ego velocity
Structured
traffic
environment
Traffic Jam
Highway
Step1
Step 3
Chaotic
traffic
environment
Parking
Off-Highway
Step 2
Step 4