More Related Content Similar to DeepRacer-Workshop-HongKong-Donnie-Prakoso (20) More from Amazon Web Services (20) DeepRacer-Workshop-HongKong-Donnie-Prakoso1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Developer, Start Your Engine!
Donnie Prakoso
Technical Evangelist, ASEAN
Amazon Web Services
@donnieprakoso
donnieprakoso
2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
> hello, world
Donnie Prakoso, MSc
Technical Evangelist, AWS
@donnieprakoso
donnieprakoso
• Speak in Go and Python
• Microservices and Machine Learning
• I AWS User Groups
• Self-proclaimed Barista and Café Racer enthusiasts
3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Agenda
• AWS DeepRacer origin
• RL for the Sunday driver
• Virtual simulator
• Under the hood
• Rubber meets the road
4. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
5. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
6. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How can we put machine learning in the hands of all
developers?
7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
ML
FRAMEWORKS &
INFRASTRUCTU
RE
AI SERVICES
REKOGNITION
IMAGE POLLY TRANSCRIBE
TRANSLATE COMPREHEND LEX
REKOGNITION
VIDEO
Vision Speech Language Chatbots
AMAZON
SAGEMAKER
BUILD
TRAIN
FORECAST
Forecasting
TEXTRACT PERSONALIZE
Recommendations
DEPLOY
Pre-built algorithms & notebooks
Data labeling (GROUND TRUTH)
One-click model training & tuning
Optimization (NEO)
One-click deployment & hosting
ML SERVICES
Frameworks Interfaces Infrastructure
EC2 P3
& P3N
EC2 C5 FPGAs GREENGRASS ELASTIC
INFERENCE
Reinforcement learning
Algorithms & models (AWS MARKETPLACE
FOR MACHINE LEARNING)
The Amazon ML stack: Broadest & deepest set of
capabilities
8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How can we put
reinforcement learning
in the hands of all
developers? literally
9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer: An exciting way for developers to get hands-on experience
with reinforcement learning
Global Racing League
DeepRacer Console
Virtual simulator, to
train and evaluate
1/18 scale autonomous
race car
10. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Reinforcement learning in the broader AI context
Reinforcement
Learning
Supervised
Learning
Unsupervised
Learning
12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Machine learning overview
SUPERVISED UNSUPERVISED REINFORCEMENT
13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Reinforcement learning in the real world
Reward positive
behavior
Don’t reward
negative
behavior
The result!
14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Reinforcement learning terms
AGENT ENVIRONMENT STATE
ACTION
EPISODEREWARD
15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The reward function
The reward function incentivizes particular
behaviors and is at the core of
reinforcement learning
16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The reward function in a race grid
S G = 2
GOALAGENT
17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Incentivizing centerline behavior
0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
S 2 2 2 2 2 2 G = 2
0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
8.6 9.5 8.5 7.5 6.3 5.0 3.5 1.9
S 10.4 9.4 8.2 6.9 5.4 3.8 G = 2
8.6 9.5 8.5 7.5 6.3 5.0 3.5 1.9
Discount per step
0.9
18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer problem formulation
STATE
19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How does learning happen? VALUE FUNCTION
POLICY FUNCTION
20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
RL algorithms: Vanilla policy gradient
* Image Source: Landscape image is CC0 1.0 public domain
Data is only used once
• High variance of rewards
• Magnitude of update could be too large
J(q)New
weights
New
weights
0.4 ± ! 0.3 ± !
21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
RL algorithms: Proximal policy optimization (PPO)
(State, action, reward,
next state)
(st,at, rt, st+1)
Advantage
Improved model
22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
METHOD Supervised learning
HOW IT WORKS Expert driver controls a real
world car, that has a camera. Save the images
from the camera as inputs and corresponding
driving actions (speed and steering angle) as
outputs. Train a model.
RESULT Provide state(image) into model and
receive driving action
RL vs. other approaches for robotic racing
METHOD Reinforcement learning
HOW IT WORKS Virtual agent repeatedly
interacts with a simulated environment and
logs experience (image, action, new state,
reward). Experience is used to train a model,
and new model is used to get more
experience.
RESULT Provide state(image) into model and
receive driving action
23. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Cloud
AWS
DeepRacer
NAT gateway
VPC
AWS DeepRacer
Models
Simulation
video
Metrics
AWS DeepRacer simulator architecture
25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer console diagram
26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Programming your own reward function
27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Track components
TRACK CENTER
TRACK WALL
TRACK SURFACE aka ON-TRACK
FIELD aka OFF-TRACK
TRACK BOUNDARIES
28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Coordinate system and track waypoints
WAYPOINTS
X
Y
TRACK WIDTH
HEADING
29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Hyper parameters control the training algorithm
30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Action space
31. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer car specifications
CAR 18th scale 4WD with monster truck chassis
CPU Intel Atom Processor
MEMORY 4 GB RAM
STORAGE 32 GB (expandable)
WI-FI 802.11ac
CAMERA 4 MP camera with MJPEG
DRIVE BATTERY 1000 mAh lithium polymer
COMPUTE BATTERY 13600 mAh USB-C
SENSORS Integrated accelerometer and gyroscope
PORTS 4x USB-A, 1x USB-C, 1x Micro-USB, 1x HDMI
SOFTWARE Ubuntu OS 16.04.3 LTS, Intel OpenVINO
toolkit, ROS Kinetic
33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
ROS msg node
Stored file
ROS nodes
Web
Server
Publisher
Model
Optimizer
Video
M-JPEG
Web Server
Video
Inference
Results
Autonomous
Drive
Control
Node
Optimized
Model
Media engine
Camera
Model
Inference
engine
Manual
Drive
Navigation
Node
Servo & Motor
AWS DeepRacer software architecture
34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Simulation-to-real domain transfer
SIM-to-REAL CHALLENGE
Train model using simulated images, but
the race car using the images the car
experiences in the real world
STRATEGIES
Environment control
Domain randomization
Modularity and abstraction
35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Lab 1 – AWS DeepRacer service
OBJECTIVE Build your first AWS DeepRacer RL model
TIME 50 min.
1. Find the lab content here:
https://github.com/aws-samples/aws-deepracer-workshops/
2. Navigate to: Workshops/2019-AWSSummits-AWSDeepRacerService/Lab1
36. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Race for prizes and glory in the AWS DeepRacer League
Train your AWS DeepRacer model and compete:
• Online in the Virtual Circuit
• In person in the Summit Circuit (visit the Expo Hall)
www.deepracerleague.com
38. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.