2017/12/8 Deep Learning JP: http://deeplearning.jp/seminar-2/

1. 1
DEEP LEARNING JP
[DL Papers]
http://deeplearning.jp/
Deep Reinforcement Learning that Matters
Reiji Hatsugai

2. •
–
–
•
•
difficulty
•
•
2

3. 3

4. 4

5. : HalfCheetah
5

6. : Hopper
6

7. 7

8. 8

9. 9

10. 10

11. 11
!"~$(&|(")
("*+~,(-.|(", !")
0"*+ = 0((", !", ("*+)

12. 12
!"~$(&|(")
("*+~,(-.|(", !")
0"*+ = 0((", !", ("*+)
$
π∗
= argmax
π
Eπ [ γ τ
rτ ]
τ =0
∞
∑

13. 13
TRPO
DQN DDQN
A3C
UNREAL PCL
ACER
PPO
Q-Prop
IPG
ACKTR
DDPG
D4PG
SAC
Soft Q

14. 14
TRPO
DQN DDQN
A3C
UNREAL PCL
ACER
PPO
Q-Prop
IPG
ACKTR
DDPG
D4PG
SAC
Soft Q
『『深深層層』』強強化化学学習習ににななっっててかからら
たたくくささんんのの手手法法がが開開発発さされれたた

15. •
1.
2.
3.
4.
15

16. Deep Reinforcement Learning that Matters
• ICML2017 reproducibility work shop Reproducibility of
Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• AAAI2018 accepted
•
–
–
•
•
16

17. Deep Reinforcement Learning that Matters
•
– ACKTR (Wu et al. 2017)
– PPO (Schulman et al. 2017)
– DDPG (Lillicrap et al. 2015)
– TRPO (Schulman et al. 2015)
• ACKTR, PPO
• DDPG, TRPO baseline
•
17

18. Deep Reinforcement Learning that Matters
• Network Architecture
• Reward Scale
• Random Seeds and Trials
• Environments
• Codebases
• Reporting Evaluation Metrics
18

19. Deep Reinforcement Learning that Matters
• Network Architecture
• Reward Scale
• Random Seeds and Trials
• Environments
• Codebases
• Reporting Evaluation Metrics
19
外因的なもの

20. Deep Reinforcement Learning that Matters
• Network Architecture
• Reward Scale
• Random Seeds and Trials
• Environments
• Codebases
• Reporting Evaluation Metrics
20
内因的なもの

21. Network Architecture
•
– (64, 64) (rllab)
– (100, 50, 25) (Q-Prop)
– (400, 300) (DDPG)
•
• Activation Function
21

22. Policy Architecture
22

23. Activation Function
23

24. Network Architecture
• PPO
• Tanh
• PPO
• “This also suggests a possible need for hyper parameter agnostic algorithms”
•
24

25. Reward Scale
• Q DQN cliping
• 0.
= 20
• σ=0.1
•
LeCun et al .2012; Glorot and Bengio 2010; Vincent, de Brebisson, and Bouthillier 2015
•
25

26. Reward Scale
26

27. Reward Scale
• Reward Scale
•
• Reward Scale
• Layer norm
• Learning values across many orders of magnitude (Hado van Hasselt et al. 2016)
– adaptive
• HumanoidStandup-v1 100
– Reward Scale
27

28. Deep Reinforcement Learning that Matters
• Network Architecture
• Reward Scale
• Random Seeds and Trials
• Environments
• Codebases
• Reporting Evaluation Metrics
28
内因的なもの

29. Random Seeds and Trials
• 10 seed
• 10 5 5
•
29

30. Random Seeds and Trials
30

31. Random Seeds and Trials
31

32. Random Seeds and Trials
32
<0.05

33. Random Seeds and Trials
• 2
–
–
•
seed
• power analysis
•
33

34. Environment
• Hopper, HalfCheetah, Swimmer, Walker2D
•
34

35. HalfCheetah
35

36. Hopper
36

37. HalfCheetah
• HalfCheetah DDPG
• Hopper DDPG
• Reproducibility of Benchmarked Deep
Reinforcement Learning Tasks for Continuous Control
• DDPG Q
• HalfCheetah DDPG DDPG base
HalfCheetah unfair
37

38. Swimmer
38

39. Swimmer
• TRPO
• policy local optimal
•
•
39

40. Code base
• TRPO DDPG rllab, baseline
•
40

41. Code base
41

42. Code base
•
• dramatic impacts on performance
•
42

43. Reporting Evaluation Metrics
•
•
•
–
–
–
43

44. Deep Reinforcement Learning that Matters
•
•
–
–
–
–
•
– hyperparameters agnostic algorithm
• “There is often no clear winner among all benchmark environments.”
44

45. • HalfCheetah Hopper DDPG
stable, unstable
• task difficulty algorithm
• Simple Nearest Neighbor Policy Method for Continuous Control Tasks
– Nearest Neighbor Policy
– task difficulty task
– NN task
45

46. • NN-1, NN-2
•
• NN-1
1.
2. action
• NN-2
1.
2. action 1step 1
• Sparse reward
46

47. NN
47

48. Simple Nearest Neighbor
• Sparse Mountain Car
• HalfCheetah
• HalfCheetah
• task difficulty
• ICLR3,4,4
• NNPolicy
48

49. •
HalfCheetah
•
–
– sensor
• 3 MLP
• Towards Generalization and Simplicity in Continuous Control
– Policy parameterize RBF
– Natural Gradient
– Neural Net humanoid
– mujoco Todorov Natural Gradient Kakade 49

50. Towards Generalization and Simplicity in Continuous Control
50

51. •
• sensor DeepLearning
•
•
– sparse reward
–
• IL, IRL??
–
normalize
51