this talk was an introduction to Reinforcement Learning based on the book by Andrew Barto and Richard S. Sutton. We explained the main components of an RL problem and detailed the tabular solutions and approximate solutions methods.
2. Hi
I am a Research Engineer in Applied RL
at InstaDeep and ML GDE
For the past the 3 years, I worked on
applying scalable DeepRL methods
application for placement on system
on chips and routing for printed circuit
boards
8. Why these models are labeled as smart
Machine Learning models are able to learn to make decisions to
achieve a predetermined goal
8
9. 9
Key types of Machine Learning tasks
Supervised
Learning
Unsupervised
Learning
Clustering
Association
Regression
Classification Translation
Identify population groups
Recommending products Recommending friends
Weather forecasting
Object
detection
Generate Image based
on latent Variables
10. 10
All tasks optimize a prediction loss
- Mean squared error loss
- Cross entropy loss
- Categorical cross entropy loss
- Cosine similarity loss
And many more ...
Using Stochastic gradient descent Algorithm
to optimize an objective function:
11. Tasks optimize for a goal by taking a
sequence of decision within an
environment
11
13. 13
Sequential decision making via Reinforcement Learning
Winning a chess game
- Optimise behavior based on a feedback signal (Reward)
- Learn an optimal behavior (policy) by interactions
with the world (environment) without provided examples
by interacting
- The feedback signal (Reward) on your actions can be
immediate or deferred (win or lose the game)
- The quality of the action you take depends on the current
state and the final outcome of the task (episode)
15. 15
1. The Reinforcement Learning framework
Environment
Agent
Reward
Next
observation
Action
- The agent interacts with an environment
within a finite horizon (episode)
- At each step t:
- Environment emits observation Oₜ
- Agent chooses an action Aₜ
- Environment executes the agent action
- Environment emits the reward Rₜ₊₁ and next observation Oₜ₊₁
16. The reward hypothesis
Any goal can be formalized as the outcome of maximizing a
cumulative reward
16
17. 17
2. The Rewards hypothesis
- A reward Rₜ indicates how well the agent is doing at timestep t
- The goal is to maximize the cumulative reward for the given task collected within an episode
- The episode return of state Sₜ depends on the sequence of actions that follows.
- The return can be discounted by 0 ≤ 𝛄 ≤ 1 to determine how much the agents cares about rewards in
the distant future relative to those in the immediate future.
For the rest of the presentation 𝛄 = 1
18. Estimators in maths
Estimation means having a rough calculation of the value, number,
quantity, or extent of something
18
19. 19
3. State Value function V(s)
- V(Sₜ) represents the expected return (cumulative reward) starting from state Sₜ and picking
actions following a policy
- Since we can define the return recursively
20. 20
3. State-Action Value function
4. State-Action Value function q(s,a)
- q(sₜ, a) represents the expected return (cumulative reward) starting from state sₜ and taking
action a then continue picking actions following a policy
- Given state action value function, we can derive a policy by picking the action corresponding to
highest Q value (Q-learning https://arxiv.org/abs/1312.5602)
20
21. 21
3. State-Action Value function
5. Agent observation
- The agent observation is a mapped from the environment state, Oₜ = f(Sₜ).
- The agent observation is not necessarily equal to the environment state.
- The environment is fully observable if Oₜ = Sₜ
- The environment is partially observable if Oₜʹ = Oₜ and Sₜʹ = Sₜ
21
Partially observed
environemnt
22. 22
- A mathematical formulation of the agent interaction with the environment
- It requires that the environment is fully observable
6. Markov decision process
- An MDP is a tuple (S, A, p, γ) where:
- S is the set of all possible states
- A is the set of all possible actions
- p(r,s′ | s,a) is the transition function or joint probability of a reward r and next state s′,
given a state s and action a
- γ ∈ [0, 1] is a discount factor that trades off later rewards to earlier ones
23. Markov decision principal
The Future is independent from the past given the present
The current state summarizes the history of the agent
23
24. 24
- Given the full horizon
Hₜ
7. Markovian state
- A state is called markovian only if
- If the environment is partially observable then the state is not Markovian
- We can turn a state to a Markovian state by stacking horizon data
Markovian state
Non Markovian state
25. Recap
- MDP is the representation of the agent-environment interaction
25
- Every RL problem can be formulated to a reward goal
- Agent components are: State, Value function, Policy, the world model
31. 31
3. State-Action Value function
1. Prediction and Control
- Prediction: given a policy, we can predict (evaluate) the future return given the current state
(learn value function)
- Control: improve your actions choices (learn policy function)
- Prediction and control can be strongly related
31
32. 32
3. State-Action Value function
2. Learning and Planning
- At first, the environment can be unknown to the agent.
- The agent learn the model of the world by interaction and exploration
- Once the model is learnt (sometime given ie: chess), the agent start planning actions to reach
optimal policy
32
35. 35
3. State-Action Value function
Tabular MDPs explained
- The state and action space is small enough to be represented by arrays or tables
- Given the exact quantification of the possible states and actions, we can find exactly the optimal
solution for the prediction (value function) and control (policy) problems
- 27 states
- 4 actions
- A reward of -1 for each step
35
37. Definition
The term dynamic programming (DP) refers to a collection of
algorithms that can be used to compute optimal policies given a
perfect model of the environment as a Markov decision process (MDP).
- Richard S.Sutton and Andrew G. Barto -
37
38. 38
3. State-Action Value function
1. Policy Evaluation
- Given an arbitrary policy π ,we want to compute the corresponding state value function V𝜋
- We iteratively iterate over all the states and update the state value using the equation
below until we reach a state of convergence
38
40. 40
3. State-Action Value function
2. Policy Improvement
- The goal of computing the value function for a policy is to help find a better policy
- Given the new value function, we can define the new policy
40
44. Notes
Monte Carlo methods require only experience—sample sequences of
states, actions, and rewards from actual or simulated interaction
with an environment without the need for the full probability
distribution of state, reward over actions
- Richard S.Sutton and Andrew G. Barto -
44
45. 45
3. State-Action Value function
1. First visit Monte-Carlo for prediction
- Given an arbitrary policy π ,we can estimate V𝜋
- Once the Algorithm converges we can move to policy improvement
- An acceptable estimate of Gₜ would be the average of all the encountered discounted
returns after infinite visits to the state Gₜ
45
47. 47
3. State-Action Value function
2. First visit Monte-Carlo for control
- Given an arbitrary initial policy π ,we can estimate state action value V𝜋.
- Instead of averaging the return of the visited state Sₜ, we average the return of the
visited state action pair Sₜ, Aₜ .
- The new policy can be calculated by choosing the action corresponding to best Q value
47
49. Exploration vs Exploitation problem
49
All learning control methods face a dilemma: they seek to learn
action values conditional on subsequent optimal behavior, but they
need to behave non-optimally in order to explore all actions
- Richard S.Sutton and Andrew G. Barto -
50. 50
3. State-Action Value function
Off policy and On policy methods
- Learning control methods fall into two categories: off policy and on policy methods
- On policy methods update the current policy using the data generated by the former (which
what we have been doing so far)
- Off policy methods update the current policy using data generated by two policies
- Target policy: the current policy being learned about
- Behavior policy: the policy responsible of generating a exploratory behavior ( random
actions, data generated by old policies )
50
51. 51
3. State-Action Value function
3. Monte-Carlo Generalized Policy Iteration
- Sample episode 1, . . ., k, . . ., using π: {S₁, A₂, R₂, ..., Sₜ } ∼ π
- For each state St and action At in the episode
- Improve policy based on new action-value function
51
52. Problems with MC methods
- High variance given
52
- Waiting until the end of the of the episode
54. 54
3. State-Action Value function
TD-learning explained
- TD-learning is a combination of monte-carlo and dynamic programming ideas.
- It is the backbone of most of state of the art Deep Reinforcement Learning algorithm DQN, PPO ...
- Like DP, TD-learning update the estimate based on another estimate. We call this Bootstrapping
- Like MC, TD-L learns directly from experiences without the need for a model of the environment.
54
55. 55
3. State-Action Value function
1. TD Prediction
- MC methods uses the episode return Gₜ as the target for the value for Sₜ.
- Unlike MC methods, TD methods update the value at each step and use an estimate of
Gₜ, we call the TD-Target.
55
57. 57
3. State-Action Value function
1. Example of MC vs TD prediction
- we are driving home from work and we try
to estimate how long it will take us.
- At each step, we re-estimated our time
because of complications (e.g. car
doesn’t work, highway is busy, etc).
- How can we update our estimate of the
time it takes to get home for next time
we leave work?
57
58. 58
3. State-Action Value function
1. Example of MC vs TD prediction
- we are driving home from work and we try
to estimate how long it will take us.
- At each step, we re-estimated our time
because of complications (e.g. car
doesn’t work, highway is busy, etc).
- How can we update our estimate of the
time it takes to get home for next time
we leave work?
Monte Carlo TD-Learning
58
59. 59
3. State-Action Value function
2. Sarsa: On Policy TD Control
- Similarly to MC method, we learn a policy by learning the action value function Q(S,A)
- The Algorithm is called Sarsa as it relies on transition { state, action, reward }
- Theis Algorithm is the backbone to the famous Deep Q-learning paper
59
63. 63
3. State-Action Value function
Dynamic programming
Policy Evaluation
Policy Improvement
Value Iteration
Tabular solution methods
Model based Model free
Monte Carlo methods
TD-learning methods
1. The family of tabular solution methods
63
65. OpenAI: solving the rubix cube using a single handed robot
- The robots observes the world
through camera lenses,
censors..ect
- The state space is infinite and
it's not practical to store in
a table
- The state space consists of a
set of unstructured data and
not tabular data
65
66. Deep neural network are the best fit for unstructured data
Function approximator
Action values
State value
State
Rubik's cube image
Linear or non
Linear function Output of the
function
66
68. Function derivatives and Gradient
68
- The derivative of a function f measure the sensitivity to change
with respect to the argument x
- The gradient of a function with respect to x, measure by how much x
needs to change so we reach a minimum
70. 70
1. Value function approximation
- Given a function approximator with a set of
weights 𝔀, minimize 𝘑(w)
- Using stochastic Gradient Descent
algorithms we form a good estimator for the
loss
- The loss target can be the MC return of the
TD target.
72. 72
3. State-Action Value function
Dynamic programming
Policy Evaluation
Policy Improvement
Value Iteration
Tabular solution
methods
Model based Model free
Monte Carlo methods
TD-learning methods
What we've learnt doay
Value approximation
Approximation
method
Policy gradient
72
