1. CS 221: Artificial Intelligence Lecture 8: MDPs Sebastian Thrun and Peter Norvig Slide credit: Dan Klein, Stuart Russell

2. Rhino Museum Tourguide

3. Minerva Robot

4. Pearl Robot

5. Mine Mapping Robot Groundhog

6. Mine Mapping Robot Groundhog

11. Stochastic, Partially Observable sign hell? heaven? [Sondik 72] [Littman/Cassandra/Kaelbling 97]

12. Stochastic, Partially Observable sign hell heaven sign heaven hell

13. Stochastic, Partially Observable sign heaven hell sign ? ? sign hell heaven start 50% 50%

14. Stochastic, Partially Observable sign ? ? start sign heaven hell sign hell heaven 50% 50% sign ? ? start

15. A Quiz  -dim continuous actions # states size belief space? sensors 3: s 1 , s 2 , s 3 3: s 1 , s 2 , s 3 2 3 -1: s 1 , s 2 , s 3 , s 12 , s 13 , s 23 , s 123 2-dim continuous: p ( S = s 1 ), p ( S = s 2 )  -dim continuous aargh! stochastic 1-dim continuous stochastic deterministic 3 perfect stochastic 3 perfect deterministic 3 abstract states deterministic 3 stochastic stochastic 3 none 2-dim continuous: p ( S = s 1 ), p ( S = s 2 ) deterministic 1-dim continuous stochastic stochastic  -dim continuous stochastic

18. Grid Futures Deterministic Grid World Stochastic Grid World E N S W E N S W ? X X X X X X

21. Example Optimal Policies R(s) = -2.0 R(s) = -0.4 R(s) = -0.03 R(s) = -0.01

31. Example: Bellman Updates max happens for a=right, other actions not shown Example:  =0.9, living reward=0, noise=0.2

35. Value Iteration: Example

36. Another Example Value Function and Plan Map

37. Another Example Value Function and Plan Map

38. Stochastic, Partially Observable sign ? ? start sign heaven hell sign hell heaven 50% 50% sign ? ? start

40. Introduction to POMDPs (1 of 3) p ( s 1 ) [Sondik 72, Littman, Kaelbling, Cassandra ‘97] 80  100 b a  0 b a  40 s 2 s 1 action a action b s 2 s 1  100 0 100 action a action b

41. Introduction to POMDPs (2 of 3) 80  100 b a  0 b a  40 s 2 s 1 p ( s 1 ) s 2 s 1  100 0 100 [Sondik 72, Littman, Kaelbling, Cassandra ‘97] 80% c 20% p ( s 1 ) s 2 s 1 ’ s 1 s 2 ’ p ( s 1 ’)

42. Introduction to POMDPs (3 of 3) 80  100 b a  0 b a  40 s 2 s 1 80% c 20% p ( s 1 ) s 2 s 1  100 0 100 [Sondik 72, Littman, Kaelbling, Cassandra ‘97] p ( s 1 ) s 2 s 1 s 1 s 2 p ( s 1 ’|A) B A 50% 50% 30% 70% B A p ( s 1 ’|B)

44. Why is This So Complex? State Space Planning (no state uncertainty) Belief Space Planning (full state uncertainties) ?

45. Belief Space Structure The controller may be globally uncertain... but not usually.

46. Augmented MDPs: [Roy et al, 98/99] conventional state space uncertainty (entropy)

47. Path Planning with Augmented MDPs Conventional planner Probabilistic Planner [Roy et al, 98/99] information gain