Planning Under Uncertainty With Markov Decision Processes

Planning under Uncertainty with Markov Decision Processes: Lecture II Craig Boutilier Department of Computer Science University of Toronto

Recap ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Overview ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Dimensions of Abstraction (recap) A B C A B C A B C A B C A B C A B C A B C A B C A A B C A B A B C A B C = Uniform Nonuniform Exact Approximate Adaptive Fixed 5.3 5.3 5.3 5.3 2.9 2.9 9.3 9.3 5.3 5.2 5.5 5.3 2.9 2.7 9.3 9.0

Classical Regression ,[object Object],[object Object],[object Object],G  G C  C do(a)

Example: Regression in SitCalc ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Decision-Theoretic Regression ,[object Object],[object Object],[object Object]

Decision-Theoretic Regression ,[object Object],[object Object],[object Object],[object Object],Q t (a) V t-1 G 2 G 3 G 1 C 1 p1 p2 p3

Functional View of DTR ,[object Object],[object Object],CR M -10 0 V t-1 f Rm (Rm t, Rm t+1 ) f M (M t, M t+1 ) f T (T t, T t+1 ) f L (L t, L t+1 ) f Cr (L t, Cr t, Rc t, Cr t+1 ) f Rc (Rc t, Rc t+1 ) T t L t CR t RHC t T t+1 L t+1 CR t+1 RHC t+1 RHM t RHM t+1 M t M t+1

Functional View of DTR ,[object Object],Q a t (Rm t ,M t ,T t ,L t ,Cr t, Rc t ) = R +  Rm,M,T,L,Cr,Rc(t+1) Pr a (Rm t-1 ,M t-1 ,T t-1 ,L t-1 ,Cr t-1, Rc t-1 | Rm t ,M t ,T t ,L t ,Cr t, Rc t ) * V t-1 (Rm t-1 ,M t-1 ,T t-1 ,L t-1 ,Cr t+1, Rc t-1 ) = R +  Rm,M,T,L,Cr,Rc(t+1) f Rm (Rm t, Rm t-1 ) f M (M t, M t-1 ) f T (T t, T t-1 ) f L (L t, L t-1 ) f Cr (L t, Cr t, Rc t, Cr t-1 ) f Rc (Rc t, Rc t-1 ) V t-1 (M t-1 ,Cr t-1 ) = R +  M,Cr,Rc(t+1) f M (M t, M t-1 ) f Cr (L t, Cr t, Rc t, Cr t-1 ) V t-1 (M t-1 ,Cr t-1 ) = f (M t ,L t ,Cr t, Rc t )

Functional View of DTR ,[object Object],[object Object],[object Object],[object Object],[object Object]

Planning by DTR ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Structured Value Iteration ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Structured Policy and Value Function DelC BuyC GetU Noop U R W Loc Go Loc HCR HCU 8.36 8.45 7.45 U R W 6.81 7.64 6.64 U R W 5.62 6.19 5.19 U R W 6.10 6.83 5.83 U R W HCR HCU 9.00 W 10.00 Loc Loc

Structured Policy Evaluation: Trees ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

A Simple Action/Reward Example X Y Z X Y Z X Y 0.9 0.0 X 1.0 0.0 1.0 Y Z 0.9 0.0 1.0 Z 10 0 Network Rep’n for Action A Reward Function R

Example: Generation of V 1 V 0 = R Z 0 10 Y Z Z: 0.9 Z: 0.0 Z: 1.0 Step 1 Y Z 9.0 0.0 10.0 Step 2 Y Z 8.1 0.0 19.0 Step 3: V 1

Example: Generation of V 2 Y Z 8.1 0.0 19.0 V 1 Step 1 Step 2 Y X Y Z Y: 0.9 Z: 0.9 Y: 0.9 Z: 0.0 Y:0.9 Z: 1.0 Z Y: 1.0 Y: 0.0 Z: 0.0 Y:0.0 Z: 1.0 X Y Y: 0.9 Y: 0.0 Y: 1.0

Some Results: Natural Examples

A Bad Example for SPUDD/SPI Action a k makes X k true; makes X 1 ... X k-1 false; requires X 1 ... X k-1 true Reward: 10 if all X 1 ... X n true (Value function for n = 3 is shown)

A Good Example for SPUDD/SPI Action a k makes X k true; requires X 1 ... X k-1 true Reward: 10 if all X 1 ... X n true (Value function for n = 3 is shown)

DTR: Relative Merits ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Approximate DTR ,[object Object],[object Object],[object Object],[object Object]

A Pruned Value ADD 8.36 8.45 7.45 U R W 6.81 7.64 6.64 U R W 5.62 6.19 5.19 U R W HCR HCU 9.00 W 10.00 Loc [7.45, 8.45] Loc HCR HCU [9.00, 10.00] [6.64, 7.64] [5.19, 6.19]

Approximate Structured VI ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Approximate DTR: Relative Merits ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

First-order DT Regression ,[object Object],[object Object],[object Object],[object Object]

SitCal: Domain Model (Recap) ,[object Object],[object Object],[object Object],[object Object]

Axiomatizing Causal Laws (Recap)

Stochastic Action Axioms (Recap) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Specifying Objectives (Recap) ,[object Object]

First-Order DT Regression: Input ,[object Object],[object Object],[object Object],[object Object],[object Object], t.On(B,t,s) : 10  t.On(B,t,s) : 0 load(b,t) loadS(b,t) : On(b,t) loadF(b,t) : ----- Rain ¬Rain 0.7 0.9 0.3 0.1

First-Order DT Regression: Output ,[object Object],[object Object],[object Object],[object Object],[object Object]

Step 1 ,[object Object],A. B. C. D.

Step 2 ,[object Object],[object Object],[object Object],A: LoadS, pr =0.7,val=10 D: LoadF, pr =0.3,val=0

Step 2: Graphical View  t.On(B,t,s) : 10  t.On(B,t,s) : 0  t.On(B,t,s) & Rain(s) & b=B & loc(b,s)=loc(t,s)  t.On(B,t,s) (  b=B v  loc(b,s)=loc(t,s)) &  t.On(B,t,s)  t.On(B,t,s) &  Rain(s) & b=B & loc(b,s)=loc(t,s) 10 7 9 0 1.0 0.7 0.1 0.9 0.3 1.0

Step 2: With Logical Simplification

DP with DT Regression ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Intra-action Maximization ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Intra-action Maximization Example

Inter-action Maximization ,[object Object],[object Object]

FODTR: Summary ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

FODTR: Implementation ,[object Object],[object Object],[object Object],[object Object]

Example Optimal Value Function

Benefits of F.O. Regression ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Function Approximation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Linear Function Approximation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Flexibility of Linear Decomposition ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Linear Approx: Components ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Approximate Value Iteration ,[object Object],[object Object],[object Object],[object Object],[object Object]

Projection ,[object Object],[object Object],[object Object],[object Object]

Projection as Linear Program ,[object Object],[object Object],[object Object],[object Object],Vars: w 1 , ..., w k ,  Minimize:  S.T.   V(s) – A w (s) ,  s   A w (s) - V(s) ,  s  measures max norm difference between V and “best fit”

Approximate Value Iteration ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Factored MDPs ,[object Object],[object Object],[object Object],[object Object]

Assumptions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],X 1 X’ 1 X 2 X 3 X’ 2 X’ 3 R(X 1 X 2 X 3 ) = R 1 (X 1 X 2 ) + R 2 (X 3 )

Factored AVI ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Compactness of Bellman Backup ,[object Object],[object Object]

Compactness of Bellman Backup ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Factored Projection ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Variable Elimination ,[object Object],[object Object],max X 1 X 2 X 3 X 4 X 5 X 6 { f 1 (X 1 X 2 X 3 ) + f 2 (X 3 X 4 ) + f 3 (X 4 X 5 X 6 ) } Elim X 1 : Replace f 1 (X 1 X 2 X 3 ) with f 4 (X 2 X 3 ) = max X 1 { f 1 (X 1 X 2 X 3 ) } Elim X 3 : Replace f 2 (X 3 X 4 ) and f 4 (X 2 X 3 ) with f 5 (X 2 X 4 ) = max X 3 { f 1 (X 1 X 2 X 3 ) + f 4 (X 2 X 3 ) } etc. (eliminating each variable in turn until maximum value is computed over entire state space)

Factored Projection: Factored LP ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Vars: w 1 , ..., w k ,  Minimize:  S.T.   V(s) – A w (s) ,  s   A w (s) - V(s) ,  s

Factored Projection: Factored LP ,[object Object],[object Object],[object Object],[object Object]

Factored Projection: Factored LP ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Factored Projection: Factored LP ,[object Object],[object Object],u(f j ,z 1 ,...,z n ) = f j (z 1 ,...,z n ; w) ,  z 1 ,...,z n

Factored Projection: Factored LP ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],u(g k ,z 1 ,...,z n )  g k1 (z 1 ,...,z n1 ) + g k1 (z 1 ,...,z n1 )+ ... ,  x k ,  z 1 ,...,z n

Factored Projection: Factored LP ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],  u final ()   max {  f j ( z j ; w) , x  X } = max {V(s) – A w (s) , s  S}

Some Results [GKP-01] ,[object Object],[object Object],[object Object]

Some Results [GKP-01] Computation Time

Some Results [GKP-01] Relative error wrt optimal VF (small problems)

Linear Approximation: Summary ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

An LP Formulation ,[object Object],[object Object],[object Object],Vars: V(s) Minimize:  s  V(s) S.T. V(s)  (L a V)(s) ,  a,s

Using Structure in LP Formulation ,[object Object],[object Object],[object Object],[object Object]

Good Basis Sets ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Parallel Problem Decomposition ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],MDP1 MDP2 MDP3

Generating SubMDPs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Generating SubMDPs Dynamic Bayes Net over Variable Set

Generating SubMDPs Green SubMDP (subset of variables)

Generating SubMDPs Red SubMDP (subset of variables)

Composing Solutions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Search-based Composition ,[object Object],s2 a1 s3 a1 a2 a2 a1 s4 a2 s5 s1 p2 p2 p3 p4 Exp Exp Max

Search-based Composition ,[object Object],[object Object],[object Object],s2 a1 s3 a1 a2 a2 a1 s4 a2 s5 s1 p2 p2 p3 p4 Exp Exp Max V(s) <= f 1 (s) + f 2 (s) + ... + f k (s) V(s) >= max { f 1 (s) , f 2 (s) , ... f k (s) }

Offline Composition ,[object Object],[object Object],[object Object],[object Object],[object Object]

Wrap Up ,[object Object],[object Object],[object Object]

Other Techniques ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Extending the Model ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

References ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

References (con’t) ,[object Object],[object Object],[object Object]

References (con’t) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Planning Under Uncertainty With Markov Decision Processes

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (10)

Similar to Planning Under Uncertainty With Markov Decision Processes

Similar to Planning Under Uncertainty With Markov Decision Processes (20)

More from ahmad bassiouny

More from ahmad bassiouny (20)

Recently uploaded

Recently uploaded (20)

Planning Under Uncertainty With Markov Decision Processes