Consider the following gridworld with 5 states. States D and E are terminal states and Assume there A RoboAnt would like to use MDP to maximize its expected utility. There are four possible actions of \{East, West, North, South\} for non-terminal states. The only possible action for terminal states is Exit. After RoboAnt exits the gridworld, it gets the reward associated with that exiting/terminal state. If there is a wall in a specific direction, RobotAnt stays in a current state. RoboAnt has a mechanical issue with his legs, and which makes it slide into the side squares with the probability of 0.2 when it intends to move forward. for example: state=B, T(B, south, A)=0.2, T(B,south,D )=0.2,T(B, south, C)=0.6. Assume living reward is -1. Assume discounting factor is =1 i) Compute V values in states B and C and F after two iterations when the policy of moving north for all the non-terminal states is applied. (Reminder: for terminal states, onl action 'Exit' action is available. V2i(B)= ?,V2i(C)= ? Part 2: Now, we assume we run the value iteration for the fixed policy described in part (i) and we converge to the following values (note these numbers are not the actual values and only considered to simplify the calculation) Vi(A)=4,Vi(B)=2,Vi(C)=1,Vi(D)=10,Vi(E)=+10 ii) Use policy iteration method to update the policy in states A and B after one iteration. Compute i+1(A),i+1(B).

1. Consider the following gridworld with 5 states. States D and E are terminal states and Assume
there A RoboAnt would like to use MDP to maximize its expected utility. There are four
possible actions of {East, West, North, South} for non-terminal states. The only possible action
for terminal states is Exit. After RoboAnt exits the gridworld, it gets the reward associated with
that exiting/terminal state. If there is a wall in a specific direction, RobotAnt stays in a current
state. RoboAnt has a mechanical issue with his legs, and which makes it slide into the side
squares with the probability of 0.2 when it intends to move forward. for example: state=B, T(B,
south, A)=0.2, T(B,south,D )=0.2,T(B, south, C)=0.6. Assume living reward is -1. Assume
discounting factor is =1 i) Compute V values in states B and C and F after two iterations when
the policy of moving north for all the non-terminal states is applied. (Reminder: for terminal
states, onl action 'Exit' action is available. V2i(B)= ?,V2i(C)= ? Part 2: Now, we assume we run
the value iteration for the fixed policy described in part (i) and we converge to the following
values (note these numbers are not the actual values and only considered to simplify the
calculation) Vi(A)=4,Vi(B)=2,Vi(C)=1,Vi(D)=10,Vi(E)=+10 ii) Use policy iteration method to
update the policy in states A and B after one iteration. Compute i+1(A),i+1(B)