SBGames 2013

1. ADAPTIVE HIGH-LEVEL
STRATEGY LEARNING IN
STARCRAFT
SBGames 2013
Jiéverson Maissiat and Felipe Meneguzzi

2. Background

3. Reinforcement Learning
A learning technique for agents acting in the
environment, observing the reached states and
the received rewards at each step

4. Reinforcement Learning

5. Q-Learning
Updates the value of a state-action pair after
the action has been taken in the state and an
immediate reward has been received

6. Q-Learning
Q(s,a)
●
●
●
s represents the current state of the world;
a represents the action chosen by the agent;
Q(s,a) represents the value obtained the last time action a was executed at state s. This value is
often called Q-value.

7. Q-Learning
Q(s,a)
●
r
r represents the reward obtained after performing action a in state s;

8. Q-Learning
Q(s,a)
●
fit is responsible to adjust the reward;
fit( r )

9. Q-Learning
Q(s,a) = Q(s,a)
●
+
The Q-value is increased with the adjusted reward;
fit( r )

10. Q-Learning
Q(s,a) = Q(s,a)
●
+
α
*
r
α is the learning-rate, which determines the weight of new information over what the agent
already knows: a factor of 0 prevents the agent from learning anything (by keeping the Q-value
identical to its previous value) whereas a factor of 1 makes the agent consider all newly obtained
information;

11. Q-Learning
Q(s,a) =
●
0
If the Q-value starts in 0...
+
α
*
r

12. Q-Learning
Q(s,a) =
●
0
and the received reward is 2...
+
α
*
2

13. Q-Learning
Q(s,a) =
●
0
with a learning-rate of 0.3...
+
0.3 *
2

14. Q-Learning
0.6
●
=
0
than the Q-value is updated to 0.6;
+
0.3 *
2

15. Q-Learning
-2.4
●
=
0.6
+
Others updates will be done over the first;
0.3 *
-10

16. Q-Learning
Q(s,a) =
●
And so it follows.
-2.4
+
0.3 *
r

17. Parameter Control
α / learning-rate
Influence how the state values are computed,
and ultimately how a policy is generated

18. Learning Rate: α
●
High Learning-rate?
It can prevent the algorithm from converging, or lead to inaccuracies in the
computed value of each state (e.g. a local maximum);
●
Low Learning-rate?
It can hampers the convergence of the Q-value;
●
Varying the Learning-rate?
A typical way to vary the α-value, is to start interactions with a value close
to 1, and then decrease it over time toward 0.However, this approach is not
effective for dynamic environments, since a drastic change in the
environment with a learning-rate close to 0 prevents the agent from
learning the optimal policy in the changed environment.

19. Exploration Policy
ε-greedy
It has the probability ε to select a random action, and a
probability 1 - ε to select the optimal action (which has the
highest Q-value)

20. Meta-Level Reasoning

21. Meta-Level Reasoning

22. StarCraft:
Brood War

23. StarCraft
●
Real-Time Strategy
(RTS)
●
Resource Management
●
Battle!

24. StarCraft

25. StarCraft

26. META-LEVEL
REINFORCEMENT
LEARNING

27. Meta-Level Reasoning

28. Meta-Level Reasoning
Reinforcement
Learning

29. Meta-Level Learning

30. Meta-Level Reasoning
ε-greedy

31. Meta-Level Action Selection
ε-greedy modified:
Instead of keeping the exploitation-rate (ε-value) constant, we apply the
same meta-level reasoning to the ε-value, increasing the exploration rate,
whenever we find that the agent must increase its learning-rate.
The more the agent wants to learn, the more it wants to
explore; if there is nothing to learn, there is nothing to
explore.
ε=α

32. Experiments
in StarCraft

33. Code Injection at StarCraft
● BWAPI
○ Framework in C++ to inject code at
StarCraft;
● BTHAI
○ Bot implemented in BWAPI, with different
high-level strategies;

34. A Reinforcement Learning
Approach for StarCraft
● High-Level Strategy
○ Learning and selection of strategies rather
than micro-actions;
● Learn the end of each match
○ Feedback of victory: +1
○ Feedback of defeat: -1
● Terran vs CPU
○ It was assumed that the agent plays only
as Terran;

35. List of Terran Strategies
of BTHAI
● Marine Rush
● Wraith Harass
● Terran Defensive
● Terran Defensive FB
● Terran Push

36. Results

41. Source Code
BWAPI & StarCraft
https://github.com/jieverson/BTHAIMOD

43. Thank You
contact@jieveron.com
Porto Alegre, 18 de outubro de 2013