Online evolution for multi action adversarial games

1. Online Evolution for Multi-Action
Adversarial Games
Niels Justesen IT University of Copenhagen
Tobias Mahlmann Lund University
Julian Togelius New York University

2. Multi-action games
Sid Meier's Civilizations Heroes of Might and Magic Advance Wars

3. Hero Academy
https://github.com/njustesen/hero-aicademy

4. ● 5 action points each turn
○ Movement ■
○ Healing ■
○ Attacking ■
○ Equipping ■
○ Swapping ■
● Branching factor:
○ One action: ~60
○ One turn: 605
= 7.78 × 108
= 778,000,000
Hero Academy

5. Hand-written heuristic
(b) Bonus added to units on special squares.(a) Bonus added to units with items.

6. Search algorithms in Hero Academy
● 1-ply search
○ Greedy on action-level
● 5-ply (1 turn) depth-first search
○ ~500,000 unique outcomes evaluated each turn (6 seconds)
○ Action pruning and sorting
○ Similar to MiniMax search depth-limited to 5 plies
○ Greedy on turn-level

7. Monte Carlo Tree Search
Chaslot, Guillaume, et al. "Monte-Carlo Tree Search: A New Framework for Game AI." AIIDE. 2008.

8. MCTS in Hero Academy
● No longer an anytime algorithm
● Rollouts have negative effects - use the evaluation function
● ~200 unique outcomes evaluated each turn (6 seconds)
● Side effect of best-first searches in multi-action games: avoids searching the
opponents turn

9. Comparison

10. Online Evolution
Rolling Horizon Evolution
Perez, Diego, et al. "Rolling horizon evolution versus tree search for navigation in single-player real-time games", 2013

11. Online Evolution
● Population size of 100
● 50% elitism
● Random selection of parents
● Uniform crossover
● 10% mutation rate

13. Online Evolution in Hero Academy
● ~10,000 unique outcomes evaluated each turn (6 seconds)
● ~3,500 generations each turn on average

14. Video

15. Future work
● Considering opponent actions
○ Rollouts
○ Competitive co-evolution
● Evolving heuristics
○ Parameter tuning of existing heuristic
○ 1-ply evolution
○ NEAT / Deep Learning
● MCTS variations for Hero Academy
● Online Evolution in multi-action games with more actions

16. Questions?
Email: njustesen@gmail.com
Hero AIcademy: https://github.com/njustesen/hero-aicademy