Talk on Akka 2.0, load testing of MMO games, and performance analysis/monitoring of actor-based systems given at scaladay#2 (scaladev.ru)

1. a million bots can't be wrong
@remeniuk, Viaden Media
#ScalaSBP, 18-05-2012

2. In Viaden our aim is to make the best poker ever

3. we know that
performance tests should be
the first-class citizens

4. and kill 2 birds with one stone,
using bots for testing

5.
#1 we can emulate 50k
players using just one
medium EC2 instance
#2 bots are interactive,
so client teams can use
them in development,
and QA for testing

6. everyone knows Akka, huh?

7. why Scala and Akka is a perfect
choice for making bots?
actors are !(interactive)
straightforward remoting
simple scalability/clustering
~30 minutes to make a DSL and CLI
with Scala and SBT

8. ..., but, I have to warn you ...

9. 4 dead simple tips
for keeping your sanity
when you do asynch with Akka 2.0

10. tip #1: live fast, die young

11. typical approach in Akka 1.x
lobby
desk
login
l ink
bot
tourney

12. Akka 1.x: actors have a long lifecycle
lobby
desk
l ink nk
bot li
un
l ink
play game
bot
tourney

13. Akka 1.x: actors have a long lifecycle
lobby
desk
bot
k
lin
bot
un
tourney
play tournament link
bot

14. in Akka 2.0 the world has changed
props
paths
new
supervision
actors

15. now, you're forced to do "the right thing" (c)
lobby
desk
tournament
DeskBot
desk
DeskBot
IdleBot

16. all actors are supervised
lobby
login
desk

17. easy come
lobby
desk
play game
IdleBot

18. easy go (when supervisor to be changed)
lobby
desk
IdleBot dies
DeskBot
borns

19. class Lobby extends Actor {
case Login(username, password) =>
context.actorOf(Props(new IdlePokerBot(...)))
case JoinAnyDesk(props) => findDesk ! JoinDesk(props)
}
class Desk extends Actor {
case JoinDesk(botProps)=>
context.actorOf(Props(new DeskPokerBot(botProps)))
}
class IdlePokerBot(props: BotProps) extends Actor {
case PlayNow =>
context.parent ! JoinAnyDesk(props); context.stop(self)
}

20. Props Pattern - "the soul" of an actor
IdleBot
BotProps
Props remains alive
between actor
"reincarnations"
DeskBot
BotProps

21. case class BotProperties(id: Long,
login: String,
script: Seq[BotEvent],
aggressiveness: Int,
sessionId: Protocol.SessionId)
class IdlePokerBot(val botProperties: BotProperties)
extends Bot[Poker]
class DeskPokerBot(val botProperties: BotProperties)
extends Bot[Poker]

22. tip #2: think beyond

23. when you know, who's supervising, life's
simple
akka://gpdev/user/lobby/player1234
akka://gpdev/user/lobby/desk1/player1234
akka://gpdev/user/lobby/tournament1/desk1/
player1234

24. Bad news
ActorRegistry, actor UUID
were removed from Akka
but what should I do, now,
when I don't know, where
to look for my bot?

25. you can make your own registry
(using Extensions, backed with a
distributed data structure)...

26. or, use the full power of location transparency
Projection
lobby Manager
Projection
desk var location:
ActorPath
DeskBot
/projection/player123
IdleBot /lobby/desk123/player123

27. class Projection(var container: ActorPath) extends Actor {
def receive = {
case newContainer: ActorPath => container = newContainer
case msg =>
context.actorFor(container.child(self.path.name)) !
msg
}
}
class ProjectionManager extends Actor {
def receive = {
case Add(actor) => context.actorOf(Props(new
Projection(actor.path.parent)), actor.path.name)
case Forward(msg, path) => context.actorFor(path) ! msg
}
}
projectionManager ! Add(actorRef)
projectionManager ! Forward("ping", "actor1")

29. class Projection(var container: ActorPath) extends Actor {
def receive = {
case newContainer: ActorPath => container = newContainer
case msg =>
context.actorFor(container.child(self.path.name)) ! msg
}
}
class ProjectionManager extends Actor {
def receive = {
case Add(actor) => context.actorOf(Props(new
Projection(actor.path.parent)), actor.path.name)
case Forward(msg, path) => context.actorFor(path) ! msg
}
}
projectionManager ! Add(actorRef)
system.actorFor(projectionManager.path.child("actor" + i)) !
"ping"

31. class ProjectionManager extends Actor {
def receive = {
case Add(actor) => context.actorOf(Props(new
Projection(actor.path.parent)), actor.path.name)
case Forward(msg, path) =>
context.actorSelection("../*/" + path) ! msg
}
}
val projectionManager = system.actorOf(Props
[ProjectionManagerRoutee]
.withRouter(RoundRobinRouter(resizer = Some
(DefaultResizer(lowerBound = 10, upperBound = 20)))),
"projection")
projectionManager ! Add(actorRef)
projectionManager ! Forward("ping", "actor1")

33. case class CustomRouter(n: Int, routerDispatcher: String =
DefaultDispatcherId, supervisorStrategy: SupervisorStrategy =
defaultStrategy) extends RouterConfig {
def createRoute(props: Props, provider: RouteeProvider) = {
provider.registerRoutees((1 to n).map(i =>
provider.context.actorOf(Props[ProjectionManager], i.
toString)))
def destination(sender: ActorRef, path: String) =
List(Destination(sender,
provider.routees(abs(path.hashCode) %
n)))
{
case m@(sender, Add(actor)) ⇒
destination(sender, actor.path.name)
case m@(sender, Forward(_, name)) ⇒
destination(sender, name)
}
}
}

35. tip #3: don't do anything stupid

36. you've tried all the options, system load is fine,
only 1/10 of the heap is used, but you still can
start not more than 1k bots!?

37. ulimit -n <proper value>

38. your actor is lacking of throughput?
wait before adding pools
share responsibility!
one fine-grained actor is enough in 99% of the cases

39. 100-300 threads are serving 300 bots!?

40. spawn futures, backed with standalone
[bounded] pools, for blocking operations
class ThirdPartyWrapper extends Actor {
case F(x) => sender ! thirdPartyService.f(x)
// call to a function that takes a lot of time to
// complete
}
class ThirdPartyWrapper extends Actor {
case F(x) => val _sender = sender
Future(thirdPartyService.f(x)).map(_sender ! _)
// ugly, but safe, and perfectly right
}

41. use separate dispatchers
lobby-dispatcher projection-manager-dispatcher
PinnedDispatcher BalancingDispatcher
lobby Projection
Manager
desk
Projection
DeskBot
projection-dispatcher
container-dispatcher desk-bot-dispatcher Dispatcher
Dispatcher Dispatcher

42. GOTCHA: Akka successfully bootstraps, even if your
dispatcher is not configured, or the config is wrong
Always check the logs to make sure that dispatchers are used!
[WARN][gpdev-akka.actor.default-dispatcher-1] [Dispatchers]
Dispatcher [bot-system.container-dispatcher] not
configured, using default-dispatcher
[WARN][gpdev-bot-system.container-dispatcher-1]
[PinnedDispatcherConfigurator] PinnedDispatcher [bot-
system.lobby-dispatcher] not configured to use
ThreadPoolExecutor, falling back to default config.
[DEBUG][gpdev-akka.actor.default-dispatcher-24] [akka:
//gpdev/user/poker/lobby] logged in
[DEBUG][gpdev-akka.actor.default-dispatcher-14] [akka:
//gpdev/user/poker/projeciton/$g/player20013] starting
projection...

43. tip #4: analyze that

44. how to measure?
Metrics - pushes various collected metrics to Graphite
Carbon and Graphite - gather metrics, and expose them via web
interface

45. object BotMetrics {
1. val loggedInCount = new Counter(Metrics.newCounter(classOf[Lobby
[_]],
"logged-in-count"))
3. GraphiteReporter.enable(1, TimeUnit.MINUTES, "localhost", 2003)
}
class Lobby extends Actor {
2. case Login(username, pass) => BotMetrics.loggedInCount += 1
}
1. add logged-in user counter 4.
2. update it
3. enable reporting to
Graphite
4. build a graph in Grtaphite

46. what to measure?
- mailbox size1
- throughput
- time, before the message is processed (both in
actor and future)2
- time to process a message
- count of threads
- actor pool size
- heap size
1
requires implementation of a custom mailbox that can expose mailbox size
2
every message should be stuffed with a timestamp

47. how to tune dispatchers?
VisualVM - thread timeline shows, if thread polls behind dispatchers
are used effectively

48. don't neglect old good logging
[ERROR][05/06/2012 12:55:43.826] [gpdev-bot-system.
desk-bot-dispatcher-7]
[akka://gpdev/user/
poker/lobby/tournament5382577/desk109129/player2012
1]
unprocessed game event: GameEvent(CHAT,None,None)

49. thanks for listening

50. we're hiring!
viaden.com/careers/vacancies.html