3. What we do
• Casual social games
• Casual: simpler, easier, nicer, cuter
• Social: interact with your friends
• Average title: 1 000 000 players every day
tirsdag 4. desember 12
4. Purpose of backend
• Play anywhere, HTTP
• Cheat prevention
• Ensure state integrity
• Highscores
• Multiplayer
• Payments
tirsdag 4. desember 12
6. The first backend: What
• Stateless: state only in DB, mutate in app
• Linux, on EC2
• Apache
• MySQL
• Ruby on Rails
tirsdag 4. desember 12
7. The first backend: Challenges
• MySQL becomes a bottleneck
• Sharding necessary
• Working data set must fit in memory
• Slow restores
tirsdag 4. desember 12
8. The first backend: Challenges
app db
10+ db reqs per HTTP req
tirsdag 4. desember 12
9. The first backend: Challenges
• Very sensitive to network latency, synchronous IO
• 16 ruby processes per node * 125 nodes = 2000
concurrent reqs
• 10K HTTP rps / 2000 workers = 5 reqs/worker/second
• Each request must be served in 1000ms / 5 = 200 ms
tirsdag 4. desember 12
10. The first backend: Solutions
Challenge Solution
MySQL bottleneck Shard, tune, use Redis
Sensitive to latency Redis, more app servers
~200 nodes Chef, Scalarium
tirsdag 4. desember 12
11. The first backend: Conclusions
• Pain to operate massive DB cluster
• Must read & write on master
• Many single points of failure
• Inefficient
• All data in memory
tirsdag 4. desember 12
13. The second backend: What
• Evolution
• Dedicated hardware and network
• Ruby on Rails
• Redis only
• Fewer and faster DB reqs
tirsdag 4. desember 12
14. The second backend:
Conclusions
• Good progress
• Still sensitive to latency
• Single points of failure
• Can we do better?
tirsdag 4. desember 12
16. The third backend: What
• Revolution
• Stateful: app stores state while user is
online, flush to disk when user goes offline
• Only used data in memory
• Not sensitive to latency
• Use Database-as-a-service
tirsdag 4. desember 12
17. The third backend: Challenges
• Erlang
• State
• Resource locking
• Load balancing
• Deployment
• Lack of libraries
tirsdag 4. desember 12
18. The third backend: Challenges
State
• Data structures
• Functional language
• Immutability
• Transactions
tirsdag 4. desember 12
19. The third backend: Challenges
Resource locking
• Only one process per user
• Central serialization
• Redis, CAS, SETNX
• Single point of failure
tirsdag 4. desember 12
20. The third backend: Challenges
Load balancing
• Users play for 3-5 minutes
• “proxy” forwards requests
• “worker” notifies “proxy” about health
tirsdag 4. desember 12
21. The third backend: Challenges
Deployment
• Use Erlang code reload
• “git pull && make && ./upgrade.sh”
• State migration on the fly
• Rolling upgrades
tirsdag 4. desember 12
22. The third backend: Challenges
Lack of libraries
• No CPAN, PiP, RubyGems
• Roll our own
• Eredis, github.com/wooga/eredis
tirsdag 4. desember 12
23. The third backend: Conclusions
First backend Second backend Third backend
Frequent downtime Little downtime No downtime
150+ servers 10+ servers 3 servers
100k DB reqs 10k DB reqs 1k DB reqs
:( :| :)
tirsdag 4. desember 12
25. The fourth backend: What
• Evolution
• Multiplayer!
• Real-time push updates
• Improve operations
• Use hosted databases
• Remove SPOF
tirsdag 4. desember 12
26. The fourth backend: Challenges
Multiplayer
• User and world as Erlang processes
• Serialization
• Reason about concurrency
• Scale
tirsdag 4. desember 12
27. The fourth backend: Challenges
Real-time push updates
• Push updates to client
• Need browser support
• “Transfer-Encoding: chunked”
• Elli, github.com/knutin/elli
tirsdag 4. desember 12
28. The fourth backend: Challenges
Improve operations
• Better load balancing
• Nodes broadcasts “alive” message
• Local failure detector
• Divergent cluster views
tirsdag 4. desember 12
29. The fourth backend: Challenges
Improve operations, cont.
• Collect more metrics
• ETS
• Histograms
• github.com/knutin/statman
tirsdag 4. desember 12
34. The future backends
• Erlang: processes, good IO, clustering
• Harder, better games
• Faster, stronger backends
• Stateful, also with JVM
• Another revolution?
tirsdag 4. desember 12
35. Conclusions
• Understand the past
• The future is stateful
• Identify, think, do
tirsdag 4. desember 12