Online games pose a few interesting challenges on their backend: A single user generates one http call every few seconds and the balance between data read and write is close to 50/50 which makes the use of a write through cache or other common scaling approaches less effective.
Starting from a rather classic rails application as the traffic grew we gradually changed it in order to meet the required performance. And when small changes no longer were enough we turned inside out parts of our data persistency layer migrating from SQL to NoSQL without taking downtimes longer than a few minutes.
Follow the problems we hit, how we diagnosed them, and how we got around limitations. See which tools we found useful and which other lessons we learned by running the system with a team of just two developers without a sysadmin or operation team as support.
6. Social games need to scale quite a bit
1 million daily users
200 million daily HTTP requests
7. Social games need to scale quite a bit
1 million daily users
200 million daily HTTP requests
100,000 DB operaGons per second
40,000 DB updates per second
8. Social games need to scale quite a bit
1 million daily users
200 million daily HTTP requests
100,000 DB operaGons per second
40,000 DB updates per second
… but no cache
9. A journey to 1,000,000 daily users
Start of the journey
6 weeks of pain
Paradise
Looking back
10. We used two MySQL shards right from the start
data_fabric gem
Easy to use
Sharding by Facebook’s user id
Sharding on controller level
11. Scaling applicaLon servers was very easy
Running at Amazon EC2
Scalarium for automaGon and deployment
Role based
Chef recipes for customiza5on
Scaling app servers up and out was simple
21. MySQL hiccups were becoming more frequent
Everything was running fine ...
... and then DB throughput went down to 10%
A]er a few minutes everything stabilizes again ...
... just to repeat the cycle 20 minutes later!
22. AcLveRecord’s checks caused 20% extra DB load
Check connecGon state before each SQL query
MySQL process list full of ‘status’ calls
23. AcLveRecord’s status checks caused 20% extra DB
Check connecGon state before each SQL query
MySQL process list full of ‘status’ calls
24. I/O on MySQL masters sLll was the boYleneck
New Relic shows DB table usage
60% of all UPDATEs were done on the ‘Gles’ table
26. Tiles are part of the core game loop
Core game loop:
1) plant
2) wait
3) harvest
All are
operaGons
on Tiles table.
27. We started to shard on model, too
We put this table in 2 extra shards
old old
master slave
28. We started to shard on model, too
We put this table in 2 extra shards
1) Setup new masters as slaves of old ones
old old new
master slave master
29. We started to shard on model, too
We put this table in 2 extra shards
1) Setup new masters as slaves of old ones
old old new new
master slave master slave
30. We started to shard on model, too
We put this table in 2 extra shards
1) Setup new masters as slaves of old ones
2) App servers start using new masters, too
old old new new
master slave master slave
31. We started to shard on model, too
We put this table in 2 extra shards
1) Setup new masters as slaves of old ones
2) App servers start using new masters, too
3) Cut replica5on
old old new new
master slave master slave
32. We started to shard on model, too
We put this table in 2 extra shards
1) Setup new masters as slave of old ones
2) App servers start using new masters, too
3) Cut replica5on
4) Truncate not-‐used tables
old old new new
master slave master slave
33. 8 DBs and a few more servers
lb
app app app app app app app app
app app app app app app app app
5les 5les
db db
db db
db db 5les 5les
slave slave slave slave
35. We improved our MySQL setup
RAID-‐0 of EBS volumes
Using XtraDB instead of vanilla MySQL
Tweaking my.cnf
innodb_flush_log_at_trx_commit
innodb_flush_method
38. 2 + 2 masters and sLll I/O was not fast enough
We were geing desperate:
“If 2 + 2 is not enough, ...
… perhaps 4 + 4 masters will do?”
39. It’s no fun to handle 16 MySQL DBs
lb
app app app app app app app app app
app app app app app app app app app
5les 5les
db db
db db
db db 5les 5les
slave slave slave slave
40. It’s no fun to handle 16 MySQL DBs
lb
app app app app app app app app app
app app app app app app app app app
5les 5les 5les 5les
db db db db
db db db db
db db db db 5les 5les 5les 5les
slave slave slave slave slave slave slave slave
41. We were at a dead end with MySQL
May Jun Jul Aug Sep Oct Nov Dec
42. I/O remained the boYleneck for MySQL UPDATEs
Peak throughput overall: 5,000 writes/s
Peak throughput single master: 850 writes/s
We could get to ~1,000 writes/s …
… but then what?
44. Redis is fast but goes beyond simple key/value
Redis is a key-‐value store
Hashes, Sets, Sorted Sets, Lists
Atomic opera5ons like set, get, increment
50,000 transacGons/s on EC2
Writes are as fast as reads
45. Shelf Lles : An ideal candidate for using Redis
Shelf Lles:
{ plant1 => 184,
plant2 => 141,
plant3 => 130,
plant4 => 112,
… }
46. Shelf Lles : An ideal candidate for using Redis
Redis Hash
HGETALL: load whole hash
HGET: load a single key
HSET: set a single key
HINCRBY: increase value of single key
…
48. Migrate on the fly -‐ but only once
true if id could be added
else false
49. Migrate on the fly -‐ and clean up later
1. Let this running everything cools down
2. Then migrate the rest manually
3. Remove the migraGon code
4. Wait unGl no fallback necessary
5. Remove the SQL table
50. MigraLons on the fly all look the same
Typical migraGon throughput over 3 days
Ini5al peak at 100 migra5ons / second
51. A journey to 1,000,000 daily users
Start of the journey
6 weeks of pain
Paredise (or not?)
Looking back
52. Again: Tiles are part of the core game loop
Core game loop:
1) plant
2) wait
3) harvest
All are
operaGons
on Tiles table.
53. Size maYers for migraLons
MigraGon check overloaded Redis
Migra5on only on startup
We overlooked an edge case
Next 5me only migrate 1% of users
Migrate the rest when everything worked out
54. In-‐memory DBs don’t like to saving to disk
You sGll need to write data to disk eventually
SAVE is blocking
BGSAVE needs free RAM
Dumping on master increased latency by 100%
55. In-‐memory DBs don’t like to dump to disk
You sGll need to write data to disk eventually
SAVE is blocking
BGSAVE needs free RAM
Dumping on master increased latency by 100%
Running BGSAVE slaves every 15 minutes
56. ReplicaLon puts Redis master under load
ReplicaGon on master starts with a BGSAVE
Not enough free RAM => big problem
Master queue requests that slave cannot process
Restoring dump on slaves is blocking
57. Redis has a memory fragmenLon problem
44 GB
in 8 days
24 GB
58. Redis has a memory fragmenLon problem
38 GB
in 3 days
24 GB
59. If MySQL is a truck...
Fast enough for reads
Can store on disk
Robust replicaGon
hp://www.flickr.com/photos/erix/245657047/
60. If MySQL is a truck, Redis is a Ferrari
Fast enough for reads Super fast reads/writes
Can store on disk Out of memory => dead
Robust replicaGon Fragile replicaGon
hp://www.flickr.com/photos/erix/245657047/
61. Big and staLc data in MySQL, rest goes to Redis
Fast enough for reads Super fast reads/writes
Can store on disk Out of memory => dead
Robust replicaGon Fragile replicaGon
256 GB data 60 GB data
10% writes 50% writes
hp://www.flickr.com/photos/erix/245657047/
62. 95% of operaLon effort is handling DBs!
lb lb
app app app app app app app app app app app app app
app app app app app app app app app app app app app
app app app app app app app app app app app app app
db db db db db redis redis redis redis redis
db db db db db redis redis redis redis redis
slave slave slave slave slave slave slave slave slave slave
63. We fixed all our problem -‐ we were in Paradise!
May Jun Jul Aug Sep Oct Nov Dec
64. A journey to 1,000,000 daily users
Start of the journey
6 weeks of pain
Paredise (or not?)
Looking back
65. Data handling is most important
How much data will you have?
How o]en will you read/update that data?
What data is in the client, what in the server?
It’s hard to “refactor” data later on
66. Monitor and measure your applicaLon closely
On applicaGon level / on OS level
Learn your app’s normal behavior
Store historical data so you can compare later
React if necessary!
67. Listen closely and answer truthfully
Q & A
Jesper Richter-‐Reichhelm
@jrirei slideshare.net/wooga
68. If you are in Berlin just drop by
Thank you
Jesper Richter-‐Reichhelm
@jrirei slideshare.net/wooga