Games for the Masses: Scaling Rails to the Extreme
•
17 gefällt mir•1,408 views
This presentation explains and compares the work of two engineering teams that build Facebook game backends at wooga: Cloud vs. dedicated servers, SQL vs. NoSQL and in-memory database vs. a database hierarchy. He will highlight their respective advantages and disadvantages and discuss some common patterns both teams came up with to solve their problems.
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ähnlich wie Games for the Masses: Scaling Rails to the Extreme
Ähnlich wie Games for the Masses: Scaling Rails to the Extreme (20)
Mehr von Wooga
Mehr von Wooga (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Games for the Masses: Scaling Rails to the Extreme
- 1. GAMES FOR THE MASSES SCALING RAILS TO THE EXTREME EuRuKo 2011 in Berlin
- 2. Who is that guy? Jesper Richter-‐Reichhelm @jrirei Berlin, Germany Head of Engineering @ wooga
- 3. Wooga develops social games
- 4. Wooga has dedicated game teams PHP Ruby Ruby SQL SQL + Redis Redis Cloud Cloud Bare Metal
- 5. Wooga has dedicated game teams PHP Ruby Ruby SQL SQL + Redis Redis Cloud Cloud Bare Metal
- 6. Both games are similar in their setup Flash client Ruby backend
- 7. Both games have similar problems High traffic • 200+ million h=p requests / day High DB throughput • 100.000 DB operaDons / second Hardest problems to solve • DB migraDons, DB failover, DB performance, ...
- 8. What’s best? Big cloud vs. bare metal Hybrid (SQL + Redis) vs. Redis only In-‐memory vs. DB hierarchy Conclusion: So what’s best?
- 9. Big cloud vs. bare metal Big cloud: Amazon EC2 • new server in 5 minutes • automated deployment • load based setup
- 10. Big cloud vs. bare metal Big cloud: Amazon EC2 Bare metal: Hetzner • new server in 5 minutes • new server in 2-‐14 days • automated deployment • automated deployment • load based setup
- 11. What kind of app server are available? XL CPU instance • 8 virtual cores • 8 GB RAM
- 12. Bare metal servers are more powerful XL CPU instance EQ10 server • 8 virtual cores • 12 “processing threads” • 8 GB RAM • 24 GB RAM
- 13. Bare metal servers are more powerful XL CPU instance EQ10 server • 8 virtual cores • 12 “processing threads” • 8 GB RAM • 24 GB RAM => 390,-‐ € / month => 200,-‐ € / month
- 14. And how is the network? min ping: 0.2 -‐ 0.3 ms min ping: 0.2 ms
- 15. Want a fast network? Get your own! min ping: 0.2 -‐ 0.3 ms min ping: 0.2 ms max ping: 7.8 -‐ 59 ms max ping: 0.3 ms avg ping: 0.8 -‐ 4.4 ms avg ping: 0.2 ms
- 16. Want a fast network? Get your own! min ping: 0.2 -‐ 0.3 ms min ping: 0.2 ms max ping: 7.8 -‐ 59 ms max ping: 0.3 ms avg ping: 0.8 -‐ 4.4 ms avg ping: 0.2 ms 22 DB calls per hfp calls saved 13 -‐ 92 ms latency
- 17. What’s befer? Depends on your needs! Big cloud is great if you • don’t know your load • scale up / down quickly
- 18. What’s befer? Depends on your needs! Big cloud is great if you Bare metal is great if you • don’t know your load • want to save money • scale up / down quickly • need stable performance
- 19. What’s befer? Depends on your needs! Big cloud is great if you Bare metal is great if you • don’t know your load • want to save money • scale up / down quickly • need stable performance Say: “Don’t worry” Say: “Think ahead”
- 20. What’s best? Big cloud vs. bare metal Hybrid (SQL + Redis) vs. Redis only In-‐memory vs. DB hierarchy Conclusion: So what’s best?
- 21. Hybrid (SQL + Redis) vs. Redis Hybrid (SQL + Redis) Redis only • started with MySQL • hit a wall due to IO limits • introduced Redis later
- 22. If MySQL is a truck, ... Fast enough for reads Can store on disk Robust replicakon h=p://www.flickr.com/photos/erix/245657047/
- 23. If MySQL is a truck, Redis is a Ferrari Fast enough for reads Super fast reads/writes Can store on disk Out of memory => dead Robust replicakon Fragile replicakon h=p://www.flickr.com/photos/erix/245657047/
- 24. Hybrid (SQL + Redis) vs. Redis Hybrid (SQL + Redis) Redis only • started with MySQL • hit a wall due to IO limits • introduced Redis later
- 25. Hybrid (SQL + Redis) vs. Redis Hybrid (SQL + Redis) Redis only • started with MySQL • hit a wall due to IO limits • introduced Redis later MySQL has its hard limits Works fine if done right
- 26. How many tables to store a user in? 22 MySQL tables 7 Redis hashes
- 27. How many tables to store a user in? 22 MySQL tables 1 Redis Hash 7 Redis hashes
- 28. How many tables to store a user in? 22 MySQL tables 1 Redis Hash 7 Redis hashes Easy to implement Less top level keys Faster for small changes Export and import is easy
- 29. What’s best? Big cloud vs. bare metal Hybrid (SQL + Redis) vs. Redis only In-‐memory vs. DB hierarchy Conclusion: So what’s best?
- 30. Redis storage type Redis works in-‐memory Dumps slow down master • Dumping on slaves only
- 31. Redis storage type Redis works in-‐memory Redis used vm Dumps slow down master Dumps to disk impossible • Dumping on slaves only • Own system to store JSON files to disk
- 32. Redis storage type Redis works in-‐memory Redis used vm Dumps slow down master Dumps to disk impossible • Dumping on slaves only • Own system to store JSON files to disk Redis became a cache only
- 33. Redis storage type Redis works in-‐memory Redis now uses diskstore Dumps slow down master • Dumping on slaves only
- 34. Redis storage type Redis works in-‐memory Redis now uses diskstore Dumps slow down master • Experimental branch • Dumping on slaves only • Stores async. to disk
- 35. Redis storage type Redis works in-‐memory Redis now uses diskstore Dumps slow down master • Experimental branch • Dumping on slaves only • Stores async. to disk Redis became a DB again
- 36. There are many inackve users hourly 0,3M hourly 0,1M daily 1,3M daily 0,4M monthly 8,0M monthly 3,0M lifetime 20,8M lifetime 6,0M 0M 8M 16M 24M 0M 2M 4M 6M
- 37. Not all users play at the same day In-‐memory DB hierarchy • Redis in-‐memory • Redis diskstore on disk 100% of users in RAM 3% of users in RAM
- 38. What’s best? Big cloud vs. bare metal Hybrid (SQL + Redis) vs. Redis only In-‐memory vs. DB hierarchy Conclusion: So what’s best?
- 39. So, what didn’t work? Big cloud Bare metal Hybrid (SQL + Redis) Redis only In-‐memory “only” DB hierarchy
- 40. So, what didn’t work? Big cloud Bare metal Hybrid (SQL + Redis) Redis only In-‐memory “only” DB hierarchy
- 41. Let’s dig a bit deeper 1,300,000 daily users 400,000 daily users 5000 hfp calls/s peak 2500 hfp calls/s peak 100 ms response kme 10 ms response kme 35-‐70 servers 11 servers
- 42. Hoskng costs / DAU compared HosDng 5x higher
- 43. There are 2 main reasons for higher cost Databases Other Other: 3x higher DB: 20x higher
- 44. There are 2 main reasons for higher cost Databases Other Other: 3x higher No DB hierarchy! DB: 20x higher (both RAM and cost)
- 45. So, what didn’t work? Big cloud Bare metal Hybrid (SQL + Redis) Redis only In-‐memory “only” DB hierarchy
- 46. So, what didn’t work? ds ... Big cloud dep en Bare metal Hybrid (SQL + Redis) Redis only In-‐memory “only” DB hierarchy
- 47. So, what didn’t work? ds ... Big cloud dep en Bare metal Hybrid (SQL + Redis) Redis only nev er In-‐memory “only” in! DB hierarchy a ga
- 48. So, what didn’t work? ds ... Big cloud dep en Bare metal Hybrid (SQL + Redis) er, Redis only er, ev In-‐memory “only” n ev n! DB hierarchy ag ai ev er
- 49. Next game will do things befer Start in cloud and then go metal? Redis only DB hierarchy
- 50. BTW, wooga is hiring! Q & A Jesper Richter-‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobs
- 51. BTW, wooga is hiring! Thank you Jesper Richter-‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobs