SlideShare ist ein Scribd-Unternehmen logo
1 von 111
Downloaden Sie, um offline zu lesen
1,000,000 daily users and no cache (Splash 2011)
Who  is  that  guy?

  Jesper  Richter-­‐Reichhelm
  Twi1er:  @jrirei


  Head  of  Engineering
  wooga  
  Berlin,  Germany
wooga  is  #3  game  developer  on  Facebook
Wooga  has  dedicated  game  teams

                                     Cooming
                                       soon
1,000,000 daily users and no cache (Splash 2011)
Flash  client  sends  state  changes  to  backend

       Flash  client               Ruby  backend
Social  games  need  to  scale  quite  a  bit


                400  million  PIs  /  month
Social  games  need  to  scale  quite  a  bit


                400  million  PIs  /  month
Social  games  need  to  scale  quite  a  bit


              14  billion  requests  /  month
Social  games  need  to  scale  quite  a  bit


              14  billion  requests  /  month
Social  games  need  to  scale  quite  a  bit


              14  billion  requests  /  month



           100,000  DB  operaKons  /  second
Social  games  need  to  scale  quite  a  bit


              14  billion  requests  /  month



             50,000  DB  updates  /  second
Social  games  need  to  scale  quite  a  bit


              14  billion  requests  /  month



             50,000  DB  updates  /  second



                        no  cache
A  journey  to  1,000,000  daily  users

  Start  of  the  journey
  6  weeks  of  pain
  Paradise
  Conclusion
October  2009:  wooga’s  first  simulaKon  game
Instead  of  PHP  we  used  Ruby
Our  database  was  MySQL
Our  database  was  MySQL




           even  user  ids   odd  user  ids
And  we  went  into  the  cloud
Master-­‐slave  replicaKon  for  DBs  worked  fine

                          lb




                    app   app   app




                     db         db
We  added  a  few  applicaKon  servers  over  Kme

                                lb




        app   app   app   app   app   app   app   app   app




                          db          db
250K  daily  users  and  no  problems
&$!!!$!!!"




%$#!!$!!!"




%$!!!$!!!"




 #!!$!!!"
                            Life  was  good

        !"
        '()*%!"   +,-*%!"       ./0*%!"       +12*%%"   '()*%%"   +,-*%%"   ./0*%%"
Life  was  well  and  I  went  on  a  nice  vacaKon
                                                       TO  DO

                            <picture:  Jesper  in  clot  
                                    canyon>
1,000,000 daily users and no cache (Splash 2011)
Our  bane:  MySQL  hiccups

 (!!"#


  '!"#


  &!"#


  %!"#


  $!"#


   !"#
         !#   )#   (!#   ()#   $!#   $)#   *!#   *)#   %!#
Our  bane:  MySQL  hiccups

 (!!"#


  '!"#


  &!"#


  %!"#


  $!"#


   !"#
         !#   )#   (!#   ()#   $!#   $)#   *!#   *)#   %!#
Our  bane:  MySQL  hiccups

 (!!"#


  '!"#


  &!"#


  %!"#


  $!"#


   !"#
         !#   )#   (!#   ()#   $!#   $)#   *!#   *)#   %!#
A  journey  to  1,000,000  daily  users

  Start  of  the  journey
  6  weeks  of  pain
  Paradise
  Conclusion
SQL  queries  generated  by  Rubyamf  gem

  AMF  responses  to  Flash  client
SQL  queries  generated  by  Rubyamf  gem

  AMF  responses  to  Flash  client
  Wrong  config...
  ...  so  associated  data  was  included,  too
SQL  queries  generated  by  Rubyamf  gem

  AMF  responses  to  Flash  client
  Wrong  config...
  ...  so  associated  data  was  included,  too


  =>  Easy  to  fix
More  traffic  using  the  same  cluster

                                lb




        app   app   app   app   app   app   app   app   app




                          db          db
Config  tweaks  brought  us  to  300K  DAU
&$!!!$!!!"




%$#!!$!!!"




%$!!!$!!!"




 #!!$!!!"
                            Config  fixes

        !"
        '()*%!"   +,-*%!"      ./0*%!"    +12*%%"   '()*%%"   +,-*%%"   ./0*%%"
AcKveRecord’s  checks  caused  20%  extra  DB  

  Checking  connecKon  state
  MySQL  process  list  full  of  ‘status’  calls
AcKveRecord’s  checks  caused  20%  extra  DB  

  Checking  connecKon  state
  MySQL  process  list  full  of  ‘status’  calls


  =>  Fixed  by  1  line  of  code
I/O  on  MySQL  masters  sKll  was  the  bo^leneck

 New  Relic:  60%  of  all  UPDATEs  on  ‘Kles’  table
Tiles  are  part  of  the  core  game  loop




                                       Core  game  loop
                                       1)  plant
                                       2)  wait
                                       3)  harvest
We  started  to  shard  on  model,  too

  Adding  new  shards




     old           old  
    master        slave
We  started  to  shard  on  model,  too

  Adding  new  shards
    1)  Setup  new  masters  as  slaves  of  old  ones




     old             old            new  
    master          slave          master
We  started  to  shard  on  model,  too

  Adding  new  shards
    1)  Setup  new  masters




     old           old          new       new  
    master        slave        master     slave
We  started  to  shard  on  model,  too

  Adding  new  shards
    1)  Setup  new  masters
    2)  Start  using  new  masters




     old            old               new     new  
    master         slave             master   slave
We  started  to  shard  on  model,  too

  Adding  new  shards
    1)  Setup  new  masters
    2)  Start  using  new  masters
    3)  Cut  replica<on




     old            old               new     new  
    master         slave             master   slave
We  started  to  shard  on  model,  too

  Adding  new  shards
    1)  Setup  new  masters
    2)  Start  using  new  masters
    3)  Cut  replica<on
    4)  Truncate



     old            old               new     new  
    master         slave             master   slave
4  DB  masters  and  a  few  more  servers

                                       lb




          app   app   app        app        app      app    app   app


          app   app   app        app        app      app    app   app



                                                  <les     <les
                 db         db
                                                   db       db
Sharding  by  model  brought  us  to  400K  DAU
&$!!!$!!!"




%$#!!$!!!"




%$!!!$!!!"




 #!!$!!!"
                            Shard  by  model


        !"
        '()*%!"   +,-*%!"    ./0*%!"   +12*%%"   '()*%%"   +,-*%%"   ./0*%%"
We  improved  our  MySQL  setup

  RAID-­‐0  of  EBS  volumes
We  improved  our  MySQL  setup

  RAID-­‐0  of  EBS  volumes


  Using  XtraDB
We  improved  our  MySQL  setup

  RAID-­‐0  of  EBS  volumes


  Using  XtraDB


  Tweaking  my.cnf
Sharding  gem  circumvented  AR’s  internal  cache

  AcKveRecord  caches  SQL  queries...
Sharding  gem  circumvented  AR’s  internal  cache

  AcKveRecord  caches  SQL  queries...


  ...  only  in  our  development  environment!
Sharding  gem  circumvented  AR’s  internal  cache

  AcKveRecord  caches  SQL  queries...


  ...  only  in  our  development  environment!


  =>  Fixed  by  2  lines  of  code
I/O  sKll  was  not  fast  enough

  If  2  +  2  is  not  enough,  ...
I/O  sKll  was  not  fast  enough

  If  2  +  2  is  not  enough,  ...


  …  perhaps  4  +  4  masters  will  do?
It’s  no  fun  to  handle  8+8  MySQL  DBs

                                  lb




         app   app    app   app   app   app    app     app   app


        app    app    app   app   app   app    app     app   app




                                        <les    <les
                     db     db
                                         db      db
It’s  no  fun  to  handle  8+8  MySQL  DBs

                                         lb




           app        app    app   app   app   app    app     app          app


           app        app    app   app   app   app    app     app      app




                                               <les    <les         <les         <les
      db         db         db     db
                                                db      db           db           db
At  500K  DAU  we  were  at  a  dead  end
&$!!!$!!!"




%$#!!$!!!"




%$!!!$!!!"




 #!!$!!!"




        !"
        '()*%!"   +,-*%!"   ./0*%!"   +12*%%"   '()*%%"   +,-*%%"   ./0*%%"
At  500K  DAU  we  were  at  a  dead  end
&$!!!$!!!"




%$#!!$!!!"




%$!!!$!!!"




 #!!$!!!"




        !"
        '()*%!"   +,-*%!"   ./0*%!"   +12*%%"   '()*%%"   +,-*%%"   ./0*%%"
I/O  remained  bo^leneck  for  MySQL  UPDATEs



            Each  DB  master  could  do
             about  1000  DB  write/s.
I/O  remained  bo^leneck  for  MySQL  UPDATEs



            Each  DB  master  could  do
             about  1000  DB  write/s.


               That’s  not  enough!
Pick  the  right  tool  for  the  job!
Redis  is  fast  but  goes  beyond  simple  key/value

  Redis  is  a  key-­‐value  store
    Hashes,  Sets,  Sorted  Sets,  Lists
    Atomic  opera<ons  like  set,  get,  increment
Redis  is  fast  but  goes  beyond  simple  key/value

  Redis  is  a  key-­‐value  store
    Hashes,  Sets,  Sorted  Sets,  Lists
    Atomic  opera<ons  like  set,  get,  increment

  50,000  transacKons/s  on  EC2
     Writes  are  as  fast  as  reads
Wooga  has  dedicated  game  teams
Shelf  Kles  :  An  ideal  candidate  for  using  




                                               Shelf  Kles:
                                           {  plant1  =>  184,
                                            plant2  =>  141,
                                            plant3  =>  130,
                                            plant4  =>  112,
                                                   …  }
Shelf  Kles  :  An  ideal  candidate  for  using  Redis  

   Redis  Hash
     HGETALL
     HGETHSET
     HINCRBY
     …
Migrate  on  the  fly  when  accessing  new  model
Migrate  on  the  fly  -­‐  but  only  once




                   true  if  id  could  be  added
                   else  false
Typical  migraKon  throughput  over  3  days
Migrate  on  the  fly  -­‐  and  clean  up  later

   1. Let  migraKon  run  unKl  everything  cools  down
Migrate  on  the  fly  -­‐  and  clean  up  later

   1. Let  migraKon  run  unKl  everything  cools  down
   2. Migrate  the  rest  manually
Migrate  on  the  fly  -­‐  and  clean  up  later

   1. Let  migraKon  run  unKl  everything  cools  down
   2. Migrate  the  rest  manually
   3. Remove  migraKon  code
Migrate  on  the  fly  -­‐  and  clean  up  later

   1. Let  migraKon  run  unKl  everything  cools  down
   2. Migrate  the  rest  manually
   3. Remove  migraKon  code
   4. Wait  unKl  no  fallback  necessary
Migrate  on  the  fly  -­‐  and  clean  up  later

   1. Let  migraKon  run  unKl  everything  cools  down
   2. Migrate  the  rest  manually
   3. Remove  migraKon  code
   4. Wait  unKl  no  fallback  necessary
   5. Remove  SQL  table
A  journey  to  1,000,000  daily  users

  Start  of  the  journey
  6  weeks  of  pain
  Paredise  (or  not?)
  Conclusion
Again:  Tiles  are  part  of  the  core  game  loop




                                       Core  game  loop
                                       1)  plant
                                       2)  wait
                                       3)  harvest
Size  ma^ers  for  migraKons

  MigraKon  check  overload
    Migra<on  only  on  startup
Size  ma^ers  for  migraKons

  MigraKon  check  overload
    Migra<on  only  on  startup


  Overlooked  an  edge  case
    Only  migrate  1%  of  users
    Con<nue  if  everything  is  ok
In-­‐memory  DBs  don’t  like  to  dump  to  disk

 Dumping  to  disk
   SAVE  is  blocking
   BGSAVE  needs  free  RAM
In-­‐memory  DBs  don’t  like  to  dump  to  disk

 Dumping  to  disk
   SAVE  is  blocking
   BGSAVE  needs  free  RAM

 Latency  increase  by  100%
In-­‐memory  DBs  don’t  like  to  dump  to  disk

 Dumping  to  disk
   SAVE  is  blocking
   BGSAVE  needs  free  RAM

 Latency  increase  by  100%


 =>  BGSAVE  on  slaves  every  15  minutes
Redis  replicaKon  starts  with  a  BGSAVE

 BGSAVE  on  master
 Slave  imports  dumped  file
Redis  replicaKon  starts  with  a  BGSAVE

 BGSAVE  on  master
 Slave  imports  dumped  file


 =>  No  RAM  means  no  new  slaves
Redis  had  a  memory  fragmenKon  problem



                                   44  GB

                                       in  8  days

                                  24  GB
Redis  had  a  memory  fragmenKon  problem




                                  38  GB
                                       in  3  days
                                  24  GB
If  MySQL  is  a  truck
                                                                Fast  enough
                                                                Disk  based
                                                                Robust




       Fast  enough                    disk  based                    robust
If  MySQL  is  a  truck,  Redis  is  a  race  car
                                                                  Super  fast
                                                                  RAM  based
                                                                  Fragile




        Super  fast                    RAM  based                    fragile
Big  and  staKc  data  in  MySQL,  rest  goes  to  Redis




        256  GB  data                               60  GB  data
         10%  writes                                50%  writes
       hSp://www.flickr.com/photos/erix/245657047/
Lots  of  boxes,  but  automaKon  helps  a  lot!

                                           lb         lb


app   app    app        app    app   app        app        app     app      app   app     app   app


app   app    app        app    app   app        app        app     app      app   app     app   app


app   app    app        app    app   app        app        app     app      app   app     app   app



db      db         db         db     db                    redis    redis     redis     redis   redis
We  reached  1  million  daily  users!
&$!!!$!!!"




%$#!!$!!!"




%$!!!$!!!"
                                      1,000,000  -­‐  Big  party!



 #!!$!!!"




        !"
        '()*%!"   +,-*%!"   ./0*%!"      +12*%%"     '()*%%"        +,-*%%"   ./0*%%"
We  started  archiving  inacKve  users
&$!!!$!!!"




%$#!!$!!!"

                                                          50%  DB
%$!!!$!!!"
                                                          reducKon



 #!!$!!!"




        !"
        '()*%!"   +,-*%!"   ./0*%!"   +12*%%"   '()*%%"     +,-*%%"   ./0*%%"
We  even  survived  a  complete  data  center  loss
&$!!!$!!!"



                                                                    EBS  no
%$#!!$!!!"
                                                                    more!


%$!!!$!!!"




 #!!$!!!"




        !"
        '()*%!"   +,-*%!"   ./0*%!"   +12*%%"   '()*%%"   +,-*%%"       ./0*%%"
We  improved  our  MySQL  schema  on-­‐the-­‐fly
&$!!!$!!!"



                                                                    30%  DB
%$#!!$!!!"
                                                                    reducKon


%$!!!$!!!"




 #!!$!!!"




        !"
        '()*%!"   +,-*%!"   ./0*%!"   +12*%%"   '()*%%"   +,-*%%"    ./0*%%"
Will  we  reach  2  million  daily  users?
&$!!!$!!!"




%$#!!$!!!"




%$!!!$!!!"




 #!!$!!!"




        !"
        '()*%!"   +,-*%!"   ./0*%!"   +12*%%"   '()*%%"   +,-*%%"   ./0*%%"
A  journey  to  1,000,000  daily  users

  Start  of  the  journey
  6  weeks  of  pain
  Paredise  (or  not?)
  Conclusion
You  do  not  know  the  future


                     Plan  ahead
You  do  not  know  the  future


                     Plan  ahead



                       Learn
You  do  not  know  the  future


                     Plan  ahead



                       Learn



                       Adapt
EvoluKon  every  week


              EVOLUTION
                                    of  sonware
        &$!!!$!!!"




        %$#!!$!!!"




        %$!!!$!!!"




         #!!$!!!"




                !"
                '()*%!"   +,-*%!"   ./0*%!"   +12*%%"   '()*%%"   +,-*%%"   ./0*%%"
EvoluKon  every  week


              EVOLUTION
                                    of  sonware
        &$!!!$!!!"




        %$#!!$!!!"




        %$!!!$!!!"




         #!!$!!!"




                !"
                '()*%!"   +,-*%!"   ./0*%!"   +12*%%"   '()*%%"   +,-*%%"   ./0*%%"
EvoluKon  every  week


          EVOLUTION
                  of  sonware
EvoluKon  every  week,  RevoluKon  if  necessary


       REVOLUTION
                   of  sonware
EvoluKon  every  week,  RevoluKon  if  necessary


       REVOLUTION
                   of  sonware
EvoluKon  every  week,  RevoluKon  if  necessary


       REVOLUTION
                                     of  sonware
         &$!!!$!!!"




         %$#!!$!!!"




         %$!!!$!!!"




          #!!$!!!"




                 !"
                 '()*%!"   +,-*%!"   ./0*%!"   +12*%%"   '()*%%"   +,-*%%"   ./0*%%"
Each  new  game  is  a  revoluKon
Each  new  game  is  a  revoluKon
Each  new  game  is  a  revoluKon
Each  new  game  is  a  revoluKon
Each  new  game  is  a  revoluKon

                                    Cooming
                                      soon
Works  for  teams  ...
Works  for  teams  and  for  companies




                 !""#$%&"'()"*+,
Thank  you!

Jesper  Richter-­‐Reichhelm
          @jrirei

 slideshare.net/wooga
    wooga.com/jobs

Más contenido relacionado

Andere mochten auch

Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)
Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)
Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)Wooga
 
Games for the Masses: Scaling Rails to the Extreme
Games for the Masses: Scaling Rails to the ExtremeGames for the Masses: Scaling Rails to the Extreme
Games for the Masses: Scaling Rails to the ExtremeWooga
 
Metrics. Driven. Design. (Developer Conference Hamburg 2012)
Metrics. Driven. Design. (Developer Conference Hamburg 2012)Metrics. Driven. Design. (Developer Conference Hamburg 2012)
Metrics. Driven. Design. (Developer Conference Hamburg 2012)Wooga
 
How to scale a company - game teams at Wooga
How to scale a company - game teams at WoogaHow to scale a company - game teams at Wooga
How to scale a company - game teams at WoogaWooga
 
Event Stream Processing with Kafka (Berlin Buzzwords 2012)
Event Stream Processing with Kafka (Berlin Buzzwords 2012)Event Stream Processing with Kafka (Berlin Buzzwords 2012)
Event Stream Processing with Kafka (Berlin Buzzwords 2012)Wooga
 
2013 04-29-evolution of backend
2013 04-29-evolution of backend2013 04-29-evolution of backend
2013 04-29-evolution of backendWooga
 
More than syntax
More than syntaxMore than syntax
More than syntaxWooga
 
You are not alone - Scaling multiplayer games
You are not alone - Scaling multiplayer gamesYou are not alone - Scaling multiplayer games
You are not alone - Scaling multiplayer gamesWooga
 
Stateful_Application_Server_RuPy 2012_Brno
Stateful_Application_Server_RuPy 2012_BrnoStateful_Application_Server_RuPy 2012_Brno
Stateful_Application_Server_RuPy 2012_BrnoWooga
 
Continuous Integration for iOS (iOS User Group Berlin)
Continuous Integration for iOS (iOS User Group Berlin)Continuous Integration for iOS (iOS User Group Berlin)
Continuous Integration for iOS (iOS User Group Berlin)Wooga
 
Painful success - lessons learned while scaling up
Painful success - lessons learned while scaling upPainful success - lessons learned while scaling up
Painful success - lessons learned while scaling upWooga
 
NoSQL Games
NoSQL GamesNoSQL Games
NoSQL GamesWooga
 
Wooga: Internationality meets Agility @Zutaten 2013
Wooga: Internationality meets Agility @Zutaten 2013Wooga: Internationality meets Agility @Zutaten 2013
Wooga: Internationality meets Agility @Zutaten 2013Wooga
 
Monitoring with Syslog and EventMachine
Monitoring with Syslog and EventMachineMonitoring with Syslog and EventMachine
Monitoring with Syslog and EventMachineWooga
 
"Common Sense" instead of "Command and Control"_Start up Camp 2013
"Common Sense" instead of "Command and Control"_Start up Camp 2013"Common Sense" instead of "Command and Control"_Start up Camp 2013
"Common Sense" instead of "Command and Control"_Start up Camp 2013Wooga
 

Andere mochten auch (15)

Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)
Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)
Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)
 
Games for the Masses: Scaling Rails to the Extreme
Games for the Masses: Scaling Rails to the ExtremeGames for the Masses: Scaling Rails to the Extreme
Games for the Masses: Scaling Rails to the Extreme
 
Metrics. Driven. Design. (Developer Conference Hamburg 2012)
Metrics. Driven. Design. (Developer Conference Hamburg 2012)Metrics. Driven. Design. (Developer Conference Hamburg 2012)
Metrics. Driven. Design. (Developer Conference Hamburg 2012)
 
How to scale a company - game teams at Wooga
How to scale a company - game teams at WoogaHow to scale a company - game teams at Wooga
How to scale a company - game teams at Wooga
 
Event Stream Processing with Kafka (Berlin Buzzwords 2012)
Event Stream Processing with Kafka (Berlin Buzzwords 2012)Event Stream Processing with Kafka (Berlin Buzzwords 2012)
Event Stream Processing with Kafka (Berlin Buzzwords 2012)
 
2013 04-29-evolution of backend
2013 04-29-evolution of backend2013 04-29-evolution of backend
2013 04-29-evolution of backend
 
More than syntax
More than syntaxMore than syntax
More than syntax
 
You are not alone - Scaling multiplayer games
You are not alone - Scaling multiplayer gamesYou are not alone - Scaling multiplayer games
You are not alone - Scaling multiplayer games
 
Stateful_Application_Server_RuPy 2012_Brno
Stateful_Application_Server_RuPy 2012_BrnoStateful_Application_Server_RuPy 2012_Brno
Stateful_Application_Server_RuPy 2012_Brno
 
Continuous Integration for iOS (iOS User Group Berlin)
Continuous Integration for iOS (iOS User Group Berlin)Continuous Integration for iOS (iOS User Group Berlin)
Continuous Integration for iOS (iOS User Group Berlin)
 
Painful success - lessons learned while scaling up
Painful success - lessons learned while scaling upPainful success - lessons learned while scaling up
Painful success - lessons learned while scaling up
 
NoSQL Games
NoSQL GamesNoSQL Games
NoSQL Games
 
Wooga: Internationality meets Agility @Zutaten 2013
Wooga: Internationality meets Agility @Zutaten 2013Wooga: Internationality meets Agility @Zutaten 2013
Wooga: Internationality meets Agility @Zutaten 2013
 
Monitoring with Syslog and EventMachine
Monitoring with Syslog and EventMachineMonitoring with Syslog and EventMachine
Monitoring with Syslog and EventMachine
 
"Common Sense" instead of "Command and Control"_Start up Camp 2013
"Common Sense" instead of "Command and Control"_Start up Camp 2013"Common Sense" instead of "Command and Control"_Start up Camp 2013
"Common Sense" instead of "Command and Control"_Start up Camp 2013
 

Mehr von Wooga

Story of Warlords: Bringing a turn-based strategy game to mobile
Story of Warlords: Bringing a turn-based strategy game to mobile Story of Warlords: Bringing a turn-based strategy game to mobile
Story of Warlords: Bringing a turn-based strategy game to mobile Wooga
 
Instagram Celebrities: are they the new cats? - Targetsummit Berlin 2015
Instagram Celebrities: are they the new cats? - Targetsummit Berlin 2015Instagram Celebrities: are they the new cats? - Targetsummit Berlin 2015
Instagram Celebrities: are they the new cats? - Targetsummit Berlin 2015Wooga
 
In it for the long haul - How Wooga boosts long-term retention
In it for the long haul - How Wooga boosts long-term retentionIn it for the long haul - How Wooga boosts long-term retention
In it for the long haul - How Wooga boosts long-term retentionWooga
 
Leveling up in localization! - Susan Alma & Dario Quondamstefano
Leveling up in localization! - Susan Alma & Dario QuondamstefanoLeveling up in localization! - Susan Alma & Dario Quondamstefano
Leveling up in localization! - Susan Alma & Dario QuondamstefanoWooga
 
Evoloution of Ideas
Evoloution of IdeasEvoloution of Ideas
Evoloution of IdeasWooga
 
Entitas System Architecture with Unity - Maxim Zaks and Simon Schmid
Entitas System Architecture with Unity - Maxim Zaks and Simon Schmid Entitas System Architecture with Unity - Maxim Zaks and Simon Schmid
Entitas System Architecture with Unity - Maxim Zaks and Simon Schmid Wooga
 
Saying No to the CEO: A Deep Look at Independent Teams - Adam Telfer
Saying No to the CEO: A Deep Look at Independent Teams - Adam TelferSaying No to the CEO: A Deep Look at Independent Teams - Adam Telfer
Saying No to the CEO: A Deep Look at Independent Teams - Adam TelferWooga
 
Innovation dank DevOps (DevOpsCon Berlin 2015)
Innovation dank DevOps (DevOpsCon Berlin 2015)Innovation dank DevOps (DevOpsCon Berlin 2015)
Innovation dank DevOps (DevOpsCon Berlin 2015)Wooga
 
Big Fish, small pond - strategies for surviving in a maturing market - Ed Biden
Big Fish, small pond - strategies for surviving in a maturing market - Ed BidenBig Fish, small pond - strategies for surviving in a maturing market - Ed Biden
Big Fish, small pond - strategies for surviving in a maturing market - Ed BidenWooga
 
Review mining aps2014 berlin
Review mining aps2014 berlinReview mining aps2014 berlin
Review mining aps2014 berlinWooga
 
Riak & Wooga_Geeek2Geeek Meetup2014 Berlin
Riak & Wooga_Geeek2Geeek Meetup2014 BerlinRiak & Wooga_Geeek2Geeek Meetup2014 Berlin
Riak & Wooga_Geeek2Geeek Meetup2014 BerlinWooga
 
Staying in the Game: Game localization practices for the mobile market
Staying in the Game: Game localization practices for the mobile marketStaying in the Game: Game localization practices for the mobile market
Staying in the Game: Game localization practices for the mobile marketWooga
 
Startup Weekend_Makers and Games_Philipp Stelzer
Startup Weekend_Makers and Games_Philipp StelzerStartup Weekend_Makers and Games_Philipp Stelzer
Startup Weekend_Makers and Games_Philipp StelzerWooga
 
DevOps goes Mobile (daho.am)
DevOps goes Mobile (daho.am)DevOps goes Mobile (daho.am)
DevOps goes Mobile (daho.am)Wooga
 
DevOps goes Mobile - Jax 2014 - Jesper Richter-Reichhelm
DevOps goes Mobile - Jax 2014 - Jesper Richter-ReichhelmDevOps goes Mobile - Jax 2014 - Jesper Richter-Reichhelm
DevOps goes Mobile - Jax 2014 - Jesper Richter-ReichhelmWooga
 
CodeFest 2014_Mobile Game Development
CodeFest 2014_Mobile Game DevelopmentCodeFest 2014_Mobile Game Development
CodeFest 2014_Mobile Game DevelopmentWooga
 
Jelly Splash: Puzzling your way to the top of the App Stores - GDC 2014
Jelly Splash: Puzzling your way to the top of the App Stores - GDC 2014Jelly Splash: Puzzling your way to the top of the App Stores - GDC 2014
Jelly Splash: Puzzling your way to the top of the App Stores - GDC 2014Wooga
 
How to hire the best people for your startup-Gitta Blat-Head of People
How to hire the best people for your startup-Gitta Blat-Head of PeopleHow to hire the best people for your startup-Gitta Blat-Head of People
How to hire the best people for your startup-Gitta Blat-Head of PeopleWooga
 
Two Ann(e)s and one Julia_Wooga Lady Power from Berlin_SGA2014
Two Ann(e)s and one Julia_Wooga Lady Power from Berlin_SGA2014Two Ann(e)s and one Julia_Wooga Lady Power from Berlin_SGA2014
Two Ann(e)s and one Julia_Wooga Lady Power from Berlin_SGA2014Wooga
 
Pocket Gamer Connects 2014_The Experience of Entering the Korean Market
Pocket Gamer Connects 2014_The Experience of Entering the Korean MarketPocket Gamer Connects 2014_The Experience of Entering the Korean Market
Pocket Gamer Connects 2014_The Experience of Entering the Korean MarketWooga
 

Mehr von Wooga (20)

Story of Warlords: Bringing a turn-based strategy game to mobile
Story of Warlords: Bringing a turn-based strategy game to mobile Story of Warlords: Bringing a turn-based strategy game to mobile
Story of Warlords: Bringing a turn-based strategy game to mobile
 
Instagram Celebrities: are they the new cats? - Targetsummit Berlin 2015
Instagram Celebrities: are they the new cats? - Targetsummit Berlin 2015Instagram Celebrities: are they the new cats? - Targetsummit Berlin 2015
Instagram Celebrities: are they the new cats? - Targetsummit Berlin 2015
 
In it for the long haul - How Wooga boosts long-term retention
In it for the long haul - How Wooga boosts long-term retentionIn it for the long haul - How Wooga boosts long-term retention
In it for the long haul - How Wooga boosts long-term retention
 
Leveling up in localization! - Susan Alma & Dario Quondamstefano
Leveling up in localization! - Susan Alma & Dario QuondamstefanoLeveling up in localization! - Susan Alma & Dario Quondamstefano
Leveling up in localization! - Susan Alma & Dario Quondamstefano
 
Evoloution of Ideas
Evoloution of IdeasEvoloution of Ideas
Evoloution of Ideas
 
Entitas System Architecture with Unity - Maxim Zaks and Simon Schmid
Entitas System Architecture with Unity - Maxim Zaks and Simon Schmid Entitas System Architecture with Unity - Maxim Zaks and Simon Schmid
Entitas System Architecture with Unity - Maxim Zaks and Simon Schmid
 
Saying No to the CEO: A Deep Look at Independent Teams - Adam Telfer
Saying No to the CEO: A Deep Look at Independent Teams - Adam TelferSaying No to the CEO: A Deep Look at Independent Teams - Adam Telfer
Saying No to the CEO: A Deep Look at Independent Teams - Adam Telfer
 
Innovation dank DevOps (DevOpsCon Berlin 2015)
Innovation dank DevOps (DevOpsCon Berlin 2015)Innovation dank DevOps (DevOpsCon Berlin 2015)
Innovation dank DevOps (DevOpsCon Berlin 2015)
 
Big Fish, small pond - strategies for surviving in a maturing market - Ed Biden
Big Fish, small pond - strategies for surviving in a maturing market - Ed BidenBig Fish, small pond - strategies for surviving in a maturing market - Ed Biden
Big Fish, small pond - strategies for surviving in a maturing market - Ed Biden
 
Review mining aps2014 berlin
Review mining aps2014 berlinReview mining aps2014 berlin
Review mining aps2014 berlin
 
Riak & Wooga_Geeek2Geeek Meetup2014 Berlin
Riak & Wooga_Geeek2Geeek Meetup2014 BerlinRiak & Wooga_Geeek2Geeek Meetup2014 Berlin
Riak & Wooga_Geeek2Geeek Meetup2014 Berlin
 
Staying in the Game: Game localization practices for the mobile market
Staying in the Game: Game localization practices for the mobile marketStaying in the Game: Game localization practices for the mobile market
Staying in the Game: Game localization practices for the mobile market
 
Startup Weekend_Makers and Games_Philipp Stelzer
Startup Weekend_Makers and Games_Philipp StelzerStartup Weekend_Makers and Games_Philipp Stelzer
Startup Weekend_Makers and Games_Philipp Stelzer
 
DevOps goes Mobile (daho.am)
DevOps goes Mobile (daho.am)DevOps goes Mobile (daho.am)
DevOps goes Mobile (daho.am)
 
DevOps goes Mobile - Jax 2014 - Jesper Richter-Reichhelm
DevOps goes Mobile - Jax 2014 - Jesper Richter-ReichhelmDevOps goes Mobile - Jax 2014 - Jesper Richter-Reichhelm
DevOps goes Mobile - Jax 2014 - Jesper Richter-Reichhelm
 
CodeFest 2014_Mobile Game Development
CodeFest 2014_Mobile Game DevelopmentCodeFest 2014_Mobile Game Development
CodeFest 2014_Mobile Game Development
 
Jelly Splash: Puzzling your way to the top of the App Stores - GDC 2014
Jelly Splash: Puzzling your way to the top of the App Stores - GDC 2014Jelly Splash: Puzzling your way to the top of the App Stores - GDC 2014
Jelly Splash: Puzzling your way to the top of the App Stores - GDC 2014
 
How to hire the best people for your startup-Gitta Blat-Head of People
How to hire the best people for your startup-Gitta Blat-Head of PeopleHow to hire the best people for your startup-Gitta Blat-Head of People
How to hire the best people for your startup-Gitta Blat-Head of People
 
Two Ann(e)s and one Julia_Wooga Lady Power from Berlin_SGA2014
Two Ann(e)s and one Julia_Wooga Lady Power from Berlin_SGA2014Two Ann(e)s and one Julia_Wooga Lady Power from Berlin_SGA2014
Two Ann(e)s and one Julia_Wooga Lady Power from Berlin_SGA2014
 
Pocket Gamer Connects 2014_The Experience of Entering the Korean Market
Pocket Gamer Connects 2014_The Experience of Entering the Korean MarketPocket Gamer Connects 2014_The Experience of Entering the Korean Market
Pocket Gamer Connects 2014_The Experience of Entering the Korean Market
 

Último

How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Libraryshyamraj55
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfTejal81
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInThousandEyes
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxSatishbabu Gunukula
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...DianaGray10
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updateadam112203
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 

Último (20)

How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Library
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
SheDev 2024
SheDev 2024SheDev 2024
SheDev 2024
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptx
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 update
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 

1,000,000 daily users and no cache (Splash 2011)

  • 2. Who  is  that  guy? Jesper  Richter-­‐Reichhelm Twi1er:  @jrirei Head  of  Engineering wooga   Berlin,  Germany
  • 3. wooga  is  #3  game  developer  on  Facebook
  • 4. Wooga  has  dedicated  game  teams Cooming soon
  • 6. Flash  client  sends  state  changes  to  backend Flash  client Ruby  backend
  • 7. Social  games  need  to  scale  quite  a  bit 400  million  PIs  /  month
  • 8. Social  games  need  to  scale  quite  a  bit 400  million  PIs  /  month
  • 9. Social  games  need  to  scale  quite  a  bit 14  billion  requests  /  month
  • 10. Social  games  need  to  scale  quite  a  bit 14  billion  requests  /  month
  • 11. Social  games  need  to  scale  quite  a  bit 14  billion  requests  /  month 100,000  DB  operaKons  /  second
  • 12. Social  games  need  to  scale  quite  a  bit 14  billion  requests  /  month 50,000  DB  updates  /  second
  • 13. Social  games  need  to  scale  quite  a  bit 14  billion  requests  /  month 50,000  DB  updates  /  second no  cache
  • 14. A  journey  to  1,000,000  daily  users Start  of  the  journey 6  weeks  of  pain Paradise Conclusion
  • 15. October  2009:  wooga’s  first  simulaKon  game
  • 16. Instead  of  PHP  we  used  Ruby
  • 18. Our  database  was  MySQL even  user  ids odd  user  ids
  • 19. And  we  went  into  the  cloud
  • 20. Master-­‐slave  replicaKon  for  DBs  worked  fine lb app app app db db
  • 21. We  added  a  few  applicaKon  servers  over  Kme lb app app app app app app app app app db db
  • 22. 250K  daily  users  and  no  problems &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" Life  was  good !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
  • 23. Life  was  well  and  I  went  on  a  nice  vacaKon TO  DO <picture:  Jesper  in  clot   canyon>
  • 25. Our  bane:  MySQL  hiccups (!!"# '!"# &!"# %!"# $!"# !"# !# )# (!# ()# $!# $)# *!# *)# %!#
  • 26. Our  bane:  MySQL  hiccups (!!"# '!"# &!"# %!"# $!"# !"# !# )# (!# ()# $!# $)# *!# *)# %!#
  • 27. Our  bane:  MySQL  hiccups (!!"# '!"# &!"# %!"# $!"# !"# !# )# (!# ()# $!# $)# *!# *)# %!#
  • 28. A  journey  to  1,000,000  daily  users Start  of  the  journey 6  weeks  of  pain Paradise Conclusion
  • 29. SQL  queries  generated  by  Rubyamf  gem AMF  responses  to  Flash  client
  • 30. SQL  queries  generated  by  Rubyamf  gem AMF  responses  to  Flash  client Wrong  config... ...  so  associated  data  was  included,  too
  • 31. SQL  queries  generated  by  Rubyamf  gem AMF  responses  to  Flash  client Wrong  config... ...  so  associated  data  was  included,  too =>  Easy  to  fix
  • 32. More  traffic  using  the  same  cluster lb app app app app app app app app app db db
  • 33. Config  tweaks  brought  us  to  300K  DAU &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" Config  fixes !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
  • 34. AcKveRecord’s  checks  caused  20%  extra  DB   Checking  connecKon  state MySQL  process  list  full  of  ‘status’  calls
  • 35. AcKveRecord’s  checks  caused  20%  extra  DB   Checking  connecKon  state MySQL  process  list  full  of  ‘status’  calls =>  Fixed  by  1  line  of  code
  • 36. I/O  on  MySQL  masters  sKll  was  the  bo^leneck New  Relic:  60%  of  all  UPDATEs  on  ‘Kles’  table
  • 37. Tiles  are  part  of  the  core  game  loop Core  game  loop 1)  plant 2)  wait 3)  harvest
  • 38. We  started  to  shard  on  model,  too Adding  new  shards old   old   master slave
  • 39. We  started  to  shard  on  model,  too Adding  new  shards 1)  Setup  new  masters  as  slaves  of  old  ones old   old   new   master slave master
  • 40. We  started  to  shard  on  model,  too Adding  new  shards 1)  Setup  new  masters old   old   new   new   master slave master slave
  • 41. We  started  to  shard  on  model,  too Adding  new  shards 1)  Setup  new  masters 2)  Start  using  new  masters old   old   new   new   master slave master slave
  • 42. We  started  to  shard  on  model,  too Adding  new  shards 1)  Setup  new  masters 2)  Start  using  new  masters 3)  Cut  replica<on old   old   new   new   master slave master slave
  • 43. We  started  to  shard  on  model,  too Adding  new  shards 1)  Setup  new  masters 2)  Start  using  new  masters 3)  Cut  replica<on 4)  Truncate old   old   new   new   master slave master slave
  • 44. 4  DB  masters  and  a  few  more  servers lb app app app app app app app app app app app app app app app app <les <les db db db db
  • 45. Sharding  by  model  brought  us  to  400K  DAU &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" Shard  by  model !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
  • 46. We  improved  our  MySQL  setup RAID-­‐0  of  EBS  volumes
  • 47. We  improved  our  MySQL  setup RAID-­‐0  of  EBS  volumes Using  XtraDB
  • 48. We  improved  our  MySQL  setup RAID-­‐0  of  EBS  volumes Using  XtraDB Tweaking  my.cnf
  • 49. Sharding  gem  circumvented  AR’s  internal  cache AcKveRecord  caches  SQL  queries...
  • 50. Sharding  gem  circumvented  AR’s  internal  cache AcKveRecord  caches  SQL  queries... ...  only  in  our  development  environment!
  • 51. Sharding  gem  circumvented  AR’s  internal  cache AcKveRecord  caches  SQL  queries... ...  only  in  our  development  environment! =>  Fixed  by  2  lines  of  code
  • 52. I/O  sKll  was  not  fast  enough If  2  +  2  is  not  enough,  ...
  • 53. I/O  sKll  was  not  fast  enough If  2  +  2  is  not  enough,  ... …  perhaps  4  +  4  masters  will  do?
  • 54. It’s  no  fun  to  handle  8+8  MySQL  DBs lb app app app app app app app app app app app app app app app app app app <les <les db db db db
  • 55. It’s  no  fun  to  handle  8+8  MySQL  DBs lb app app app app app app app app app app app app app app app app app app <les <les <les <les db db db db db db db db
  • 56. At  500K  DAU  we  were  at  a  dead  end &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
  • 57. At  500K  DAU  we  were  at  a  dead  end &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
  • 58. I/O  remained  bo^leneck  for  MySQL  UPDATEs Each  DB  master  could  do about  1000  DB  write/s.
  • 59. I/O  remained  bo^leneck  for  MySQL  UPDATEs Each  DB  master  could  do about  1000  DB  write/s. That’s  not  enough!
  • 60. Pick  the  right  tool  for  the  job!
  • 61. Redis  is  fast  but  goes  beyond  simple  key/value Redis  is  a  key-­‐value  store Hashes,  Sets,  Sorted  Sets,  Lists Atomic  opera<ons  like  set,  get,  increment
  • 62. Redis  is  fast  but  goes  beyond  simple  key/value Redis  is  a  key-­‐value  store Hashes,  Sets,  Sorted  Sets,  Lists Atomic  opera<ons  like  set,  get,  increment 50,000  transacKons/s  on  EC2 Writes  are  as  fast  as  reads
  • 63. Wooga  has  dedicated  game  teams
  • 64. Shelf  Kles  :  An  ideal  candidate  for  using   Shelf  Kles: {  plant1  =>  184, plant2  =>  141, plant3  =>  130, plant4  =>  112, …  }
  • 65. Shelf  Kles  :  An  ideal  candidate  for  using  Redis   Redis  Hash HGETALL HGETHSET HINCRBY …
  • 66. Migrate  on  the  fly  when  accessing  new  model
  • 67. Migrate  on  the  fly  -­‐  but  only  once true  if  id  could  be  added else  false
  • 68. Typical  migraKon  throughput  over  3  days
  • 69. Migrate  on  the  fly  -­‐  and  clean  up  later 1. Let  migraKon  run  unKl  everything  cools  down
  • 70. Migrate  on  the  fly  -­‐  and  clean  up  later 1. Let  migraKon  run  unKl  everything  cools  down 2. Migrate  the  rest  manually
  • 71. Migrate  on  the  fly  -­‐  and  clean  up  later 1. Let  migraKon  run  unKl  everything  cools  down 2. Migrate  the  rest  manually 3. Remove  migraKon  code
  • 72. Migrate  on  the  fly  -­‐  and  clean  up  later 1. Let  migraKon  run  unKl  everything  cools  down 2. Migrate  the  rest  manually 3. Remove  migraKon  code 4. Wait  unKl  no  fallback  necessary
  • 73. Migrate  on  the  fly  -­‐  and  clean  up  later 1. Let  migraKon  run  unKl  everything  cools  down 2. Migrate  the  rest  manually 3. Remove  migraKon  code 4. Wait  unKl  no  fallback  necessary 5. Remove  SQL  table
  • 74. A  journey  to  1,000,000  daily  users Start  of  the  journey 6  weeks  of  pain Paredise  (or  not?) Conclusion
  • 75. Again:  Tiles  are  part  of  the  core  game  loop Core  game  loop 1)  plant 2)  wait 3)  harvest
  • 76. Size  ma^ers  for  migraKons MigraKon  check  overload Migra<on  only  on  startup
  • 77. Size  ma^ers  for  migraKons MigraKon  check  overload Migra<on  only  on  startup Overlooked  an  edge  case Only  migrate  1%  of  users Con<nue  if  everything  is  ok
  • 78. In-­‐memory  DBs  don’t  like  to  dump  to  disk Dumping  to  disk SAVE  is  blocking BGSAVE  needs  free  RAM
  • 79. In-­‐memory  DBs  don’t  like  to  dump  to  disk Dumping  to  disk SAVE  is  blocking BGSAVE  needs  free  RAM Latency  increase  by  100%
  • 80. In-­‐memory  DBs  don’t  like  to  dump  to  disk Dumping  to  disk SAVE  is  blocking BGSAVE  needs  free  RAM Latency  increase  by  100% =>  BGSAVE  on  slaves  every  15  minutes
  • 81. Redis  replicaKon  starts  with  a  BGSAVE BGSAVE  on  master Slave  imports  dumped  file
  • 82. Redis  replicaKon  starts  with  a  BGSAVE BGSAVE  on  master Slave  imports  dumped  file =>  No  RAM  means  no  new  slaves
  • 83. Redis  had  a  memory  fragmenKon  problem 44  GB in  8  days 24  GB
  • 84. Redis  had  a  memory  fragmenKon  problem 38  GB in  3  days 24  GB
  • 85. If  MySQL  is  a  truck Fast  enough Disk  based Robust Fast  enough                    disk  based                    robust
  • 86. If  MySQL  is  a  truck,  Redis  is  a  race  car Super  fast RAM  based Fragile Super  fast                    RAM  based                    fragile
  • 87. Big  and  staKc  data  in  MySQL,  rest  goes  to  Redis 256  GB  data 60  GB  data 10%  writes 50%  writes hSp://www.flickr.com/photos/erix/245657047/
  • 88. Lots  of  boxes,  but  automaKon  helps  a  lot! lb lb app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app db db db db db redis redis redis redis redis
  • 89. We  reached  1  million  daily  users! &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" 1,000,000  -­‐  Big  party! #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
  • 90. We  started  archiving  inacKve  users &$!!!$!!!" %$#!!$!!!" 50%  DB %$!!!$!!!" reducKon #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
  • 91. We  even  survived  a  complete  data  center  loss &$!!!$!!!" EBS  no %$#!!$!!!" more! %$!!!$!!!" #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
  • 92. We  improved  our  MySQL  schema  on-­‐the-­‐fly &$!!!$!!!" 30%  DB %$#!!$!!!" reducKon %$!!!$!!!" #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
  • 93. Will  we  reach  2  million  daily  users? &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
  • 94. A  journey  to  1,000,000  daily  users Start  of  the  journey 6  weeks  of  pain Paredise  (or  not?) Conclusion
  • 95. You  do  not  know  the  future Plan  ahead
  • 96. You  do  not  know  the  future Plan  ahead Learn
  • 97. You  do  not  know  the  future Plan  ahead Learn Adapt
  • 98. EvoluKon  every  week EVOLUTION of  sonware &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
  • 99. EvoluKon  every  week EVOLUTION of  sonware &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
  • 100. EvoluKon  every  week EVOLUTION of  sonware
  • 101. EvoluKon  every  week,  RevoluKon  if  necessary REVOLUTION of  sonware
  • 102. EvoluKon  every  week,  RevoluKon  if  necessary REVOLUTION of  sonware
  • 103. EvoluKon  every  week,  RevoluKon  if  necessary REVOLUTION of  sonware &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
  • 104. Each  new  game  is  a  revoluKon
  • 105. Each  new  game  is  a  revoluKon
  • 106. Each  new  game  is  a  revoluKon
  • 107. Each  new  game  is  a  revoluKon
  • 108. Each  new  game  is  a  revoluKon Cooming soon
  • 110. Works  for  teams  and  for  companies !""#$%&"'()"*+,
  • 111. Thank  you! Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobs