Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Who  is  that  guy?  Jesper  Richter-­‐Reichhelm  Twi1er:  @jrirei  Head  of  Engineering  wooga    Berlin,  Germany
wooga  is  #3  game  developer  on  Facebook
Wooga  has  dedicated  game  teams                                     Cooming                                       soon
Flash  client  sends  state  changes  to  backend       Flash  client               Ruby  backend
Social  games  need  to  scale  quite  a  bit                400  million  PIs  /  month
Social  games  need  to  scale  quite  a  bit                400  million  PIs  /  month
Social  games  need  to  scale  quite  a  bit              14  billion  requests  /  month
Social  games  need  to  scale  quite  a  bit              14  billion  requests  /  month
Social  games  need  to  scale  quite  a  bit              14  billion  requests  /  month           100,000  DB  operaKon...
Social  games  need  to  scale  quite  a  bit              14  billion  requests  /  month             50,000  DB  updates...
Social  games  need  to  scale  quite  a  bit              14  billion  requests  /  month             50,000  DB  updates...
A  journey  to  1,000,000  daily  users  Start  of  the  journey  6  weeks  of  pain  Paradise  Conclusion
October  2009:  wooga’s  first  simulaKon  game
Instead  of  PHP  we  used  Ruby
Our  database  was  MySQL
Our  database  was  MySQL           even  user  ids   odd  user  ids
And  we  went  into  the  cloud
Master-­‐slave  replicaKon  for  DBs  worked  fine                          lb                    app   app   app          ...
We  added  a  few  applicaKon  servers  over  Kme                                lb        app   app   app   app   app   a...
250K  daily  users  and  no  problems&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!"                            Life  was  good   ...
Life  was  well  and  I  went  on  a  nice  vacaKon                                                       TO  DO          ...
Our  bane:  MySQL  hiccups (!!"#  !"#  &!"#  %!"#  $!"#   !"#         !#   )#   (!#   ()#   $!#   $)#   *!#   *)#   %!#
Our  bane:  MySQL  hiccups (!!"#  !"#  &!"#  %!"#  $!"#   !"#         !#   )#   (!#   ()#   $!#   $)#   *!#   *)#   %!#
Our  bane:  MySQL  hiccups (!!"#  !"#  &!"#  %!"#  $!"#   !"#         !#   )#   (!#   ()#   $!#   $)#   *!#   *)#   %!#
A  journey  to  1,000,000  daily  users  Start  of  the  journey  6  weeks  of  pain  Paradise  Conclusion
SQL  queries  generated  by  Rubyamf  gem  AMF  responses  to  Flash  client
SQL  queries  generated  by  Rubyamf  gem  AMF  responses  to  Flash  client  Wrong  config...  ...  so  associated  data  ...
SQL  queries  generated  by  Rubyamf  gem  AMF  responses  to  Flash  client  Wrong  config...  ...  so  associated  data  ...
More  traffic  using  the  same  cluster                                lb        app   app   app   app   app   app   app   ...
Config  tweaks  brought  us  to  300K  DAU&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!"                            Config  fixes   ...
AcKveRecord’s  checks  caused  20%  extra  DB    Checking  connecKon  state  MySQL  process  list  full  of  ‘status’  calls
AcKveRecord’s  checks  caused  20%  extra  DB    Checking  connecKon  state  MySQL  process  list  full  of  ‘status’  cal...
I/O  on  MySQL  masters  sKll  was  the  bo^leneck New  Relic:  60%  of  all  UPDATEs  on  ‘Kles’  table
Tiles  are  part  of  the  core  game  loop                                       Core  game  loop                        ...
We  started  to  shard  on  model,  too  Adding  new  shards     old           old      master        slave
We  started  to  shard  on  model,  too  Adding  new  shards    1)  Setup  new  masters  as  slaves  of  old  ones     old...
We  started  to  shard  on  model,  too  Adding  new  shards    1)  Setup  new  masters     old           old          new...
We  started  to  shard  on  model,  too  Adding  new  shards    1)  Setup  new  masters    2)  Start  using  new  masters ...
We  started  to  shard  on  model,  too  Adding  new  shards    1)  Setup  new  masters    2)  Start  using  new  masters ...
We  started  to  shard  on  model,  too  Adding  new  shards    1)  Setup  new  masters    2)  Start  using  new  masters ...
4  DB  masters  and  a  few  more  servers                                       lb          app   app   app        app   ...
Sharding  by  model  brought  us  to  400K  DAU&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!"                            Shard  b...
We  improved  our  MySQL  setup  RAID-­‐0  of  EBS  volumes
We  improved  our  MySQL  setup  RAID-­‐0  of  EBS  volumes  Using  XtraDB
We  improved  our  MySQL  setup  RAID-­‐0  of  EBS  volumes  Using  XtraDB  Tweaking  my.cnf
Sharding  gem  circumvented  AR’s  internal  cache  AcKveRecord  caches  SQL  queries...
Sharding  gem  circumvented  AR’s  internal  cache  AcKveRecord  caches  SQL  queries...  ...  only  in  our  development ...
Sharding  gem  circumvented  AR’s  internal  cache  AcKveRecord  caches  SQL  queries...  ...  only  in  our  development ...
I/O  sKll  was  not  fast  enough  If  2  +  2  is  not  enough,  ...
I/O  sKll  was  not  fast  enough  If  2  +  2  is  not  enough,  ...  …  perhaps  4  +  4  masters  will  do?
It’s  no  fun  to  handle  8+8  MySQL  DBs                                  lb         app   app    app   app   app   app ...
It’s  no  fun  to  handle  8+8  MySQL  DBs                                         lb           app        app    app   ap...
At  500K  DAU  we  were  at  a  dead  end&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!"        !"        ()*%!"   +,-*%!"   ./0*%...
At  500K  DAU  we  were  at  a  dead  end&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!"        !"        ()*%!"   +,-*%!"   ./0*%...
I/O  remained  bo^leneck  for  MySQL  UPDATEs            Each  DB  master  could  do             about  1000  DB  write/s.
I/O  remained  bo^leneck  for  MySQL  UPDATEs            Each  DB  master  could  do             about  1000  DB  write/s....
Pick  the  right  tool  for  the  job!
Redis  is  fast  but  goes  beyond  simple  key/value  Redis  is  a  key-­‐value  store    Hashes,  Sets,  Sorted  Sets,  ...
Redis  is  fast  but  goes  beyond  simple  key/value  Redis  is  a  key-­‐value  store    Hashes,  Sets,  Sorted  Sets,  ...
Wooga  has  dedicated  game  teams
Shelf  Kles  :  An  ideal  candidate  for  using                                                 Shelf  Kles:             ...
Shelf  Kles  :  An  ideal  candidate  for  using  Redis     Redis  Hash     HGETALL     HGETHSET     HINCRBY     …
Migrate  on  the  fly  when  accessing  new  model
Migrate  on  the  fly  -­‐  but  only  once                   true  if  id  could  be  added                   else  false
Typical  migraKon  throughput  over  3  days
Migrate  on  the  fly  -­‐  and  clean  up  later   1. Let  migraKon  run  unKl  everything  cools  down
Migrate  on  the  fly  -­‐  and  clean  up  later   1. Let  migraKon  run  unKl  everything  cools  down   2. Migrate  the ...
Migrate  on  the  fly  -­‐  and  clean  up  later   1. Let  migraKon  run  unKl  everything  cools  down   2. Migrate  the ...
Migrate  on  the  fly  -­‐  and  clean  up  later   1. Let  migraKon  run  unKl  everything  cools  down   2. Migrate  the ...
Migrate  on  the  fly  -­‐  and  clean  up  later   1. Let  migraKon  run  unKl  everything  cools  down   2. Migrate  the ...
A  journey  to  1,000,000  daily  users  Start  of  the  journey  6  weeks  of  pain  Paredise  (or  not?)  Conclusion
Again:  Tiles  are  part  of  the  core  game  loop                                       Core  game  loop                ...
Size  ma^ers  for  migraKons  MigraKon  check  overload    Migra<on  only  on  startup
Size  ma^ers  for  migraKons  MigraKon  check  overload    Migra<on  only  on  startup  Overlooked  an  edge  case    Only...
In-­‐memory  DBs  don’t  like  to  dump  to  disk Dumping  to  disk   SAVE  is  blocking   BGSAVE  needs  free  RAM
In-­‐memory  DBs  don’t  like  to  dump  to  disk Dumping  to  disk   SAVE  is  blocking   BGSAVE  needs  free  RAM Latenc...
In-­‐memory  DBs  don’t  like  to  dump  to  disk Dumping  to  disk   SAVE  is  blocking   BGSAVE  needs  free  RAM Latenc...
Redis  replicaKon  starts  with  a  BGSAVE BGSAVE  on  master Slave  imports  dumped  file
Redis  replicaKon  starts  with  a  BGSAVE BGSAVE  on  master Slave  imports  dumped  file =>  No  RAM  means  no  new  sla...
Redis  had  a  memory  fragmenKon  problem                                   44  GB                                       ...
Redis  had  a  memory  fragmenKon  problem                                  38  GB                                       i...
If  MySQL  is  a  truck                                                                Fast  enough                       ...
If  MySQL  is  a  truck,  Redis  is  a  race  car                                                                  Super  ...
Big  and  staKc  data  in  MySQL,  rest  goes  to  Redis        256  GB  data                               60  GB  data  ...
Lots  of  boxes,  but  automaKon  helps  a  lot!                                           lb         lbapp   app    app  ...
We  reached  1  million  daily  users!&$!!!$!!!"%$#!!$!!!"%$!!!$!!!"                                      1,000,000  -­‐  ...
We  started  archiving  inacKve  users&$!!!$!!!"%$#!!$!!!"                                                          50%  D...
We  even  survived  a  complete  data  center  loss&$!!!$!!!"                                                             ...
We  improved  our  MySQL  schema  on-­‐the-­‐fly&$!!!$!!!"                                                                 ...
Will  we  reach  2  million  daily  users?&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!"        !"        ()*%!"   +,-*%!"   ./0*...
A  journey  to  1,000,000  daily  users  Start  of  the  journey  6  weeks  of  pain  Paredise  (or  not?)  Conclusion
You  do  not  know  the  future                     Plan  ahead
You  do  not  know  the  future                     Plan  ahead                       Learn
You  do  not  know  the  future                     Plan  ahead                       Learn                       Adapt
EvoluKon  every  week              EVOLUTION                                    of  sonware        &$!!!$!!!"        %$#!!...
EvoluKon  every  week              EVOLUTION                                    of  sonware        &$!!!$!!!"        %$#!!...
EvoluKon  every  week          EVOLUTION                  of  sonware
EvoluKon  every  week,  RevoluKon  if  necessary       REVOLUTION                   of  sonware
EvoluKon  every  week,  RevoluKon  if  necessary       REVOLUTION                   of  sonware
EvoluKon  every  week,  RevoluKon  if  necessary       REVOLUTION                                     of  sonware         ...
Each  new  game  is  a  revoluKon
Each  new  game  is  a  revoluKon
Each  new  game  is  a  revoluKon
Each  new  game  is  a  revoluKon
Each  new  game  is  a  revoluKon                                    Cooming                                      soon
Works  for  teams  ...
Works  for  teams  and  for  companies                 !""#$%&"()"*+,
Thank  you!Jesper  Richter-­‐Reichhelm          @jrirei slideshare.net/wooga    wooga.com/jobs
Thank  you!Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga
Thank  you!Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga
Thank  you!Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga
Nächste SlideShare
Wird geladen in …5
×

Thank  you!Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga 1,000,000 daily users and no cache (Splash 2011)

73.587 Aufrufe

Veröffentlicht am

Thank  you!Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobs

Veröffentlicht in: Technologie, Unterhaltung & Humor
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy &amp; Proven Way to Build Good Habits &amp; Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • At peak we do about 8,000 https requests that generate 100,000 DB operations per second (of which 50,000 are updates). Most http requests generate only 2-3 DB updates, but there are a few very complex requests that cause dozens of updates, hence the high ratio.
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • page 12 said you got 14 billion requests/month, wihch equals with 4,000 requests per second, while there are 5000 DB updates per second. It seems these two number do not fit with each other.
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Re myst1313: We only serve the landing page using SSL (400 million hits per month). We just added another load balancer to handle the load. Our Flash client does not use SSL to call our API so we do not have a problem there.
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

×