The Player Accounts team at Riot Games needed to consolidate the player account infrastructure and provide a single, global accounts system for the League of Legends player base. To do this, they migrated hundreds of millions of player accounts into a consolidated, globally replicated composite database cluster in AWS. This provided higher fault tolerance and lower latency access to account data. In this talk, we discuss this effort to migrate eight disparate database clusters into AWS as a single composite database cluster replicated in four different AWS regions, provisioned with terraform, and managed and operated by Ansible.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Globalizing Player Accounts
at Riot Games While
Maintaining Availability
Tyler Turk
Senior Infrastructure Engineer
Riot Games
A R C 3 1 4

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Setting the foundation

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
League of Legends accounts history
Architecting a global service
Implementing a global service
Lessons learned
Recap

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Origin of accounts
Ten unique account databases
Hundreds of millions of players
Malicious login traffic

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Early challenges (small indie company)
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Enter RiotSignOn
Built on top of OpenID Connect
Replaced all authentication &
authorization systems
Adopted security standards

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Replicate RiotSignOn infrastructure

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Global Data Protection
Regulation (GDPR)
Modify data schema
to protect players
Encrypt personal information
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Desired features
Globally unique player
identifier (PUUID)
Global account lookup system
Third-party support
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What did we want?
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Repeatable infrastructure
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
High availability
Geographical relevance/routing
Fault tolerance
Deployable worldwide
Internationalization
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data consistency
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you verify?
Automated tests
Validate throughput limitations
Global service, global tests
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Enter Atlas
Overhaul of sharded
account system
Truly global accounts system
Decoupled from game platform
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Infrastructure management
Terraform managed
Global configuration
Ansible automated

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability
Interconnected via AWS
Direct Connect
Four AWS Regions
Low latency worldwide using NS1

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Consistency
Front-end service router
Global MySQL for accounts
Global DynamoDB
for authorizations
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Load testing

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Chaos testing
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why MySQL and not “X”?
Widely adopted at Riot
Used extensively by Tencent
Ensures ACID-compliance
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Continuent database clustering

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Continuent database clustering
r4.8xlarge
5TB MySQL GP2 EBS
15TB Logs/backups GP2 EBS
10 Gbps Network

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Managing the database cluster

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Schema migration
Initial migration modified schema
Schema wasn’t final

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Backups: Origins
Elect backup server
Perform xtrabackup
Compress with pigz
Upload to S3

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Backups: The horror story

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Backups: The recovery
First pass—Single region backup
Long term—Cluster slave replicator

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Issues we faced
Monitoring
Initial data import

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Recap
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Failure is simply the opportunity to begin
again, this time more intelligently.”
—Henry Ford

51. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tyler Turk
tturk@riotgames.com

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.