Rob Cameron presented on migrating the League of Legends platform to AWS. Some key points: - The platform includes chat, store, matchmaking, player statistics, and game servers running on microservices. - They used Terraform for infrastructure as code and Packer for image builds. Custom tools were developed in Golang. - The architecture places shared services, platforms, load testing, and game servers in separate VPCs with Direct Connect to Riot datacenters. - Successes included strong adoption of tools, faster deployment times, and scaling infrastructure from 50 to over 2000 servers in under a year.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Migrating the League of Legends
Platform into the AWS Cloud
R o b C a m e r o n
N o v e m b e r 2 7 , 2 0 1 7
G A M 3 0 1

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rob Cameron
Senior Infrastructure Engineer
• 15+ year veteran in network, security,
and systems
• Six time author around network
security
• Loves programming, coffee, and
helping people
• Hates repetitive tasks
• Rioter for two years
• MrCrapper—(NA)/EUNE/TR/RU/LAN
Introductions

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What are we trying to solve?

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Platform
• Chat
• Store
• Matchmaking
• Player statistics
rCluster
• Riot container engine
• Microservices
• All new services
Game servers
• Highly latency sensitive
• Target goal of ~60 ms
or lower
• Hundreds of games per
server
What is a League of Legends shard?

5. Riot shard locations
Riot
Garena
Tencent
PBE
NA
LAN
BR
LAS
EUW
EUNE
TR
RU CN
KR JP
SG
TW
PH
VN
ID
TH
OCE

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges for shard launches

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Initial concepts for AWS deployment
VPCApp
Subnet
DB
Subnet
Direct Connect

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Initial concepts for AWS deployment
VPC
FE App
Subnet
DB
Subnet
Direct Connect
BE App
Subnet

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How are we going to manage this?

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS API driven architecture

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Bare metal support All major cloud vendors Ability to ship quickly
Configuration as code Allows others to contribute Long-term support
Managing infrastructure

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Packer
Builds
• AWS AMIs
• VMware OVF
• Vagrant Virtualbox
• MAAS images
Successes
• Strong adoption within Riot
• Solved our imaging needs
Challenges
• Build pipeline
• Windows images

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Terraform
Infrastructure as code
• Simple to edit and version control
• Allows for collaboration
• One repository, many deployments
Successes
• Strong adoption within Riot
• Used between AWS and MAAS
Challenges
• State management
• Difficult to modify long term

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Workflow for infrastructure
Code Review Validate Apply

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Terraform state system
Code Known state True state

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Terraform state inheritance
VPC
DB
Servers
Web
Servers
ELB

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
resource "aws_vpc" "shard" {
cidr_block = "${var.aws_vpc_cidr}"
tags {
Name = "${var.vpc_name}"
Accounting = "${var.aws_accounting_id}"
ShardName = "${var.env_name}"
BuiltBy = "terraform"
}
}
output "vpc_id" {
value = "${aws_vpc.shard.id}"
}
output "vpc_cidr" {
value = "${var.aws_vpc_cidr}"
}
output "vpc_route_table_id" {
value = "${aws_vpc.shard.main_route_table_id}"
}
Code snippet

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
resource "aws_instance" "beapp" {
count = "${lookup(var.instance_count, "beapp")}"
ami = "${lookup(var.platform_ami, var.aws_region)}"
instance_type = "${lookup(var.instance_type, "beapp")}"
subnet_id = "${aws_subnet.platform.id}"
key_name = "${aws_key_pair.shard_key.key_name}"
user_data = "${format(file("${path.module}/user_data/basic_user_data.txt"), format("beapp%03d", count.index+1))}"
ebs_optimized = "${var.ebs_optimized}"
disable_api_termination = "${var.global_termination_protection}"
lifecycle {
ignore_changes = ["user_data"]
}
vpc_security_group_ids = ["${aws_security_group.beapp.id}", "${aws_security_group.common.id}"]
tags {
Name = "${format("beapp%03d", count.index + 1)}"
FQDN = "${format("beapp%03d.%s", count.index + 1, var.domain_name)}"
AppName = "beapp"
Owner = "${var.owner_email}"
Accounting = "${var.aws_accounting_id}"
ShardName = "${var.env_name}"
BuiltBy = "terraform"
}
}
Making instances

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Power DNS
• Create DNS entries
• Custom JWT auth
• Similar to Route 53
F5 configuration
• Create VIPs
• Auto populate servers
• Similar to an ELB/ALB
MAAS provider
• Deploy bare metal
• Custom user data
• Define deployed OS
• Similar to AWS
instances
Extending Terraform

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
When do you build a tool?
• Niche problem
• Existing tools don’t match workflow
• Full control over the situation
• Fun?
What do you build it in?
• A language in your ecosystem
• Supportability in your organization
• Something you can learn in?
Building your own tools

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Golang at Riot Games
We love the gopher!
• Tools
• Microservices
• Sidecars
Ekko
• Manages Chef data bags
EkkoEdge
• Manages edge load balancing
DNS Manager
• Manages AWS instances to DNS

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Architecture

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Architecture
Shared
Tools
Mock
Game servers
Test
Clients
Internet Riot
Direct
AWS Region
Production game servers rCluster
Riot WAN
Platforms

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Shared services
Proxy Auto Scaling group
region
VPC subnet
virtual private cloud
VPC subnet VPC subnet
DNS NTP DNS NTPDNSChef/Merlin

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Platform
region
virtual private cloud
Store Matchmaking Chat DatabasesBackend AppsFrontend Apps
Tools
VPC
FE Apps BE Apps Store
Services
Matchmaking Chat
Services
Persistence
Data
Stores

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Platform VPC2
region
virtual private cloud
Store DB Matchmaking DB Chat DBPlatform DB
Tools
VPC
Platform DB Store
DB
Matchmaking
DB
Chat DB Platform
VPC

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Load testing harness
region
virtual private cloud
Tools
VPC
LT A
LT Harness
LT Harness
LT B
LT Harness
LT Harness
LT C
LT Harness
LT Harness
LT E
LT Harness
LT Harness
LT D
LT Harness
LT Harness
LT F
LT Harness
LT Harness
LT G
LT Harness
LT Harness
LT H
LT Harness
LT Harness
LT …
LT Harness
LT Harness

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Load testing game servers
region
virtual private cloud
Tools
VPC
GS A
Game server
Game server
GS B
Game server
Game server
GS C
Game server
Game server
Platform
VPC

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Direct connect to Riot DCs
region

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Load testing flow
LT
Harness
AWS
Internet
Gateway
Internet Riot
Direct
Riot LB Direct
Connect
Platform
VPC

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Long Term Operational Challenges
ELB ScalingStuck
Instances
Lost VPN
Gateways
+/-
instances
Lost
volumes
Flapping
direct
connects

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Results

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Project successes
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Infrastructure Code
Commits
Pull Requests
Contributors

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Project successes
0
50
100
150
200
250
300
Servers Instances
Small
Large

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Project successes
0
1
2
3
4
5
6
7
8
9
10
Time in Months
Before
After

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
R i o t G a m e s a n d R o b C a m e r o n