1.
We#will#begin#shortly...

2.
Stay%Awhile%And%Fiddle%With%Your%
Smartphone
By#Jamie#Winsor

3.
The$Story

4.
Building(And(Suppor.ng
Online&Experiences

5.
NoOps
Doesn't(Mean(No(Opera/ons

6.
It#Means#No#Dedicated#Opera2ons

7.
Architec(ng,,Planning,,and,Implemen(ng,is,
40%,of,an,Engineer's,job

8.
The$other$60%$is$team$building$and$
communica6on.

9.
The$Problem

10.
How$Do$We
• Stay&development&focused
• Transi6on&from&single&player&to&massively&mul6player&online&
games
• Build&a&posi6ve&rela6onship&with&our&publishers
• Develop&and&maintain&a&strong&company&culture

11.
Develop'Your'Company,'Not'Just'
Your'So6ware

12.
Hire%Engineers,%Not%Programmers

13.
Don't&Be&Afraid&To&Contract&Work&
Out

14.
Never%Higher%A%Maybe

15.
Interview)For
• IQ$%$intelligence
• EQ$%$emo0onal$intelligence
• Personality

16.
Develop'Smart
Be#Pragma)c#And#Iterate
Leverage'Open'Source

17.
Use$The$Right$Tools$And$Pa2erns

18.
Building(A(Distributed(System?

19.
OTP
A"Framework"For"Building"Distributed"Systems

20.
Would&You&Build&Your&Own&Web&Framework?

21.
Or#Your#Own#Game#Engine?

22.
You$Could$But...

23.
Some%Of%Us%Have%Sh_t%To%Do

24.
Erlang/OTP+Saves+Time
A"Lot"Of"Time

25.
But$Only$If$Your$Team$Buys$In

26.
Team%Language%Evalua-on
What%technology%should%we%adopt?
• C/C++
• C#&mono
• Python
• Ruby
• Erlang

27.
Game%engineers%are%generally%experienced%in%
C2like%languages

28.
Erlang/OTP+Network+Programming+Perks
• Easily(encode(and(decode(binary(message
• Monitor(/(subscribe(to(open(sockets
• "Free"(internal(rou:ng(of(messages
• A(sane(threading(model
• Crash?Resistant

29.
Erlang'isn't'very'approachable
—"Concerned"Engineer

30.
Elixir&Has&The&Same&Core&Concepts
• Processes(and(Mailboxes
• Supervision(Trees
• Pa6ern(Matching
• Tail(Recursion

31.
Started'Using'Elixir'at'0.9.0
We#Asked#Ourselves,#"Is#This#A#Risk?"

32.
Solid&Two)Way&Interoperability

33.
Leverage'Erlang'Tooling'And'Libraries
For$Free.

34.
Syntax'Doesn't'Ma.er

35.
(Except(It(Does)

36.
Elixir's(Syntax(Makes
Erlang's)Ecosystem)More)
Approachable
For$Your$Colleagues

37.
Have%Compassion

38.
Dev$Tooling

39.
Mix
• Build'Tool
• Create/Compile/Test'projects
• Dependency'management
$ mix new hello_world
$ cd hello_world
$ mix deps.get

40.
Hex

41.
How$Are$We$Gonna$Run$This?

42.
Live%In%The%Cloud
• On$demand$test$environments
• Expand$or$contract$your$live$cluster

43.
No#Opera)ons?
Who$Will$Configure$Our$Machines?

44.
Applica'on*Engineer*Owns
The$Cookbook$For$Their$Applica1on

45.
All#Applica(ons#Are#Developed
And$Designed$With$Auto/Discovery$In$Mind

46.
Discovery
Automa'cally+Discover+And+Connect+Erlang+
Nodes
github.com/undeadlabs/discovery

47.
Discovery*Breakdown
• Poller'('Discover'nodes'running'an'OTP'applica6on
• PollerEvent'('Callbacks'to'run'when'poller'finds'something
• Handler'('Behavior,'callbacks'to'run'when'poller'event'is'fired
• Heartbeat'('Broadcast'availability

48.
Chat%Supervisor%init/1%Snippet
children = [
worker(Discovery.Poller, ["tu_route", [
{Discovery.Handler.NodeConnect, [
{Tubes.RouteConnector, [:tu_chat, Chat.Repo.shard_ids]}
]}
]], id: Poller.Route),
worker(Discovery.Heartbeat, [@heartbeat_check, Config.heartbeat_ttl])
]
supervise(children, strategy: :one_for_one)

49.
Who$Will$Build$Our$Machines?

50.
Terraform
• Abstrac)on,for,Cloud,providers
• Allows,you,to,describe,network/server,resources,in,code
• Security,Groups
• VPCs,,Subnets
• Virtual,Instances
• And,more

51.
Applica'on*Engineer*Owns
And$Maintains$A$Terraform$Module
For$Their$Applica.on

52.
Configurable,Module
variable "route-servers" {
default = "3"
description = "The number of route server instances to launch."
}
variable "app-servers" {
default = "3"
description = "The number of app server instances to launch."
}
variable "replication_servers" {
default = "1"
description = "The number of database replication servers to launch."
}

53.
Terraform(To(Describe(&(Provision(
Environents
Chef%Describe%&%Configure%Your%
Nodes

54.
No#Such#Thing#As#DevOps#Team
Every&server&engineer&needs&to&understand&Elixir,&C#,&Chef,&and&the&
cloud&tooling.

55.
Foster'Adop+on
For$Your$Technology$Choices
• Make&on)ramps&for&your&teammates
• Create&a&safe&to&fail&environment
• Create&and&evangelize&pa9erns&and&standardiza:ons.&Allow&for&
collabora:on.&(Illusion&of&choice?)

56.
Turn%Your%Colleagues%Into%
Ambassadors%For%Technology

57.
Code%Reviews
• Assist&with&learning
• Keep&a&posi2ve&tone&in&your&code&reviews.&Stay&away&from&an&
authorata2ve&tone&(I&love&you&but&I&push&you&in&the&dirt)
• Generic&praise&is&not&enough
• Avoid&syntax/style&discussions

58.
DevOps
Make%opera)ons%every%engineer's%responsibility
• Protect(your(culture
• Control(your(live(service
• Don't(become(beholden(to(a(service(organiza:on(with(a(
monopoly
Operate'without'the'extra'headcount

59.
High%Level%Architecture%Overview
Of#Moonrise

60.
Three%Components%of%An%Online%Game
• Game&Client
• Game&Server(s)
• Service&Pla4orm

61.
Game%Server(s)
• Simulate*the*game
• Nego/ates*with*clients
• Communicate*with*pla6orm

62.
Pla$orm
• Game&agnos+c
• Highly&available
• Game&experience&s+tching
• Provides&services&to&games
• Auth,&Chat,&Presence,&Leaderboards,&Guilds,&Player&Profile,&and&
more

63.
Applica'on*Cookbooks
• Tubes'(Pla,orm)
• Provides'a'recipe'for'each'service
• Daemon'(Moonrise'code'name)
• Provides'a'receipt'for'each'game'server'type
• Undead'Labs'Base
• Provides'common'funcAonality'for'all'UL'nodes

64.
Describing*Environments
• cloud'environments0repo0on0Github
• One0directory0per0environment
• Remote0state0storage0in
• Atlas
• Consul

65.
$ git clone git@github.com:undeadlabs/cloud-environments.git
$ cd cloud-environments
$ cat moonrise-jwtest/default.tf

66.
module "undead" {
source = "git::git@github.com:undeadlabs/undeadlabs-base-cookbook.git//terraform"
vpc_id = "${aws_vpc.moonrise.id}"
}
module "tubes" {
source = "git::git@github.com:undeadlabs/tubes.git//terraform"
chef_env = "moonrise-tubes-jwtest"
vpc_id = "${aws_vpc.moonrise.id}"
subnet_id = "${aws_subnet.moonrise-any.id}"
lab_cidr = "${module.undead.lab_cidr}"
common_sg_id = "${module.undead.server_common_sg_id}"
split_api_client_sg_id = "${module.split-api.split_api_client_sg_id}"
}
provider "aws" {
access_key = "${var.access_key}"
secret_key = "${var.secret_key}"
region = "${var.region}"
}
resource "aws_vpc" "moonrise" {
cidr_block = "172.30.0.0/16"
tags = {
Name = "moonrise-jwtest"
}
}
resource "aws_subnet" "moonrise-any" {
vpc_id = "${aws_vpc.moonrise.id}"
cidr_block = "172.30.0.0/16"
map_public_ip_on_launch = true
tags = {
Name = "moonrise-jwtest-any"
}
}

67.
Building(An(Environment
$ terraform get
$ terraform remote -backend=Consul -path=moonrise-jwtest -address=<addr>
$ terraform apply
$ git add .
$ git commit -a "initial commit for moonrise-jwtest environment"
$ git push

68.
Teaching)Resources
• h"ps://www.airpair.com/aws/posts/n1ered4aws4docker4terraform4
guide
• h"ps://www.terraform.io

69.
Jamie&Winsor
@resetexistence
github.com/reset