At Wooga we are creating the next generation of social games. To be truly social, a game needs to offer real interaction between players in different forms. Taking one of our upcoming games as an example we will present the state synchronization used in client and server, how we do real time communication without websockets and how we are making a backend scalable enough to support millions of users every day. We will also share secrets about our prototypes for first person and real time strategy games. The Erlang process model with evented IO, asynchronous messaging, links and isolation fits very well to real time multiplayer games. Come to our talk to see how it's used to build a very successful business.

1. You are not alone
Scaling Multiplayer Games
Jesper Richter-Reichhelm - @jrirei
Knut Nesheim - @knutin

9. > 1,000,000 daily users

10. > 1,000,000 daily users
> 50,000 concurrent users

11. > 1,000,000 daily users
> 50,000 concurrent users
> 10,000 requests / second

12. Ok, you can
handle that.

13. Ok, you can
handle that.
But why only
single player?

14. The Games
The Past
The Present
The Future
The Foundation

15. Server Database

16. Server Database

17. Server Database

18. Server Database

19. Server Database

20. Server Database

21. DB is the bottleneck

23. Server Database

24. Server Database

25. Server Database

26. Server Database
One Game Session

27. Server Database
One Game Session

28. session

29. session
session
session
session

30. Server
session
session
session
session

31. Server
session
session
session
session

32. Server
session
session
session
session

33. Server
session
session
session
session

34. Server Server Server
session
session session
session session
session
session
session session
session session
session

35. The Games
The Past
The Present
The Future
The Foundation

36. es
Thoughts
t
Client
ent
Server
re
Operation
ation

37. es
Thoughts
t
Client
ent
Server
re
Operation
ation

39. You are not alone

40. Pessimistic

41. Pessimistic
Schizophrenic

42. Pessimistic
Schizophrenic
Optimistic

43. Pessimistic

44. Client Server
A A
Client Server
B B

45. Client Server
A @ B A
Client Server
B B

46. Client Server
A @ B A
Client Server
B B

47. Client Server
A @ B A
Client Server
B B

48. Client Server
A @ B A
Client Server
B B’

49. Client Server
A @ B A
Client Server
B B’

50. Client Server
A @ B A
Client Server
B B’

51. Client Server
A @ B’ A
Client Server
B B’

52. Client Server
A @ B’ A
Client Server
B B’

53. Client Server
A @ B’ A
Client Server
B’ B’

54. Waiting on server
responses?

55. Waiting on server
responses?
Are you out
of your mind?

56. Schizophrenic

57. Client Server
A A
Client Server
B B

58. Client Server
A @ B A
Client Server
B B

59. Client Server
A @ B A + B
Client Server
B B

60. Client Server
A @ B’ A + B
Client Server
B B

61. Client Server
A @ B’ A + B
Client Server
B B

62. Client Server
A @ B’ A + B’
Client Server
B B

63. Client Server
A @ B’ A + B’
Client Server
B B

64. Client Server
A @ B’ A + B’
Client Server
B B

65. Client Server
A @ B’ A + B’
Client Server
B’ B

66. Client Server
A @ B’ A + B’
Client Server
B’ B

67. Client Server
A @ B’ A + B’
Client Server
B’ B’

68. What about real
interactions?

69. What about real
interactions?
And what about
complexity
in the backend?

70. Optimistic

71. Client Server
A A
Client Server
B B

72. Client Server
A @ B A
Client Server
B B

73. Client Server
A @ B’ A
Client Server
B B

74. Client Server
A @ B’ A
Client Server
B B

75. Client Server
A @ B’ A
Client Server
B B

76. Client Server
A @ B’ A
Client Server
B B’

77. Client Server
A @ B’ A
Client Server
B B’

78. Client Server
A @ B’ A
Client Server
B B’

79. Client Server
A @ B’ A
Client Server
B’’ B’

80. Client Server
A @ B’ A
Client Server
B’’ B’

81. Client Server
A @ B’ A
Client Server
B’’ B’

82. Client Server
A @ B’ A
Client Server
B’ B’

83. Hmm, client
rollbacks?

84. Hmm, client
rollbacks?
Are you sure you
can build that?

85. es
t
ent
re
ation

86. es
Thoughts
t
Client
ent
Server
re
Operation
ation

87. Client Server
A A
Client Server
B B

90. Client A

91. state
count 0
max 3
Client A

92. user A
move
state
count 0
max 3
Client A

93. user A
move
pick
state
count 0
max 3
Client A

94. user A
move
pick
state
count 1
max 3
Client A

95. user A
move
pick
state
count 1
max 3
Client A Server

96. user A
user A
move
pick
state
count 1
max 3
Client A Server

97. user A
user A
move
pick
loca,on
state count 0
count 1
max 3
Client A Server

98. user A
user A
move
pick
loca,on
state count 1
count 1
max 3
Client A Server

99. user A
user A
move
pick
loca,on
state count 1
count 1
max 3
Client A Server

100. user A
user A
move
pick
loca,on
state count 1
count 1
max 3
history
move A
pick A
Client A Server

101. user A
user A
move
pick
loca,on
state count 1
count 1
max 3
history
move A
pick A
Client A Server

102. user A
user A
move
pick
loca,on
state count 1
count 1
max 3
history
move A
pick A
Client A Server

103. user A
user A
move
pick
loca,on
state count 1
count 1
max 3
user B
history
move A
pick A
Client A Server

104. user A
user A
move
pick
loca,on
state count 1
count 1
max 3
user B
history
move A
pick A
loca,on
Client A Server

105. user A
user A
move
pick
loca,on
state count 2
count 1
max 3
user B
history
move A
pick A
loca,on
Client A Server

106. user A
user A
move
pick
loca,on
state count 2
count 1
max 3
user B
history
move A
pick A
loca,on
Client A Server

107. user A
user A
move
pick
loca,on
state count 2
count 1
max 3
user B
history
move A
pick A
pick B loca,on
Client A Server

108. user A
user A
move
pick
loca,on
state count 2
count 1
max 3
user B
user B history
pick move A
pick A
pick B loca,on
Client A Server

109. user A
user A
move
pick
loca,on
state count 2
count 2
max 3
user B
user B history
pick move A
pick A
pick B loca,on
Client A Server

110. user A
user A
move
pick
loca,on
state count 2
count 2
max 3
user B
user B history
pick move A
pick A
pick B loca,on
Client A Server

111. user A
user A
move
pick
pick loca,on
state count 2
count 2
max 3
user B
user B history
pick move A
pick A
pick B loca,on
Client A Server

112. user A
user A
move
pick
pick loca,on
state count 2
count 3
max 3
user B
user B history
pick move A
pick A
pick B loca,on
Client A Server

113. user A
user A
move
pick
pick loca,on
state count 2
count 3
max 3
user B
user B history
pick move A
pick A
pick B loca,on
Client A Server

114. user A
user A
move
pick
pick loca,on
state count 2
count 3
max 3
user B
user B history
pick move A
pick A
pick B loca,on
Client A Server

115. user A
user A
move
pick
pick loca,on
state count 3
count 3
max 3
user B
user B history
pick move A
pick A
pick B loca,on
Client A Server

116. user A
user A
move
pick
pick loca,on
state count 3
count 3
max 3
user B
user B history
pick move A
pick A
pick B loca,on
Client A Server

117. user A
user A
move
pick
pick loca,on
state count 3
count 3
max 3
user B
user B history
pick move A
pick A
pick B loca,on
Client A Server

118. user A
user A
move
pick
pick loca,on
state count 3
count 3
max 3
user B
user B history
pick move A
pick A
pick B loca,on
pick B
Client A Server

119. user A
user A
move
pick
pick loca,on
state count 3
count 3
max 3
user B
user B history
pick move A
pick pick A
pick B loca,on
pick B
Client A Server

120. user A
user A
move
pick
pick loca,on
state count 3
count 3
max 3
user B
user B history
pick move A
pick pick A
pick B loca,on
pick B
Client A Server

121. user A
user A
move
pick
pick loca,on
state count 3
count 3
max 3
user B
user B history
pick move A
pick pick A
pick B loca,on
pick B
Client A Server

122. user A
user A
move
pick
pick loca,on
state count 3
count 3
max 3
user B
user B history
pick move A
pick pick A
pick B loca,on
pick B
Client A Server

123. user A
user A
loca,on
state count 3
count 0
max 3
user B
user B history
move A
pick A
pick B loca,on
pick B
Client A Server

124. user A
user A
move
loca,on
state count 3
count 0
max 3
user B
user B history
move A
pick A
pick B loca,on
pick B
Client A Server

125. user A
user A
move
pick
loca,on
state count 3
count 1
max 3
user B
user B history
move A
pick A
pick B loca,on
pick B
Client A Server

126. user A
user A
move
pick
loca,on
state count 3
count 2
max 3
user B
user B history
pick move A
pick A
pick B loca,on
pick B
Client A Server

127. user A
user A
move
pick
loca,on
state count 3
count 3
max 3
user B
user B history
pick move A
pick pick A
pick B loca,on
pick B
Client A Server

128. ...
(Let’s look
inside)

129. es
t
ent
re
ation

130. es
Thoughts
t
Client
ent
Server
re
Operation
ation

131. Erlang

132. Erlang
Process isolation

133. Erlang
Process isolation
Networking

134. Erlang
Process isolation
Networking
Distribution

135. User process

136. User process
Stores user state

137. User process
Stores user state
Real-time push to client

138. World process

139. World process
Serializes updates

140. World process
Serializes updates
Concurrency “control”

141. World process
Serializes updates
Concurrency “control”
Runs game logic

142. Game logic

143. Game logic
Pure

144. Game logic
Pure
f(State, Cmd) -> {NewState, Reply}

145. Game logic
Pure
f(State, Cmd) -> {NewState, Reply}
Transactions

146. Real-time push

147. Real-time push
Browser support

148. Real-time push
Browser support
“Transfer-Encoding: chunked”

149. Real-time push
Browser support
“Transfer-Encoding: chunked”
State updates as JSON

150. es
t
ent
re
ation

151. es
Thoughts
t
Client
ent
Server
re
Operation
ation

152. Elli

153. Elli
Efficient, robust

154. Elli
Efficient, robust
High-throughput, low latency

155. Elli
Efficient, robust
High-throughput, low latency
Request-response model

156. Elli
Efficient, robust
High-throughput, low latency
Request-response model
github.com/knutin/elli

157. Distribution

158. Distribution
Idea: Identical nodes, Riak

159. Distribution
Idea: Identical nodes, Riak
No centralization/serialization

160. Distribution
Idea: Identical nodes, Riak
No centralization/serialization
No SPOF

161. Distribution needs

162. Distribution needs
Locking

163. Distribution needs
Locking
Dynamic cluster

164. Meet “locker”

165. Meet “locker”
Distributed lock table

166. Meet “locker”
Distributed lock table
CAS

167. Meet “locker”
Distributed lock table
CAS
Write quorum, 2PC

168. Meet “locker”
Distributed lock table
CAS
Write quorum, 2PC
Asynchronous replication

169. Meet “locker”
Distributed lock table
CAS
Write quorum, 2PC
Asynchronous replication
330 SLOC, one gen_server

170. The Games
The Past
The Present
The Future
The Foundation

171. Research

172. Research
Aka “hacking”

173. Everything is possible

174. Everything is possible
First person

175. Everything is possible
First person
Real-time strategy

176. Everything is possible
First person
Real-time strategy
Turn-based

177. Everything is possible
First person
Real-time strategy
Turn-based
Etc

178. Nothing is possible

179. Nothing is possible
Network latency kills
gameplay

180. Lag compensation

181. Lag compensation
Many solutions

182. Lag compensation
Many solutions
Depends on game

183. FPS

184. FPS
Speed is king

185. FPS
Speed is king
Divergent simulations

186. FPS
Speed is king
Divergent simulations
Server keeps history

187. FPS
Speed is king
Divergent simulations
Server keeps history
Validates delayed commands

188. RTS

189. RTS
Synchronization is king

190. RTS
Synchronization is king
Identical simulations

191. RTS
Synchronization is king
Identical simulations
Server buffers commands

192. RTS
Synchronization is king
Identical simulations
Server buffers commands
Broadcasts, lockstep

193. The Games
The Past
The Present
The Future
The Foundation

194. Small independent teams

195. Each team adds something

196. Based on previous learnings

197. Standing on the
shoulders of giants

198. Slides: http://woo.ga/backend

199. Slides: http://woo.ga/backend
Code: http://github.com/wooga

200. Questions?

201. Questions?
@jrirei @knutin
http://wooga.com/jobs