A talk from the Login 2011 conference on how to build games that connect hundreds of thousands of players in the same experience, using NoSQL, OSS tools & tech, and good planning. Also covers BigData analysis for understanding what your players are doing and what they want.

1. Massively Social
!=
Massively Multiplayer
Paul J. Furio
Technical Director @ Novel, Inc.

2. 1989 Multiplayer
Falcon (Spectrum HoloByte),
on the Amiga 500, 1200 baud
modem, 2 player dogfighting
1999 Multiplayer
Williams F1 Team-Team Driver,
8 player racing / Total
Annihilation – 4 player RTS,
33.6kbps modems

3. Year 2000 Multiplayer Game
Constraints
•
•
•
•
•
•
8-16 Simultaneous Players
33.6kbps Modems were the standard
10Hz Update Rate
UDP Messages
~430 bytes per message max overhead
Everything is all about Speed and Hiding Sync
Problems

4. Year 2000 Multiplayer Game
Architecture
Game
Client &
Server
Game
Client
Game
Client
Game
Client
Game
Client
• Peer-to-Peer or
Client-Server
• Authoritative
Server
• Predictive Physics
• “Twitch” Games
• 1-2 Year Dev Cycles

5. Late 2000’s Typical MMO
• Sharded Servers with ~10,000 Players Per “World”
• No true interaction across worlds
Server
A
Client
Client
Client
Client
Server
C
Server
B
Client
Client
Client
Client
Client
Client
Client
Client
Client
Client
Client
Client
Client
Client

6. The Dream of Primetime
• Joined the Xbox Live
Primetime Team in 2008
• Goal: Create Simultaneous
Online Experience for 200k
Players
• Execution: First Online Game
Show, “1 vs. 100 Primetime
Live” on Xbox 360
Realtime
Game
Server
Game
Game
Game
Game
Game
Client
Client
Client
Client
Client
+200k
users

7. Xbox Live 1 vs. 100
• Real Time
Interactive,
Massively Social,
Online Game
Show Experience
on a Game
Console
• ~2 Year Dev Cycle

8. Social Gaming
• Millions of Users
• “Friends List”
Interactions
– Leaderboards
– View Friends Content
– Limited Environmental
Interaction
• Stats, Upsells, Etc.
• Extremely Short Dev
Times (4 Month Goal)

9. Shift to Social
•
•
•
•
•
Massive Player Interactions
Speed of Broad Interaction Less Critical
Asynchronous Gameplay
Ability to Analyze Group Behavior
Analyze Individuals in Aggregate

10. How Do We Get There?
Cloud Computing
Analytics
RDB
Web Services &
Distributed Servers
GameGame
Game
GameGame
ClientClient
Client
ClientClient
NoSQL
100k+ users
to 20M+ users
• Segment Local from Global
Experience
• Embrace Web Technologies
(Distributed, Stateless Event
Processing)
• Speed Up Data Access
• Perform Massive Analytics
• Embrace The Cloud
• Engineers Must Deliver Tools
That Allow Art & Design To
Be Amazing & Engaging

11. Peer to Peer for Instant Interaction
Massive
Social
Services
Broadcast
Data
Game
Client
Game
Client
Game
Client
•
•
•
•
“Sphere of Interest”
Voice Chat
Group Activities
Multicast/CDN for
“Broadcast Social”
• Massive Interactions
via Other Systems

12. Move Away From RDB Only
• Relational Databases are Slow(er than current
alternatives)
• Keep Persistent, but Infrequently Accessed Data,
in Relational Database
• Shift Transient Data to a NoSQL Solution
RDB
User
Data
Player
Items
NoSQL
Transient
Battles
(Not Bum Fights)
Session
Data
Chat

13. Fast Data Access: NoSQL
• Key-Value Storage
• No Relational Calculus
• Super Fast with InMemory Data Storage
• Only Disk Access is
Asynchronous AppendOnly Logging

14. 1 vs. 100 NoSQL Solution
•
•
•
•
Developed In-House
~50k Transactions per Second
Client Side Timing Hid Aggregation
Functions: Set, Get, Increment,
Decrement, Top-100
• Custom Operators are Key

15. Publicly Available NoSQL
•
•
•
•
Memcache, Membase
Cassandra
MongoDB
Redis
–
–
–
–
Rich Command Set
Super Fast (C++)
Rediska & Predis APIs for PHP
Asynchronus Append-Only Logging

16. Big Data Analysis
• Millions of Users + Millions of
Transactions = Lots of Data
• Incredible Value in Data Analysis
• Use MapReduce/Hadoop to
Analyze User Interactions
• Capture Everything, Analyze
Later, Re-Analyze Even Later!

17. What is MapReduce?
• Distributed Compute-Cluster Analysis of Huge Data Sets
• Map: partition data into sub-problems
• Reduce: Combine answers from sub-problems to produce
output
• Each step is iterative: Mappers can feed Mappers,
Reducers can feed Reducers
• Fast: Hadoop sorted 1TB of Random 100 Byte Records in
62 seconds (across 1460 nodes)

18. What is MapReduce?
MAP
N
P
U
Reduce
MAP
MAP
MAP
T
Shuffle/Sort
I
Reduce
Reduce
Reduce
MAP
OUTPUT

19. Canonical MapReduce Example
void map(String name, String document):
// name: document name
// document: document contents
for each word w in document:
EmitIntermediate(w, "1");
void reduce(String word, Iterator partialCounts):
// word: a word
// partialCounts: a list of aggregated partial counts
int result = 0;
for each pc in partialCounts:
result += ParseInt(pc);
Emit(AsString(result));
Credit: Wikipedia
http://en.wikipedia.org/wiki/MapReduce

20. Cloud Computing
• “Rentable Commodity Hardware By The
Box, By The Hour”
• Cost Savings mean Almost No Reason to
CoLocate Services* (* for Large Scale Apps with Dynamic Demand)
• Scale Effectively with Dynamic Demand
• Caveat: Write code to Gracefully Handle
Hardware Failure

21. Amazon EC2 / Rackspace
• Spin Up New Instances Fast
• MapReduce Clusters On-Demand (or
use Hadoop)
• Integrated CDN (CloudFront/CloudFiles)
• Proven across industries (Playfish,
Netflix, Ericsson, Yelp, many more!)

22. Tech + Design = Winning!
• Tools enable powerful interactions
• Real strength comes in design of
applications
• Enable Content & Design Staff to Develop
Engaging, Amazing Experiences
• What do you want to learn about your
Users/Players?
• How can you create value from that
information?

23. The Novel Approach
• Create Gaming Experiences that are
Engaging and Fun
• Design Experiences that Provide Useful
Information about Players
• Apply Analysis to Understand Trends,
Individuals
• Use Data to Improve Productivity,
Increase Employee Happiness, Create
Better Team Dynamics, and More!

24. Summary
•
•
•
•
•
•
Move away from Monolithic Game Servers
Embrace the Social Paradigm
Fast Data w/ NoSQL
Big Data Analysis to find Value
Scale with Demand via the Cloud
Go Make Fun!

25. (novel is hiring)
•
•
•
•
•
•
Gameplay Engineers (UI/UX, Puzzles, Engine, Etc.)
Server & Service Engineers (Analytics, Services, Database)
IT/Test/Build/Deployment Engineers
Game & Web Designers (Content, Fun, Balance)
Business/Marketing/Executive
AS3, Java, C#, PHP, C/C++, Web/iOS/Android/WinMobile
http://www.novelincorporated.com/careers

26. Thank You
Paul J. Furio
Technical Director
Novel, Inc.
paul@novelincorporated.com