In this workshop, we will together build telemetry/analytics data processing pipelines to assist game developers/architects, designers and producers. We will use a fictitious RPG and ingest data from in-game events. We will then analyze the data to help with game balancing, troubleshooting and other relevant recommendations for game developers and designers. As a participant, you will use Amazon Kinesis, Amazon Kinesis Firehose, Amazon Analytics, Amazon EMR, Amazon Redshift, Amazon S3, Amazon Athena and Amazon QuickSight. Prerequisites include having your own laptop and an interest in big data services, game data processing & analytics. In order to make your time in the workshop as productive as possible, please make sure to check out the additional information below. AWS account: Fully functional AWS Account with administrative access. Participant should have the ability to create & destroy resources in the us-west-2 and eu-west-1 regions via API, CLI & AWS Console. Device/OS: A laptop computer – running Mac OS X, a Linux flavor or Windows. The computer will need a functional ssh/Remote Desktop client. AWS service familiarity/experience:Familiarity/Experience with EC2, S3 & the AWS Console will be good. For the rest of the services, we will introduce each during the workshop. Audience: Game Developers (server programmers), Architects, Game Producers/Designers, Game Marketing/Analytics team – hands-on members

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Build a Telemetry and Analytics
Pipeline for Game Balancing
Nirav Doshi - niravdd@amazon.com - Sr. Technical Account Manager, AWS ANZ
Brent Maxwell - braxwell@amazon.com - Solutions Architect, AWS ANZ
G A M 3 1 0
N o v e m b e r 2 7 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Workshop Agenda (2 hours, 30 mins)
Prerequisites, preparation, setup
Game data, analytics
The game
The bot
1. Analytics pipeline (45 mins)
2. Serverless analytics pipeline (15 mins)
3. Analytics on the stream (20 mins)
4. Big data analytics (40 mins)
Learnings, Q&A

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Prerequisites, Preparation & Setup
AWS account and admin access setup
The instructional guide
- amzn.to/GAM310
- Start with the prerequisites section
- Keep your ACCESS_KEY/SECRET_KEY handy
Experimentation encouraged!… however -
- Follow the instructions precisely, avoid digressing
- Try out options, only when asked to
Workshop support

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Game Data & Analytics
Real-time player and game data
- Game balancing - Marketing
- Fraud investigation - Debugging, other
Gaming companies & data crunching
Common practices Challenges
Self-managed large on-
premises data crunching
environments
- Scalability
- Spiraling Costs (Infra, Resources)
Third-party products and
services
- Data Ownership/Restrictions
- Security

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Game Data & Analytics
Harvest your data!
Using flexible, extensible, managed solutions
Ingest — Store — Analyze — Consume
Interact with your data any way you like
Pick and choose tools, customize interactions
Leverage the scalability of the cloud
Take ownership of your data in a secure manner

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Game
Fictitious role-playing game
Game client emits data
- Player demographics
- Player unit—attributes
- Enemy units—attributes

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Bot
Simple data generator*
- How a real-time streaming app behaves?
- KPL, faker.js
- Game-play data is generated by the bot
Game-play data constitutes
- Player demographics
- Game-play data – game level, squad stats, boss stats, result
* Thanks to Bashar Al-fallouji (AWS Solutions Architect, Sydney) for his excellent bot!

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1. Analytics Pipeline

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Purpose & Outcome
Basic pipeline, easily customizable
Similar to existing analytics products/services in the market
Store raw stream data to Amazon Redshift and visualize
Game balancing, analysis

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Streams
For developers
Collect and stream data for
ordered, re-playable, real-
time processing
Amazon Kinesis Firehose
For developers, data scientists
Easily load massive volumes
of streaming data into
Amazon S3, Amazon Redshift
or Amazon EMR
Amazon Kinesis Analytics
For developers, data scientists
Easily analyze data streams
using standard SQL queries in
real-time
Amazon Kinesis

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Streams
Storage
AWS Global Infrastructure
App services
Deployment and administration
Networking
Analytics
Compute Database
• Ingest streaming data
• Process data in real-time
• Stream terabytes of data per hour

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Streams
Store data as a continuous stream
• Easy administration: simply create a new stream and set the desired level of capacity with shards. Scale to
match your data throughput rate and volume
• Build real-time applications: perform continual processing on streaming big data using Amazon Kinesis
Client Library (KCL), Apache Spark/Storm, AWS Lambda, and more
• Low cost: cost-efficient for workloads of any scale
Send clickstream data
to Kinesis Streams
Kinesis Streams stores and
exposes clickstream data
for processing
Custom application built
on Kinesis Client Library
makes real-time content
recommendations
Readers see personalized
content suggestions

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Load massive volumes of streaming data into destinations
• Zero administration: capture and deliver streaming data into Amazon Simple Storage Service (S3), Amazon
Redshift, and other destinations without writing an application or managing infrastructure
• Direct-to-data store integration: batch, compress, and encrypt streaming data for delivery into data
destinations in as little as 60 seconds using simple configurations
• Seamless elasticity: seamlessly scale to match data throughput without intervention
Amazon Kinesis Firehose
Capture and submit
streaming data to Kinesis
Firehose
Kinesis Firehose loads streaming
data continuously into Amazon S3
and Amazon Redshift
Analyze streaming data
using your favourite BI Tools

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage
AWS Global Infrastructure
App services
Deployment and administration
Networking
Compute Database
Analytics
Amazon S3
• Scalable object storage
• 1 byte to 5 TB in size per object,
unlimited number of objects
• 99.999999999% durability, 99.9%
availability
• Regional service, no single points
of failure
• Server Side Encryption (SSE)

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deployment and administration
Compute Storage
AWS Global Infrastructure
Database
App services
Networking
Analytics
Amazon Redshift
• Managed massively parallel
petabyte scale data warehouse
• Streaming backup/restore to
Amazon S3
• Load data from Amazon S3, Amazon
DynamoDB and Amazon EMR
• Extensive security features
• Online scaling from 160 GB to 2 PB

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage
AWS Global Infrastructure
App services
Deployment and administration
Networking
Analytics
Compute Database
Amazon QuickSight
• BI service, perform impromptu
analysis
• Build visualizations
• Share and collaborate via
storyboards
• Native access on major mobile
platforms

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Analytics Pipeline

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Analytics Pipeline
amzn.to/GAM310

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2. Serverless Analytics Pipeline

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Purpose & Outcome
Use Amazon S3 for data storage and Amazon Athena to query the data
Powerful and easy tool for data scientists, game designers, game
producers, and server developers
Perform standard SQL queries on stored data
You may also use Amazon Elasticsearch (Amazon ES) and Kibana to process
and visualize data as part of another serverless analytics pipeline
Suitable for server/client log processing and querying

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage
AWS Global Infrastructure
App services
Deployment and administration
Networking
Analytics
Compute Database
Amazon Athena
• Query and analyze data stored on
Amazon S3 with standard (ANSI)
SQL queries
• No ETL required
• Serverless and simple
• Pay per query, on the amount of
data scanned

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage
AWS Global Infrastructure
App services
Deployment and administration
Networking
Analytics
Compute Database
Amazon Elasticsearch Service
• Setup an Amazon Elasticsearch
Service cluster in minutes
• Integrated with Logstash and
Kibana
• Scale Amazon ES clusters
seamlessly

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Serverless Analytics Pipeline
amzn.to/GAM310

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
3. Analytics on the Stream

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Purpose & Outcome
Execute real-time SQL queries on the streaming data
Create alerts, get notified, and act/respond quicker

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• SQL queries on streams: easily connect to a Kinesis stream or Firehose delivery stream and run SQL
queries against the data
• Build real-time applications: perform continual processing on streaming big data with sub-second
processing latencies
• Easy scalability: elastically scales to match data throughput
Amazon Kinesis Analytics
Connect to Kinesis Streams,
Kinesis Firehose delivery
streams
Run standard SQL queries against
the data streams
Kinesis Analytics can send
processed data to analytics
tools so you can create
alerts and act in real time

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Analytics on the Stream
amzn.to/GAM310

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
4. Big Data Analytics

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Purpose & Outcome
Process large amount of data using a Hadoop cluster (Amazon EMR)
Analyse processed data to investigate game imbalance

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage
AWS Global Infrastructure
App services
Deployment and administration
Networking
Analytics
Compute Database
Amazon EMR
• Scalable Hadoop/Spark clusters as
a service
• Launch a cluster in minutes
• Easy to use: fully managed
• HDFS, Amazon EBS, and Amazon
S3 file systems

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scalability and elasticity
- Resize a running cluster based on work to be
done
Durability and availability
- Fault-tolerant for slave node (HDFS)
- Backup to Amazon S3 for resilience against
master node failures
Standard interfaces
- Hive, Pig, Spark, HBase, Impala, Hunk,
Presto, Zeppelin, and other popular tools
Amazon EMR
Amazon EMR cluster
Amazon EMR cluster
Amazon EMR cluster

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Big Data Analytics
amzn.to/GAM310

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Learnings
Takeaways:
• Set up and take control of your flexible, extensible, and secure analytics pipeline
• Ingest, process, and refine own game-play data to improve game operations
• Extract powerful insights from your own data, act quicker
• Make the best of streaming tech and managed services for smarter operations

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
Workshop team:
Nirav Doshi - niravdd@amazon.com
Brent Maxwell - braxwell@amazon.com
Keith Lafaso - lafaso@amazon.com
Beomseok Lee - beomseok@amazon.com