Many modern video games are constantly evolving post-release. New maps, game modes, and game balancing adjustments are rolled out, often on a weekly basis. This continuous iteration to improve player engagement and satisfaction requires data-driven decision making based on events and telemetry captured during gameplay, and from community forums and discussions.
In this session you will learn how OpenShift Streams for Apache Kafka and Kafka Streams can be used to analyze real-time events and telemetry reported by a game server, using a practical example that encourages audience participation. Specifically youâll learn how to:
Provision Kafka clusters on OpenShift Streams for Apache Kafka.
Develop a Java application that uses Kafka Streams and Quarkus to process event data.
Deploy the application locally, or on OpenShift and connect it to your OpenShift Streams for Apache Kafka Cluster.
Level-up your gaming telemetry using Kafka Streams | DevNation Tech Talk
1. CONFIDENTIAL designator
An Exploration of the Kafka Streams API
Processing Game Telemetry with
OpenShift Streams for Apache Kafka
Evan Shortiss
Technical Marketing Manager
1
@evanshortiss
3. CONFIDENTIAL designator
3
Apache Kafka is a distributed system designed for streams. It
is built to be an high-availability, horizontally-scalable,
fault-tolerant, commit log, and allows distributed data
streams and stream processing applications.
Known use cases are:
⸠High-performance data pipelines
⸠Streaming analytics
⸠Data integration
What is Kafka?
What is Apache Kafka?
Open-source distributed event streaming platform
Learn more on the Apache Kafka
community page:
https://kafka.apache.org/
5. CONFIDENTIAL designator
Source:
https://kafka.apache.org/intro
What is Kafka?
5
APIs
What is Apache Kafka?
⸠The Admin API to manage and inspect topics, brokers, and other Kafka objects.
⸠The Producer API to publish (write) a stream of events to one or more Kafka topics.
⸠The Consumer API to subscribe to (read) one or more topics and to process the
stream of events produced to them.
⸠The Kafka Streams API to implement stream processing applications and
microservices.
⸠The Kafka Connect API to build and run reusable data import/export connectors
that consume (read) or produce (write) streams of events from and to external
systems and applications so they can integrate with Kafka.
6. CONFIDENTIAL designator
Kafka & Video Game Telemtry
6
Use Cases
Kafka and Video Game Telemetry
⸠Modern video games are constantly evolving post-release.
⸠Developers release patches with new features and game
balance fixes.
⸠Data-driven decision making can increase the value of each
iteration.
⸠Popular video games have millions of daily active users.
⸠Generate millions/billions of telemetry events and data.
⸠Kafka, Kafka Streams, and the broader ecosystem can be used
for ingestion and processing data!
Source:
https://steamcharts.com (July 14th 2021)
600,000 Counter-Strike players firing just 20 shots could
generate 12,000,000 unique events!
7. CONFIDENTIAL designator
Shipwars, a Battleship style
game where players
face-off against AI
opponents.
First to sink opponent
ships is the winner.
Bonus round provides an
opportunity to boost
high-scores.
7
Kafka at Red Hat Summit 2021
Red Hat Summit 2021
9. CONFIDENTIAL designator
Scoring API
Kafka at Red Hat Summit 2021
9
Game Server
WebSocket
Game State
Players
Matches
Scores
Binary
Cloud Event
Broker
HTTP
Cloud Events
Scoring/Bonus APIs
HTTP
Cloud Events
Binary
Strimzi Kafka
3 Clusters, Open Hybrid Cloud
AWS, Azure, and GCP
AI Move REST API
HTTP Request
Next Move
Bot State Machines
HTTP Request
Create AI Bot Player
WebSocket
10. CONFIDENTIAL designator
Kafka at Red Hat Summit 2021
10
3 Clusters, Open Hybrid Cloud
AWS, Azure, and GCP
Search for Katia Arestiâs content about the
leaderboard and Infinispan replication on Red Hat
Developers YouTube and Blog!
11. CONFIDENTIAL designator
Scoring API
Kafka at Red Hat Summit 2021
11
Game Server
WebSocket
Game State
Players
Matches
Scores
Binary
Cloud Event
Broker
Scoring/Bonus APIs
HTTP
Cloud Events
Binary
Kafka
Leaderboard
Binary
WebSocket
3 Clusters, Open Hybrid Cloud
AWS, Azure, and GCP
HTTP
Cloud Events
AI Move REST API
HTTP Request
Next Move
Bot State Machines
HTTP Request
Create AI Bot Player WebSocket
23. CONFIDENTIAL designator
Red Hat OpenShift Streams for Apache Kafka
Complete solution for stream-based applications
Product Overview
23
Metrics
&
monitoring
Configuration
mgmt
Hosted & managed (99.95% SLA)
Kafka cluster
Broker(s)
Topic(s)
Streamlined developer experience: a curated solution
with a developer-first, consistent experience
Delivered as a service, managed by Red Hat SRE -
24x7 global support and a 99.95% service-level
agreement (SLA)
Real-time, streaming data broker - Dedicated Apache
Kafka cluster deliver as a service in the cloud and
location of choice
⸠Access to Kafka brokers, topics, and partitions
⸠Configuration management
⸠Metrics and monitoring
⸠UI / CLI / API / service bindings
⸠Integrated identity & access management
STREAMLINED DEVELOPER EXPERIENCE
UI API
CLI Service Binding
https://red.ht/TryKafka
24. CONFIDENTIAL designator
24
⸠Based on the Strimzi community project that provides an
optimized version of Apache Kafka on Kubernetes
⸠Access to customer dedicated Kafka instances. Each
instance provides Kafka core capabilities
⸠Kafka instances are configured based on opinionated
recommendations made by Red Hat experts
Product Overview
A dedicated managed Apache Kafka cluster
An opinionated experience for your Kafka service
Apache Kafka
implementation
optimized for OpenShift
https://red.ht/TryKafka
27. CONFIDENTIAL designator
End the Keynote demo with a
twist.
Prove that we can track all
events throughout the game!
27
Introduction
Red Hat Summit 2021
ALL YOUR BASE EVENT ARE BELONG
TO US.
Source:
Zero Wing by Namco (1989) and Williams Electronics (1990)
28. CONFIDENTIAL designator
Record player âconnectâ
events. Each player and
match is unique.
Capture each âattackâ by
each player.
Join these events together
to create a complete record
of each match.
28
Introduction
Aggregate Match
Visualisation
29. CONFIDENTIAL designator
Record player âconnectâ
events and player type.
Capture each âattackâ by
each player.
Create a matrix using this
data to understand where
humans tend to shoot or
place their ships.
Subscribe the AI model to
this real-time data so it
continuously learns.
29
Introduction
Real-time AI
Adjustment
https://github.com/redhat-gamedev/shipwars-move-server/blob/master/prediction.py
30. CONFIDENTIAL designator
Game Server
Game Server
Kafka Streams Topology
30
Game Server
WebSocket
Attacks Topic
Key Value (Player,Result, Cell)
GameVer:MatchA PlayerA,Hit [0,1]
GameVer:MatchA PlayerB,Hit [2,3]
GameVer:MatchA PlayerA,Hit [1,2]
GameVer:MatchB PlayerD,Miss [4,1]
Players Topic
Key Value (Name, Type, Region)
PlayerA Slow Bow, Human, EMEA
PlayerB Iron Ocelot, AI, EMEA
PlayerC Plume Boar, Human, APAC
PlayerD Rose Fang, AI, APAC
31. CONFIDENTIAL designator
Game Server
Game Server
Kafka Streams Topology
31
Game Server
WebSocket
Attacks Topic
Key Value (Player,Result, Cell)
GameVer:MatchA PlayerA,Hit [0,1]
GameVer:MatchA PlayerB,Hit [2,3]
GameVer:MatchA PlayerA,Hit [1,2]
GameVer:MatchB PlayerD,Miss [4,1]
Source:
Super Mario 3D World via https://mario.fandom.com
Players Topic
Key Value (Name, Type, Region)
PlayerA Slow Bow, Human, EMEA
PlayerB Iron Ocelot, AI, EMEA
PlayerC Plume Boar, Human, APAC
PlayerD Rose Fang, AI, APAC
32. CONFIDENTIAL designator
Game Server
Game Server
Kafka Streams Topology
32
Game Server
WebSocket
Attacks Topic
Key Value (Player,Result, Cell)
GameVer:MatchA PlayerA,Hit [0,1]
GameVer:MatchA PlayerB,Hit [2,3]
GameVer:MatchA PlayerA,Hit [1,2]
GameVer:MatchB PlayerD,Miss [4,1]
Source:
Super Mario 3D World via https://mario.fandom.com
Players Topic
Key Value (Name, Type, Region)
PlayerA Slow Bow, Human, EMEA
PlayerB Iron Ocelot, AI, EMEA
PlayerC Plume Boar, Human, APAC
PlayerD Rose Fang, AI, APAC
Kafka Streams API
36. CONFIDENTIAL designator
Game Server
Game Server
Kafka Streams Topology
36
Game Server
WebSocket
players
attacks
enriched-attacks
match-aggregates
Produce
Produce
GlobalKTable
KStream
Consume
Consume
Join on
Player ID
Produce
KStream
Consume
Update
Aggregate
Record
Group By
Key
(Game Ver.
& Match ID)
Produce
Kafka Streams: Join
Kafka Streams: Aggregation
Kafka Topics
Consume
37. CONFIDENTIAL designator
Replay Service
Each Game Server instance
produces shots to the Managed
Kafka Cluster
Kafka Streams Topology
37
Game Server
WebSocket
Produce
Consume
Records
players & attacks
match-aggregates
Produce
Match Aggregates
Query RocksDB
via HTTP API
Render Match Replay
in Web Browser
OpenShift Dedicated
Single Region Processing Cluster
OpenShift Dedicated
3 Region Game Clusters
Kafka Streams
Single Region
38. CONFIDENTIAL designator
1. Define Serdes.
2. Create shots KStream.
3. Create players GlobalKTable.
4. Join stream and table using
composite key.
5. Return new record (String)
that contains player type, shot
result, and shot cell.
38
Kafka Streams Topology
Kafka Streams: Join
39. CONFIDENTIAL designator
1. Define Serde.
2. Enriched shots KStream.
3. Group by key.
4. Aggregate grouped records.
5. Materialise record into store.
6. Write record to aggregate
topic.
39
Kafka Streams Topology
Kafka Streams: Aggregate