(Kalah Brown, Big Fish Games) Kafka Summit SF 2018 Big Fish Games is a leading producer and distributor of casual and mid-core games. They have distributed more than 2.5 billion games to customers in 150 countries, representing over 450 unique mobile games and over 3,500 unique PC games. Apache Kafka is used in our pipeline to process data generated across game play. Recently, we introduced real-time analytics of game data using Kafka Streams integrated with Elasticsearch. This allows us to monitor the results of live operations (aka, live ops such as weekend events, limited time offers and user acquisition campaigns) and to make changes to these events after they have gone live. The result is a better game experience, better control of the game’s economy and overall better optimization of the live ops. This presentation will include a detailed explanation of how we used Kafka Streams to transform raw data into Elasticsearch documents, and how the application was scaled to over a million daily active users. It will also touch on the limitations discovered with Kafka Connect integration with Elasticsearch and how to use Elasticsearch bulk processing with Kafka Streams. The presentation will also discuss how Dropwizard can provide a framework for monitoring and alerting a Kafka Stream application. Lastly, a demonstration of the real-time dashboards which are powered by Kafka Streams will be included.

1. How Big Fish Games Developed Real-Time Analytics Using
Kafka Streams and Elastic Search

2. 2 Big Fish Confidential
Big Fish Games
o Leading producer and distributor of casual
games.
o 2.5 billion games distributed to customers in
150 countries
o 450 unique mobile games
o 3500 unique PC games

3. 3 Big Fish Confidential
Cooking Craze

4. 4 Big Fish Confidential
Live Operations Real Time Dashboard
Let’s Dish

5. 5 Big Fish Confidential
Live Ops and Free to Play Games
o Monetization through “in app” purchases
o Games evolve and are extended indefinitely
o Live Operations
o Games which allow real-time updates to
improve the game or align to audiences
needs

6. 6 Big Fish Confidential
Monitoring a Forced Update

7. 7 Big Fish Confidential
Monitoring New Content

8. 8 Big Fish Confidential
Monitoring Event Participation

9. 9 Big Fish Confidential
Kafka at Big Fish

10. 10 Big Fish Confidential
Let’s Dish Dashboards
Data Sources
o Game Telemetry Events from a Kafka Topic
o restaurant leave , restaurant join
o session start, session foreground, session
background, session heartbeat
o recipe complete, recipe burn
o Hive Data
o Install Date
o bookings , bookings Last30
o channel Type
o bfgudid

11. 11 Big Fish Confidential
Elastic Search Document{
"_index": "letsdish-2018-19",
"_type": "recipes",
"_id": "f4858a61-cf4e-400c-952e-6e940774e1ef",
"_score": 1,
"_source": {
"count": 1,
"restaurantScheduleId": "french",
"restaurantsAvailable": 5,
"installDate": "2018-05-09",
"bookingsBin": "No Monetization",
"bookingsBinLast30": "No Monetization",
"channelType": "Organic",
"environment": "prod",
"appName": "LetsDish",
"bfgudid": "f8e1d52b7a5edb26aafed418e2a25c6879766f9c",
"platform": "android",
"customUtc": 1526078368,
"countryCode": "US",
"appVersion": "42",
"guid": "f4858a61-cf4e-400c-952e-6e940774e1ef“
(…)
},
"fields": {
"customUtc": [
"2018-05-11T22:39:28.000Z"
],
"installDate": [
"2018-05-09T00:00:00.000Z"
]
}
}

12. 12 Big Fish Confidential
Two Streams APIs
o Streams DSL
o Abstraction of Processor API
o Used for Let’s Dish Dashboards
o Streams Processor API
o More flexible
o Finer control of stream processing

13. 13 Big Fish Confidential
Let’s Dish Streaming Transformation
Topology

14. 14 Big Fish Confidential
Mapping, Filtering and
Branching

15. 15 Big Fish Confidential
Map Function
KeyValueMapper<String, String,
<String,GtEvent>>
Maps JSON String Value to Game
Telemetry Pojo: GtEvent

16. 16 Big Fish Confidential
Java 8 Lambda Function
Map Function
public static KeyValueMapper<String, String,
KeyValue<String, GtEvent>>
mapStringToGtEvent = (k, v) ->
{
GtEvent gtEvent;
try {
gtEvent = new Gson().fromJson(v,GtEvent.class);
} catch (Exception e) {
gtEvent = new CustomEvent();
} return
new KeyValue<String, GtEvent>(k, gtEvent);
};

17. 17 Big Fish Confidential
Filter Predicates
KStreams filter and branch methods
take a Predicate interface
KStream.filter(Predicate)
KStream.branch(Predicate[])

18. 18 Big Fish Confidential
Java 8 Lambda Functions
Filter Predicates
public static Predicate<String, GtEvent>
isRecipeComplete = (k, v) ->
"recipe_complete".equals(
v.data.eventName.toLowerCase());
public static Predicate<String, GtEvent>
isSession = (k, v) ->
Arrays.asList("session_start",
"session_foreground",
"session_background",
"session_heartbeat")
.contains(v.data.eventName.toLowerCase());

19. 19 Big Fish Confidential
Map and Filters Combined
public static KStream<String, GtEvent>[] createBranchedStreams(
KStream<String, String> source) {
return source
.map(Mappers.mapStringToGtEvent)
.branch(
FilterPredicates.isRecipeComplete,
FilterPredicates.isRecipeBurn,
FilterPredicates.isSession,
FilterPredicates.isRestaurant);
}

20. 20 Big Fish Confidential
Four KStreams
o Recipe Burn
o Recipe Complete
o Restaurant Unlocked
o Restaurant Names

21. 21 Big Fish Confidential
Aggregated Player Purchases Table
Enhance KStreams with
Hive Table
ID FIRST
INSTALL
DATE
BOOKINGS
BIN
BOOKINGS LAST
30 DAYS
CHANNEL TYPE
1 9/12/2018 $1 <= $5 $1 <= $5 paid
2 9/12/2018 $5 <= $20 $5 <= $20 organic
3 9/12/2018 $1 <= $5 $1 <= $5 paid

22. 22 Big Fish Confidential
Kafka Topic

23. 23 Big Fish Confidential
Joining Streams

24. 24 Big Fish Confidential
Types of Joins
o Windowed Join (No State)
o KStream->KStream
o Non-Windowed Joins (State Maintained )
o KStream->KTable
o KTable->KTable

25. 25 Big Fish Confidential
Using Group By Key
Creating a KTable from Kstream
.groupByKey(Serialized.with(stringSerde,longSerde))
.reduce((oldValue, newValue) ->
Math.max(oldValue, newValue),
Materialized.<String, Long,
KeyValueStore<Bytes, byte[]>>
as("restaurants-unlocked-store")
.withKeySerde(stringSerde)
.withValueSerde(longSerde));

26. 26 Big Fish Confidential
Recipe Burn Stream to Restaurant Unlocked KTable
Steps to Joining
recipeBurnKStream.leftJoin(
restaurantUnlockedKTable,
RecipeValueJoiners.joinRecipeToRestaurantUnlocked,
Joined.with(
stringSerde,enrichedEventSerde,longSerde)
);

27. 27 Big Fish Confidential
Where are the join conditions?
Join Keys
KStream<String, EnrichedEvent> recipeBurnKStream
Joined with
KTable<String, Long> restaurantUnlockedKTable
on the Keys.

28. 28 Big Fish Confidential
Value Joiner
public static ValueJoiner<EnrichedEvent, Long, EnrichedEvent>
joinRecipeToRestaurantUnlocked =
(e, restaurantsUnlocked) -> {
e.enrichedRecipeEvent.restaurantsUnlocked =
(restaurantsUnlocked == null) ?
-1L : restaurantsUnlocked;
return e;
};

29. 29 Big Fish Confidential
• EnrichedEventSerde
• org.apache.kafka.common.serialization
interfaces
Custom Serialization
Joined.with(stringSerde,enrichedEventSerde
,longSerde)

30. 30 Big Fish Confidential
Join Recap
o Keys are used for joining
o KStream<String, EnrichedEvent>
o KTable<String, Long>
o Value Joiners
o Return the number of restaurants
unlocked
o If null return -1
o Custom Serde for EnrichedEvent

31. 31 Big Fish Confidential
Elastic Search Document
{
"_index": "letsdish-2018-19",
"_type": "recipes",
"_id": "f4858a61-cf4e-400c-
952e-6e940774e1ef",
"_score": 1,
"_source": {
"count": 1,
"restaurantScheduleId": "fr
ench",
"restaurantsAvailable": 5,
"installDate": "2018-05-
09",
"bookingsBin": "No
Monetization",
"bookingsBinLast30": "No
Monetization",
…

32. 32 Big Fish Confidential
Elastic Search Bulk API
o Bulk Processing Greatly Increases the Index
Speed
o Two possible implementations of the Bulk
API
o Direct calls to the REST service
o Elastic Search Libraries
o BulkProcessor and RestClientBuilder
o Wrap lower level work

33. 33 Big Fish Confidential
Continue to Build Real Time
Dashboards for Games