Playing with Xbox Data with Dale Lane

•

0 gefällt mir•623 views

"A fun introduction to the world of Kafka Connect and Kafka Streams by using it to process data from Xbox Live. Gaming is a social activity, so Xbox includes a social aspect. Details about what games you play and when you’re playing are shared with your friends on the Xbox Live service. Achievements that you earn and complete in games are shared with your friends. This makes for a fun source of data, that we can play with using Kafka Connect and Kafka Streams. 1) Getting the data into Kafka — I’ll show how I’m using Kafka Connect to surface data from Xbox REST APIs as streams of events on Kafka topics. 2) Processing the events — I’ll show how I’m using Kafka Streams to process the Xbox events as they’re emitted - such as cleaning up some of the noisy events with filters, enriching events with additional context using transformations, and creating gamerscore leaderboards using aggregations. As a project, it is just a bit of fun. But it is a useful exercise for illustrating how to get started with Kafka Connect and Kafka Streams, and a way to demonstrate the principles using a real source of real events, rather than using fake or generated events. Plus it is a good excuse for me to spend more time gaming, and justifying it as work!"

Technologie

Using Xbox to get started with
Kafka Connect & Kafka Streams
Dale Lane (IBM)

Who is this talk for?
Connect Streams
Gamers

Kafka Connect
Connectors include:
IBM MQ
HDFS
Elasticsearch
MySQL
JDBC
MQTT
CoAP
+ many others

SourceTask
Xbox LIVE Kafka topics
Xbox LIVE
Connector
Kafka Connect
poll()
REST API calls
to fetch events
game events
List<SourceRecord> produce

SourceTask
Xbox LIVE Kafka topics
Xbox LIVE
Connector
Kafka Connect
poll()
List<SourceRecord>

SourceTask
Xbox LIVE Kafka topics
Xbox LIVE
Connector
Kafka Connect
poll()

SourceTask
Xbox LIVE Kafka topics
Xbox LIVE
Connector
Kafka Connect

SourceTask
Xbox LIVE Kafka topics
Xbox LIVE
Connector
Kafka Connect
REST API calls
to fetch events
game events
poll()
List<SourceRecord>
in-memory queue

SourceTask
Xbox LIVE Kafka topics
Xbox LIVE
Connector
Kafka Connect
REST API calls
to fetch events
game events

Xbox LIVE Kafka topics
Xbox LIVE
Connector
Kafka Connect
poll()
List<SourceRecord> produce

SourceRecord
Source Records
Kafka topic
Kafka Connect
Xbox LIVE
Connector
SourceRecord
SourceRecord

Source Records
Kafka topic
Kafka Connect
Xbox LIVE
Connector
SourceRecord
topic

Source Records
Kafka topic
Kafka Connect
Xbox LIVE
Connector
SourceRecord
topic
message

Source Records
Kafka topic
Kafka Connect
Xbox LIVE
Connector
SourceRecord
topic
message
offset
offsets topic

Startup
box LIVE Kafka topic
Xbox LIVE
Connector
Kafka Connect
poll()
REST API calls
to fetch events
Offsets

Startup
Xbox LIVE Kafka topics
Xbox LIVE
Connector
Kafka Connect

Startup
Xbox LIVE Kafka topics
Xbox LIVE
Connector
Kafka Connect
restart

Kafka Streams • Filtering
• Splitting
• Enriching
• Matching
• Counting
• Summing
• Tracking

Thank you!
https://github.com/dalelane/kafka-connect-xboxlive-source
https://github.com/dalelane/kafka-streams-xboxlive
IBM Event Streams
Apache Kafka
@dalelane@mastodon.org.uk
@dalelane
https://dalelane.co.uk/blog

Empfohlen

Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi,...Chris Fregly

Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!confluent

2017 meetup-apache-kafka-novFlorian Hussonnois

What is Apache Kafka®?Eventador

What is apache Kafka?Kenny Gorman

Apache Kafka, and the Rise of Stream ProcessingGuozhang Wang

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...Michael Noll

Streaming Microservices With Akka Streams And Kafka StreamsLightbend

Empfohlen

Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi,...Chris Fregly

Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!confluent

2017 meetup-apache-kafka-novFlorian Hussonnois

What is Apache Kafka®?Eventador

What is apache Kafka?Kenny Gorman

Apache Kafka, and the Rise of Stream ProcessingGuozhang Wang

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...Michael Noll

Streaming Microservices With Akka Streams And Kafka StreamsLightbend

Akka Streams And Kafka Streams: Where Microservices Meet Fast DataLightbend

A 7 architecture sustaining line liveLINE Corporation

What is Apache Kafka and What is an Event Streaming Platform?confluent

Moving from Big Data to Fast Data? Here's How To Pick The Right Streaming EngineLightbend

Apache Spark Structured Streaming + Apache Kafka = ♡Bartosz Konieczny

Big Data LDN 2018: STREAMING DATA MICROSERVICES WITH AKKA STREAMS, KAFKA STRE...Matt Stubbs

KSQL Deep Dive - The Open Source Streaming Engine for Apache KafkaKai Wähner

Battle of NoSQL stars: Amazon's SDB vs MongoDB vs CouchDB vs RavenDBJesse Wolgamott

Paris jug ksql - 2018-06-28Florent Ramiere

Kick Your Database to the CurbBill Bejeck

扩展世界上最大的图片Blog社区yiditushe

Fotolog: Scaling the World's Largest Photo Blogging Communityfarhan "Frank" mashraqi

Improving Streams Scalability with Transactional StateStores (KIP-892)HostedbyConfluent

Application Development and Data Modeling on Amazon DynamoDBAmazon Web Services Japan

MLOps with a Feature Store: Filling the Gap in ML InfrastructureData Science Milan

KSQL – An Open Source Streaming Engine for Apache KafkaKai Wähner

Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...confluent

Tableau ArchitectureKishore Chaganti

Hands On With Spark: Creating A Fast Data Pipeline With Structured Streaming ...Lightbend

Leveraging Kafka Eco-System For Redis Streams: Sripathi KrishnanRedis Labs

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent

Weitere ähnliche Inhalte

Ähnlich wie Playing with Xbox Data with Dale Lane

Akka Streams And Kafka Streams: Where Microservices Meet Fast DataLightbend

A 7 architecture sustaining line liveLINE Corporation

What is Apache Kafka and What is an Event Streaming Platform?confluent

Moving from Big Data to Fast Data? Here's How To Pick The Right Streaming EngineLightbend

Apache Spark Structured Streaming + Apache Kafka = ♡Bartosz Konieczny

Big Data LDN 2018: STREAMING DATA MICROSERVICES WITH AKKA STREAMS, KAFKA STRE...Matt Stubbs

KSQL Deep Dive - The Open Source Streaming Engine for Apache KafkaKai Wähner

Battle of NoSQL stars: Amazon's SDB vs MongoDB vs CouchDB vs RavenDBJesse Wolgamott

Paris jug ksql - 2018-06-28Florent Ramiere

Kick Your Database to the CurbBill Bejeck

扩展世界上最大的图片Blog社区yiditushe

Fotolog: Scaling the World's Largest Photo Blogging Communityfarhan "Frank" mashraqi

Improving Streams Scalability with Transactional StateStores (KIP-892)HostedbyConfluent

Application Development and Data Modeling on Amazon DynamoDBAmazon Web Services Japan

MLOps with a Feature Store: Filling the Gap in ML InfrastructureData Science Milan

KSQL – An Open Source Streaming Engine for Apache KafkaKai Wähner

Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...confluent

Tableau ArchitectureKishore Chaganti

Hands On With Spark: Creating A Fast Data Pipeline With Structured Streaming ...Lightbend

Leveraging Kafka Eco-System For Redis Streams: Sripathi KrishnanRedis Labs

Ähnlich wie Playing with Xbox Data with Dale Lane (20)

Akka Streams And Kafka Streams: Where Microservices Meet Fast Data

A 7 architecture sustaining line live

What is Apache Kafka and What is an Event Streaming Platform?

Moving from Big Data to Fast Data? Here's How To Pick The Right Streaming Engine

Apache Spark Structured Streaming + Apache Kafka = ♡

Big Data LDN 2018: STREAMING DATA MICROSERVICES WITH AKKA STREAMS, KAFKA STRE...

KSQL Deep Dive - The Open Source Streaming Engine for Apache Kafka

Battle of NoSQL stars: Amazon's SDB vs MongoDB vs CouchDB vs RavenDB

Paris jug ksql - 2018-06-28

Kick Your Database to the Curb

扩展世界上最大的图片Blog社区

Fotolog: Scaling the World's Largest Photo Blogging Community

Improving Streams Scalability with Transactional StateStores (KIP-892)

Application Development and Data Modeling on Amazon DynamoDB

MLOps with a Feature Store: Filling the Gap in ML Infrastructure

KSQL – An Open Source Streaming Engine for Apache Kafka

Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...

Tableau Architecture

Hands On With Spark: Creating A Fast Data Pipeline With Structured Streaming ...

Leveraging Kafka Eco-System For Redis Streams: Sripathi Krishnan

Mehr von HostedbyConfluent

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent

Evolution of NRT Data Ingestion Pipeline at TrendyolHostedbyConfluent

Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesHostedbyConfluent

Exactly-once Stream Processing with Arroyo and KafkaHostedbyConfluent

Fish Plays Pokemon | Kafka Summit LondonHostedbyConfluent

Tiered Storage 101 | Kafla Summit LondonHostedbyConfluent

Building a Self-Service Stream Processing Portal: How And WhyHostedbyConfluent

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...HostedbyConfluent

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...HostedbyConfluent

Navigating Private Network Connectivity Options for Kafka ClustersHostedbyConfluent

Apache Flink: Building a Company-wide Self-service Streaming Data PlatformHostedbyConfluent

Explaining How Real-Time GenAI Works in a Noisy PubHostedbyConfluent

TL;DR Kafka Metrics | Kafka Summit LondonHostedbyConfluent

A Window Into Your Kafka Streams Tasks | KSLHostedbyConfluent

Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceHostedbyConfluent

Data Contracts Management: Schema Registry and BeyondHostedbyConfluent

Code-First Approach: Crafting Efficient Flink AppsHostedbyConfluent

Debezium vs. the World: An Overview of the CDC EcosystemHostedbyConfluent

Beyond Tiered Storage: Serverless Kafka with No Local DisksHostedbyConfluent

Mehr von HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Renaming a Kafka Topic | Kafka Summit London

Evolution of NRT Data Ingestion Pipeline at Trendyol

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Exactly-once Stream Processing with Arroyo and Kafka

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

Building a Self-Service Stream Processing Portal: How And Why

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Navigating Private Network Connectivity Options for Kafka Clusters

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Explaining How Real-Time GenAI Works in a Noisy Pub

TL;DR Kafka Metrics | Kafka Summit London

A Window Into Your Kafka Streams Tasks | KSL

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Data Contracts Management: Schema Registry and Beyond

Code-First Approach: Crafting Efficient Flink Apps

Debezium vs. the World: An Overview of the CDC Ecosystem

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Kürzlich hochgeladen

Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun

Factors to Consider When Choosing Accounts Payable Services Providers.pptxKatpro Technologies

How to convert PDF to text with Nanonetsnaman860154

Histor y of HAM Radio presentation slidevu2urc

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo

Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer

Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays

Presentation on how to chat with PDF using ChatGPT code interpreternaman860154

Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1

Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge

Scaling API-first – The story of a global engineering organizationRadu Cotescu

The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j

Kürzlich hochgeladen (20)

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Data Cloud, More than a CDP by Matt Robison

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

How to convert PDF to text with Nanonets

Histor y of HAM Radio presentation slide

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Axa Assurance Maroc - Insurer Innovation Award 2024

Breaking the Kubernetes Kill Chain: Host Path Mount

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

How to Troubleshoot Apps for the Modern Connected Worker

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Presentation on how to chat with PDF using ChatGPT code interpreter

Boost Fertility New Invention Ups Success Rates.pdf

Driving Behavioral Change for Information Management through Data-Driven Gree...

Scaling API-first – The story of a global engineering organization

The Codex of Business Writing Software for Real-World Solutions 2.pptx

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Playing with Xbox Data with Dale Lane

1. Using Xbox to get started with Kafka Connect & Kafka Streams Dale Lane (IBM)

2. Who is this talk for? Connect Streams Gamers

4. https://dalelane.co.uk/blog/?p=4808

6. Kafka Connect Connectors include: IBM MQ HDFS Elasticsearch MySQL JDBC MQTT CoAP + many others

7. Kafka Connect

8. Xbox LIVE Kafka topics

10.

11.

12.

13.

14. Xbox LIVE Kafka topics

15. SourceTask Xbox LIVE Kafka topics

16. SourceTask Xbox LIVE Kafka topics Xbox LIVE Connector Kafka Connect poll() REST API calls to fetch events game events List<SourceRecord> produce

17. SourceTask Xbox LIVE Kafka topics Xbox LIVE Connector Kafka Connect poll() List<SourceRecord>

18. SourceTask Xbox LIVE Kafka topics Xbox LIVE Connector Kafka Connect poll()

19. SourceTask Xbox LIVE Kafka topics Xbox LIVE Connector Kafka Connect poll() List<SourceRecord>

20. xbl.io

21. SourceTask Xbox LIVE Kafka topics Xbox LIVE Connector Kafka Connect

22. SourceTask Xbox LIVE Kafka topics Xbox LIVE Connector Kafka Connect REST API calls to fetch events game events poll() List<SourceRecord> in-memory queue

23. SourceTask Xbox LIVE Kafka topics Xbox LIVE Connector Kafka Connect REST API calls to fetch events game events

24. SourceTask Xbox LIVE Kafka topics Xbox LIVE Connector Kafka Connect poll() List<SourceRecord>

25. Xbox LIVE Kafka topics Xbox LIVE Connector Kafka Connect poll() List<SourceRecord> produce

26. SourceRecord Source Records Kafka topic Kafka Connect Xbox LIVE Connector SourceRecord SourceRecord

27. Source Records Kafka topic Kafka Connect Xbox LIVE Connector SourceRecord topic

28.

29.

30.

31. Source Records Kafka topic Kafka Connect Xbox LIVE Connector SourceRecord topic

32. Source Records Kafka topic Kafka Connect Xbox LIVE Connector SourceRecord topic message

33. Source Records Kafka topic Kafka Connect Xbox LIVE Connector SourceRecord topic message offset offsets topic

34. Source Records Kafka topic Kafka Connect Xbox LIVE Connector SourceRecord topic message offset offsets topic

35. Startup box LIVE Kafka topic Xbox LIVE Connector Kafka Connect poll() REST API calls to fetch events Offsets

36. Startup Xbox LIVE Kafka topics Xbox LIVE Connector Kafka Connect

37. Startup Xbox LIVE Kafka topics Xbox LIVE Connector Kafka Connect restart

38. Startup Xbox LIVE Kafka topics Xbox LIVE Connector Kafka Connect restart

39. Startup box LIVE Kafka topic Xbox LIVE Connector Kafka Connect poll() REST API calls to fetch events Offsets

40. Source Records Kafka topic Kafka Connect Xbox LIVE Connector SourceRecord topic message offset offsets topic ...

41. Kafka Connect

42.

43.

44.

45. Kafka Streams

46. Filtering

47.

48.

49.

50.

51. Splitting

52.

53.

54.

55.

56.

57. Enriching

58.

59.

60.

61.

62. Matching

63.

64.

65.

66.

67.

68. Counting

69.

70.

71.

72.

73. Summing

74.

75.

76.

77.

78.

79. Tracking

80.

81.

82.

83.

84.

85.

86. Kafka Streams • Filtering • Splitting • Enriching • Matching • Counting • Summing • Tracking

87.

88.

89. zz85.github.io/kafka-streams-viz

90. Kafka Streams • Filtering • Splitting • Enriching • Matching • Counting • Summing • Tracking

91.

92. Thank you! https://github.com/dalelane/kafka-connect-xboxlive-source https://github.com/dalelane/kafka-streams-xboxlive IBM Event Streams Apache Kafka @dalelane@mastodon.org.uk @dalelane https://dalelane.co.uk/blog