Presentation by Tim Berglund

1. 1
@tlberglund
Four Streaming Data
Architectures

2. 2
The Old Story

3. 3

4. 4

5. 5

6. 6

7. 7
The Streaming Story

8. 8Maria-José Viñas, NASA Earth Science News Team.

9. 9Image credit: NASA/JPL/UCSD/JSC

10. 10

11. 11
Kafka

12. 12

13. 13

14. 14

15. 15
Examples

16. 16
Yelp
https://engineeringblog.yelp.com/2016/07/billions-of-messages-a-day-yelps-real-time-data-pipeline.html
Justin C., Software Engineer
(May, 2016)

17. 17
• 150 production services
• Metcalf’s Law
• REST scales poorly
• N+1 problem and bulk data services
A Streaming Backbone

18. 18
💩
💩
💩
💩
💩
💩
💩

19. 19

20. 20
• Stream-table duality
• But some services still want databases
• Individual services use MySQL
• Millions of rows across services
Streaming Databases

21. 21
• In 2010, commitment to MapReduce was strong
• That fun was soon over
• Want to capitalize on schema translation tools
• Want near-real-time results
• Storm didn’t support enough Python
• Wrote their own!
Computing Streams

22. 22
PaaStorm Instance

23. 23
Yelp Conclusions
• Streaming pipeline ensures evolvability
• Stream/Table Duality
• Schema matters
• Yelp often ahead of the game!

24. 24
LINE Corp
https://engineering.linecorp.com/en/blog/detail/80

25. 25
Goals
• Unified data delivery pipeline between systems
• Replace a legacy, bespoke system for processing background tasks
• Had to pick a framework

26. 26
Option: Apache Samza™

27. 27
Option: Apache Kafka™ Streams

28. 28
KStreamBuilder builder = new KStreamBuilder();
// Read the input Kafka topic into a KStream instance.
KStream<byte[], String> textLines = builder.stream("TextLinesTopic");
// Convert to upper case (:: is Java 8 syntax)
KStream<byte[], String> uppercasedWithMapValues = textLines.mapValues(String::toUpperCase);
// Write the results to a new Kafka topic called "UppercasedTextLinesTopic".
uppercasedWithMapValues.to("UppercasedTextLinesTopic");
KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration);
streams.start();
Streaming API for Kafka

29. 29
Streaming API for Kafka
Stateful Transformations

30. 30
Loopback Replicator
• One big input topic of all TalkOperations (1M messages/sec)
• Some services want a limited view of these
• Simple filtering

31. 31
Decaton
Source: LINE Corp

32. 32
LINE Conclusions
• Classical Kafka Streams implementation
• Better to “buy” than build

33. 33
Altspace VR

34. 34

35. 35

36. 36

37. 37

38. 38

39. 39
Altspace Architecture
• Photon game engine
• Real-time play
• Capture for playback
• An entirely classical stream processing system

40. 40
“Mirror User A to room R2”
Source: Altspace

41. 41
6 months later: “Capture User A”
Source: Altspace

42. 42
Playbacks
• Source capture data from Kafka or S3
• “Produce” it in a timed fashion

43. 43
“Playback capture to room R2” Source: Altspace

44. 44

45. 45
The Dream Stream

46. 46

47. 47

48. 48

49. 49

50. 50

51. 51

52. 52

53. 53

54. 54

55. 55

56. 56

57. 57

58. 58

59. 59

60. 60

61. 61

62. 62

63. 63

64. 64

65. 65

66. 66

67. 67

68. 68

69. 69
Get Started with Apache Kafka Today!
THE place to start with Apache Kafka!
Thoroughly tested and quality
assured
More extensible developer
experience
Easy upgrade path to
Confluent Enterprise
https://www.confluent.io/download/

70. 70
Confluent Community
Join us in Slack
https://slackpass.io/confluentcommunity
Find your local Meetup Group
https://www.confluent.io/apache-kafka-meetups/

71. 71
Thank you
@tlberglund