Dean Wampler, O’Reilly author and Big Data Strategist in the office of the CTO at Lightbend discusses practical tips for architecting stream-processing applications and explains how you can tame some of the complexity in moving from data at rest to data in motion.

1. Lightbend Fast Data Platform
A Technical Overview

2. A Bit of History:
Monoliths to Microservices

3. lightbend.com/reactive-microservices-architecture

4. • Tangled responsibilities, lead to…
• Infrequent, “big-bang” deployments
• Runtimes: months to years
• Latencies: μsecs to seconds
Monoliths

6. • Each does one thing, so…
• Updates easier to deploy frequently
• Message driven & asynchronous
• Runtimes: minutes! to forever
• Latencies: higher than function calls
Microservices

7. Reactive Systems
reactivemanifesto.org

8. Open Source Core
For Development in Scala / Java
Produc8on Suite
For Opera8ons / DevOps
Lightbend Reactive Platform

9. A Bit of History:
Big Data

10. YARN
HDFS
MR job #1
MR job #2
Flume Sqoop
DBs
Slave Node
DiskDiskDiskDiskDisk
Node Mgr
Data Node
Master
Resource
Manager
Name Node
Hadoop

11. YARN
HDFS
MR job #1
MR job #2
Flume Sqoop
DBs
Slave Node
DiskDiskDiskDiskDisk
Node Mgr
Data Node
Master
Resource
Manager
Name Node
Hadoop 2013:
Embrace Spark
Spark job #1
Spark job #2

12. YARN
HDFS
MR job #1
MR job #2
Flume Sqoop
DBs
Slave Node
DiskDiskDiskDiskDisk
Node Mgr
Data Node
Master
Resource
Manager
Name Node
Go beyond batch?
Spark job #1
Spark job #2

13. The Emergence of Fast Data:
(Time Is Money)

14. Respond to change

15. IoT

16. 16
Logs
Sockets
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Streaming
Streams
SQL
Batch
…
Streaming
Flink

17. Fast Data and Microservices;
Are they Converging?

18. •Each (stream|μservice):
•does one responsibility
•has unending (data|messages)
Synergies
18
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Streaming
Streams
SQL
Batch
…
Streaming
Flink

19. •Each (stream|μservice):
•encourages asynchrony
•offers never-ending service
Synergies
19
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Streaming
Streams
SQL
Batch
…
Streaming
Flink

20. •Reasons these architectures will
converge:
1.Similar design problems
2.Data becomes dominate problem
Thesis
20
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Streaming
Streams
SQL
Batch
…
Streaming
Flink

21. Lightbend
Fast Data Platform

22. 1. An accelerated on ramp for building
streaming data systems and
applications.
Value Three Ways

23. 2. Best practices guidance, through
sample apps, documentation, and
enablement services for what tools and
techniques to use for solving specific
design problems.
Value Three Ways

24. 3. Intelligent monitoring and
management, driven by machine
learning, to keep your systems resilient,
scalable, and responsive with minimal
user intervention.
Value Three Ways

25. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

26. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

27. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

28. Service 1
Log &
Other Files
Internet
Services
Service 2
Service 3
Services
Services
N * M links ConsumersProducers

29. Service 1
Log &
Other Files
Internet
Services
Service 2
Service 3
Services
Services
N + M links ConsumersProducers

30. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

31. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

32. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

33. •Low latency? How low?
•High volume? How high?
Streaming Tradeoffs (1/3)
33

34. •Which kinds of data processing &
analytics are required?
•How will this processing be done?
•Individual processing of events?
•Bulk processing of records?
Streaming Tradeoffs (2/3)
34

35. •Which tools and data sources/sinks
must interoperate with your
streaming tool?
Streaming Tradeoffs (3/3)
35

36. 36

37. 37
•Low latency
•Low volume
•Complex flows
•Complex Event
Processing

38. 38
•Med. latency
•High volume
•Data flows, SQL
•En masse
processing

39. 39
•Low latency
•High volume
•Data flows,
correctness
•En masse
processing

40. 40
•Low latency
•Med. volume
•ETL, “tables”
•Data flow or Per
Event

41. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

42. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

43. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

44. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

45. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles

46. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS, Marathon
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
Graviton
Cluster
Analysis
Machine
LearningMicroservices
ProducCon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles
For More Information:
lightbend.com/fast-data-platform

47. bit.ly/lightbend-fast-data
For More
Information

48. • FDP: lightbend.com/fast-data-platform
• Book: bit.ly/lightbend-fast-data