This document discusses whether it is better to process data using a stream or batch approach. It describes how one company evolved their data pipeline from a micro-batch streaming process to a batch approach. The streaming process was very expensive, costing $400,000 per year to run. It also had issues with wasted resources during idle times, slow processing during bursts of data, and long recovery times from outages. The company rearchitected the process to use discrete time windows run in isolated batch jobs. This new batch approach reduced costs by 60% to $160,000 per year and improved processing efficiency and outage recovery times.

1. Should you read Kafka
as a stream or in
batch?

2. Stream
● Micro deliveries
● Many trips
● AND - Low waits

3. Batch
Photo by Charles J. Sharp
● Large deliveries
● Few trips
● BUT - Long waits

4. This is a story about a stream …..
gone batch

5. o Project evolution
o Metrics and Results
o Insights for your projects
Agenda

6. Who Are We ?
Opher Dubrovsky
Big Data Dev Lead
Ido Nadler
Big Data Engineer
Data Pipelines, Serverless, Spark, Analytics

7. Nielsen Marketing Cloud
Cloud native
~6,000
nodes/day
~60 TB/day
5 PetaByte
~6 million/day
Create data for:
› Running campaigns
› Business decisions

8. Our Old Stream Project
Data Lake
AWS S3
Total: 60 TB/day
Biggest topic: 21 Tb/day
Flow: Consume
micro-batch Transform
Write
parquet

9. Reality
Check
Photo by Olia Nayda on

10. Total Cost
~$400,000 per Year
Daily Costs ($)

11. 300
400
500
600
700
800
900
1,000
1,100
1,200
0
:
0
0
2
:
0
0
4
:
0
0
6
:
0
0
8
:
0
0
1
0
:
0
0
1
2
:
0
0
1
4
:
0
0
1
6
:
0
0
1
8
:
0
0
2
0
:
0
0
2
2
:
0
0
0
:
0
0
2
:
0
0
4
:
0
0
6
:
0
0
8
:
0
0
1
0
:
0
0
1
2
:
0
0
1
4
:
0
0
1
6
:
0
0
1
8
:
0
0
2
0
:
0
0
2
2
:
0
0
0
:
0
0
2
:
0
0
4
:
0
0
6
:
0
0
8
:
0
0
1
0
:
0
0
1
2
:
0
0
1
4
:
0
0
1
6
:
0
0
1
8
:
0
0
2
0
:
0
0
2
2
:
0
0
GB / Hour
Problems
Cluster is
Idle
Wasted $$$

12. Data Bursts (& reprocessing)
Cluster is
underpowered
Cluster
Capacity
Long Queues !

13. Data Expiration
Processing has to be Quick

14. Slow Recovery == Ticking Timebomb
Risk to Your DATA !

15. Rearchitecting
Rearchitecting the stream to batch……

16. Spread the work
across isolated workers

17. Using Discreet Time Slots
time
4:00 5:00 6:00 7:00
3:00 8:00 9:00
Task
Task
Task

18. Task (& Cluster) Independence
Processing
Hour
4:00
5:00
6:00
8:00
3:00
7:00
9:00
1 hour 2 hours 3 hours
Some tasks are short
No more paying for
idle time !
Some tasks >1 hour

19. Mechanics
Airflow AWS S3
{ period: 05:00 - 06:00 }
{ period: 06:00 - 07:00 }
{ period: 07:00 - 08:00 }

20. Comparing to Previous System
Cluster
Capacity
Wasted
Processing
Efficiency was 30% ONLY !!
New System is 60% Cheaper !

21. Results

22. From Micro-Batch to Batch
1 hour
1 hour
1 hour 1 hour 1 hour
Missed run
Rerun
2 hours

23. Pay as You Go
Bytes per Hour
$ per Hour
$400k /year à $160k /year

24. Outage Handling
Outage start
6 hours Recovery
completed
Old System
New System
Outage start Recovery
completed
~85% Faster Recovery
40 min
5 Concurrent
isolated workers

25. Summary

26. Conclusions and Insights
o Streaming is expensive
o Fixed clusters are never perfect
o Off hours - $$$
o High loads - Slow
o Recovery is critical
o Isolation and parallelism allows you to
o Easily scale
o Save on costs
o Deal with loads

27. You Can Reach Us At:
Opher Dubrovsky /opher-dubrovsky
Ido Nadler /ido-nadler

28. Thank You