From Stata + Hadoop World, San Jose, March 2017. Video: https://player.oreilly.com/videos/9781491976159 https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/55639 Twitter is all about real time at scale. Twitter’s data centers continuously process billions of events per day the instant the data is generated. To achieve real-time performance, Twitter has developed and deployed Heron, a next-generation cloud streaming engine that provides unparalleled performance at large-scale. Heron has been successfully meeting Twitter’s strict performance requirements for various streaming applications and is now an open source project with contributors from various institutions. Bill Graham, Avrilia Floratau, and Ashvin Agrawal explore how Twitter and Microsoft have collaborated to transform Heron into a truly elastic system. The amount of data that needs to be processed in Twitter’s data centers changes significantly due to expected and unexpected global events. For example, during the Super Bowl, there are spikes of tweets that all need to be processed in real time. Similarly, unexpected events such as natural disasters can generate very large volumes of data. Twitter’s infrastructure must be robust enough to deliver real-time performance when such events take place. Heron solves this problem by supporting dynamic scaling of Heron applications, achieved by increasing or decreasing the number of Heron instances that form a topology. Providing truly elastic scaling in the context of Heron has several challenges. First, during scaling, the processing of events should not be disrupted. Heron must be able to handle running topologies while causing minimal disruption during scaling. Second, Heron has a modular and extensible architecture that allows it to integrate with various schedulers and incorporate various resource management policies. As a result, Heron’s scaling framework must be generic enough to operate efficiently on top of diverse components. Finally, Heron must be able to efficiently allocate resources during scaling so that it makes optimal use of available cluster resources. Bill, Avrilia, and Ashvin explain how Heron addresses the challenges discussed above to gracefully handle highly varying loads without sacrificing real-time performance and conclude by sharing future plans for scaling that current and future Heron contributors could tackle.

1. From Rivulets to Rivers:
Elastic Stream Processing in Heron
Bill Graham , Twitter - @billgraham
Ashvin Agrawal, Microsoft
Avrilia Floratou, Microsoft

2. Prediction is very difficult, especially if it’s about the future.
- Nils Bohr
We cannot direct the wind, but we can adjust the sails.
- Dolly Parton

3. Outline
 Heron Overview
 Elastic Scaling Opportunities
 Current Implementation
 Work in Progress – Auto-scaling

4. Heron
A realtime, distributed, fault-tolerant stream processing engine.

5. About Heron
 Developed by Twitter in 2014
 Open sourced in May 2016
 Storm API compatible
 Isolation at all levels:
- Topology
- Container
- Task (process-based)
 At least once, at most once semantics
 Backpressure
 Low resource overhead (< 10%)

6. Logical Topology
Spout 1
Spout 2
Bolt 1
Bolt 2
Bolt 3
Bolt 4
Bolt 5

7. Physical Execution
Spout 1
Spout 2
Bolt 1
Bolt 2
Bolt 3
Bolt 4
Bolt 5

8. Packing Plan
 How to distribute instances onto containers?
 IPacking.pack()

9. Topology Submission
Container 2
Container 0
Stream
Manager
S1 S2 B3
B4 B5 B6
Topology
Master
Heron
Client
Heron
Scheduler
Container 3
Stream
Manager
S1 B2 B3
B4 B5 B6
Container 1
Stream
Manager
S1 S2 B3
B4 B5 B6
heron
submit
Packing
Plan
Instances Register
Stream Manager Registers
Data Flows
Containers Allocated
Processes Initialize

10. Data Rate Variations

11. Parallelism Challenges
 Anticipating component parallelism is difficult
 Changing parallelism is costly - O(hour)
- code change, review, merge, build, kill, submit
 Tuning for load spikes or valleys is manual - O(day)
 Under-provisioning leads to back pressure leads to support costs
 Over-provisioning is the norm

12. Over-provisioning
25%
40%
CPU Used
CPU Requested

13. Elastic Scaling Opportunity
 Reduce administration cost
 Reduce support cost
 Reduce hardware cost
 Provide better SLA

14. User Tasks Heron System Tasks
Ordinary Topology Management Process
Kill
Topology
Submit
Topology
Create
Packing
Acquire
Resource
s
Monitor /
Estimate
Build
State
Start
Topology
Install
Topology
Time Consuming Tasks
Releases
Resources

15. Low-cost Topology “update”
32 2 34 4

16. User Tasks Heron System Tasks
Optimized Topology Scale-up Process
Kill
Topology
Submit
Topology
Create
Packing
Acquire
Resources
Monitor /
Estimate
Build State
Start
Topology
Install
Topology
Update
Topology
Pause
Topology
Un-Pause
Topology
Add /
Reduce
Resources
Prepare
Component
s

17. heron “update” …
Minimizes Disruption
Aggressively Prunes
Containers
Aims to Maintain Uniform
Component Distribution
$ heron update my_cluster/user/dev MyTopology
--component-parallelism=bolt1:20
--component-parallelism=bolt2:40
Available in 0.14.5
Execution Time O(mins)
Customizable Through
IRepacking.repack()

18. Current Limitations
 Automated state transition not yet supported
- Component scaling event notification : IUpdatable.update()
- Example: KafkaSpout queue partition mappings
 Fields group routing might change
- Workaround: pause topology > cache flush interval before scaling
 Algorithmic Auto-Scaling

19. User Tasks Heron System TasksUser Tasks Heron System Tasks
Algorithmic Auto-Scaling …
Submit
Topology
Create
Packing
Acquire
Resources
Monitor /
Estimate
Build State
Start
Topology
Install
Topology
Update
Topology
Pause
Topology
Un-Pause
Topology
Add /
Reduce
Resources
Prepare
Component
s

20. Auto-Scaling
Heron should
automatically identify
variations in the
incoming load and
react to them.
Dhalion periodically
observes the state of the
topology and determines
whether resources
should be scaled up or
down.
Heron uses Dhalion to
adjust to external shocks.
Dhalion is a framework
that provides
self-regulating capabilities
to Heron and will be
open-sourced in the near
future.

21. Using Dhalion to Auto-Scale
Pending Packets
Detector
Backpressure
Detector
Processing Rate
Skew Detector
Resource
Underprovisioning
Diagnoser
Data Skew
Diagnoser
Slow Instances
Diagnoser
Resolver
Invocation
Diagnosis
Symptom Detection Diagnosis Generation
Bolt Scale
Down
Resolver
Bolt Scale
Up Resolver
Restart
Instances
Resolver
Resource
Overprovisioning
Diagnoser
Data Skew
Resolver
Symptoms
Resolution
Metrics
as needed
while still keeping the topology in a steady state where backpressure is not observed

22. Initial Results
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95
100
105
110
115
120
NormalizedThroughput
Time (in minutes)
Spout Splitter Bolt Counter Bolt
Scale
Down
Scale Up
S1
S2
S3
Dhalion is able to adjust the
topology resources on-the-fly
when workload spikes occur.
Our policy eventually reaches a
healthy state where
backpressure is not observed
and the overall throughput is
maximized.

23. Future Plans
Use Dhalion to enforce
throughput and latency
SLOs
and to auto-tune
Heron topologies.
Open-source Dhalion
and the auto-scaling
policy as part of
Heron.
Combine scaling with
stateful stream
processing.

24. Get Involved
http://github.com/twitter/heron
http://heronstreaming.io
@heronstreaming

25. Up Next
Anomaly detection in real-time data streams using Heron
Arun Kejariwal, Machine Zone
Karthik Ramasamy, Twitter

26. Questions?