Dr. Elephant is a tool developed by LinkedIn to provide automated tuning recommendations for Spark jobs running on big data clusters to help improve performance, reduce costs, and increase developer productivity. It works by collecting metrics from Spark jobs and analyzing them against heuristics and rules to identify issues and recommend configuration changes. Some examples of rules include detecting when Spark memory allocation can be reduced, or when unified vs. separate executor memory settings should be used. The tool was open sourced to help grow its capabilities and user community.

1. Dr. Elephant:
Achieving Quicker, Easier, and
Cost-Effective Big Data Analytics
#EUdev9
Akshay Rai, Linkedin

2. Overview
• Why Dr. Elephant?
• How does Dr. Elephant work?
• Spark Integration Challenges
• Spark Rules/Heuristics
• Roadmap
2#EUdev9

3. Grid Scale at Linkedin
3#EUdev9
2008 2018
1 cluster 10+ clusters
20 nodes 1000s of nodes
5 users 1000s of active users
MapReduce Pig, Hive, Spark, etc.
Few workflows hundred-thousand workflows

4. How much can we scale?
4#EUdev9

5. Cluster and Job Tuning
5#EUdev9

6. Challenges in Tuning
• Too many frameworks
• Too Many Different Knobs
• Overwhelming Data
• Scattered Data
6

7. Developer
Productivity
Vs
Cluster
Efficiency
7

8. Training Users to Tune
• Train and educate
Issues:
• Newer Frameworks
• More & more People
• Varied Experience
8#EUdev9

9. Expert Review
9#EUdev9
• Experts review & approve jobs
• Every job is reviewed
Issues:
• Not Scalable
• Delayed Reviews
• Manual Review Process

10. Birth of Dr. Elephant
• Identifies Common Issues
• Gives Actionable Advice
• Recommends Best practices
• Tracks Flow and Job Health
10

11. @Dr. Elephant, please review
Ship it!
Automated Process
11

12. 12#EUdev9

13. How does it work?
13
Metrics
Fetcher
History
Server
Application
Fetcher
Resource
Manager
Run
Rule 1
Run
Rule 2
Run
Rule 3
Database
Dr. Elephant UI

14. Spark Support in Dr. Elephant
14#EUdev9

15. Spark Metrics Fetcher
15
Spark
Metrics
Fetcher
Spark
History
Server

16. Unstable Spark History Server
• Report delayed at Peak Load
• Too many Dr. Elephant REST calls - SHS Crashes
16#EUdev9

17. Solution
• Event Log Parser
• Fix Spark History Server
17#EUdev9

18. Event Log Parser
• Replay HDFS Logs with Listeners
Issues:
• Large Log Files.
– More Resources
– Backlog of jobs due to blocked threads
• Parsing logic is not public
18#EUdev9

19. Fix Spark History Server
• Use SPARK-18085 fork
• Uses Level DB with improved caching
• Some patches to add More metrics
– JVM Used Memory
– Executor disk space used
19#EUdev9

20. Spark Rules / Heuristics
20

21. Spark Memory
21#EUdev9

22. 22
JVM Used Memory Heuristic
Credits: Edwina Lu#EUdev9
Reduce spark.executor.memory!

23. Unified Memory Heuristic
23#EUdev9 Credits: Edwina Lu
If JVM used memory is low then reduce spark.executor.memory.
If JVM used memory is high then reduce spark.memory.fraction.

24. Spark Stage Heuristic
24#EUdev9

25. Spark Configuration Heuristic
25#EUdev9

26. More Heuristics
• Skewness in the Memory Regions
• Execution or Storage Spill
• GC Time heuristic
26#EUdev9

27. Customization
• Write and Configure New Rules
• Easily integrate with new schedulers
• More Fetchers
• More Job types
27#EUdev9

28. Open Source
• Active and Vibrant Community
• Hangout Meetings
• Meet-ups
• Spark Discussions
28
github.com/linkedin/dr-elephant

29. Roadmap
• Specific Suggestions
• Spark Debugging
• More Spark Heuristics
• Apache Incubation
29#EUdev9

30. References
• Engineering Blogs:
– Open Source Blog: Click here
– A year later after open source: Click here
• Open Source Github Link: github.com/linkedin/dr-elephant
• Mailing List: dr-elephant-users
• Hangout Meetings:
– Agenda: Click here
– Minutes: Click here
• Hadoop Summit 2015 (Before Open Source): Click here
• Fifth Elephant Conference 2016 (After Open Source): Click here
30#EUdev9

31. ©2014 LinkedIn Corporation. All Rights
Reserved.
©2014 LinkedIn Corporation. All Rights
Reserved.
31
Thank You