Uber developed an new Spark ingestion system, Marmaray, for data ingestion from various sources. It’s designed to ingest billions of Kafka messages every 30 minutes. The amount of data handled by the pipeline is of the order hundreds of TBs. Omar details how to tackle such scale and insights into the optimizations techniques. Some key highlights are how to understand bottlenecks in Spark applications, to cache or not to cache your Spark DAG to avoid rereading your input data, how to effectively use accumulators to avoid unnecessary Spark actions, how to inspect your heap and nonheap memory usage across hundreds of executors, how you can change the layout of data to save long-term storage cost, how to effectively use serializers and compression to save network and disk traffic, and how to reduce amortize the cost of your application by multiplexing your jobs, different techniques for reducing memory footprint, runtime, and on-disk usage. CGI was able to significantly (~10%–40%) reduce memory footprint, runtime, and disk usage. Speaker: Omkar Joshi (Uber) Omkar Joshi is a senior software engineer on Uber’s Hadoop platform team, where he’s architecting Marmaray. Previously, he led object store and NFS solutions at Hedvig and was an initial contributor to Hadoop’s YARN scheduler.

1. How to
performance-tune
spark applications in
large clusters
- Omkar Joshi

2. Omkar Joshi
omkar@uber.com
● Software engineer on Hadoop
Data Platform team
● 2.5 years at Uber
● Architected Object store & NFS
solutions at Hedvig
● Hadoop Yarn committer
● Enjoy gardening & hiking in free
time

3. 01 Hadoop profiler
02 Flame graph
03 Auto tune
04 Storage improvements
05 Cpu / Runtime improvements
06 Efficiency improvements
07 Memory improvements
Agenda

4. Challenge: Distributed
Profiling at a Large Scale
● Uber’s Scale
Multi-Tenancy, Distributed
1m+ Applications per Week
1000+ Executors Per Application
● Resource Utilization: Usage ≠ Allocation

5. Hadoop Profiler
● Java Agent attached to each
executor
● Collects metrics via JMX and
/proc
● Instruments arbitrary Java user
code
● Emits to Kafka, InfluxDB, and
other data sinks
Method
Argument
Profiler
Method
duration
Profiler
CPU/Memory
Profiler
Reporter
Java Agent
Kafka InfluxDB
https://github.com/uber-common/jvm-profiler

6. System Integration

7. How to Use
● Spark: spark-submit command line:
$ spark-submit --deploy-mode cluster --master yarn
--conf spark.jars=hdfs://hdfs_url/lib/jvm-profiler-1.0.0.jar
--conf spark.driver.extraJavaOptions=-javaagent:jvm-profiler-1.0.0.jar
--conf spark.executor.extraJavaOptions=-javaagent:jvm-profiler-1.0.0.jar
--class com.uber.spark.MyJob my_spark_job.jar
● Plain Java Program: -javaagent option:
$ java -javaagent:jvm-profiler-1.0.0.jar --classpath foo.jar YourAppClass

8. Common Use Cases
● Memory Deep Dive: Heap, RSS, Direct Buffer, Mapped Buffer Pool
○ Right-size Container : Reduced Memory from 7G to 5G for 1000+ Executors
○ Off-Heap/Native Memory Tuning
● Hive On Spark Efficiency
○ ~70% queries
use <80%
allocated memory resource

9. Advanced Use Cases
● Trace arbitrary Java method argument
e.g. org.apache.hadoop.hdfs.protocolPB.
ClientNamenodeProtocolTranslatorPB.addBlock.1
Access Auditing:
Read/Write HDFS Files from individual Spark App

10. Advanced Use Cases
● Trace arbitrary Java method during
runtime
○ For example:
org.apache.hadoop.hdfs.protocolPB.Client
NamenodeProtocolTranslatorPB.getBlock
Locations
● Flamegraph
○ Trace CPU Hotspot
○ Helped finding serialization take
significant time during writing shuffle
data

11. Auto Tune (Work in Progress)
● Problem: data scientist using team level Spark conf template
● Known Daily Applications
○ Use historical run to set Spark configurations (memory, vcore, etc.) *
● Ad-hoc Repeated (Daily) Applications
○ Use Machine Learning to predict resource usage
○ Challenge: Feature Engineering
■ Execution Plan
Project [user_id, product_id, price]
Filter (date = 2019-01-31 and product_id = xyz)
UnresolvedRelation datalake.user_purchase
* Done by our colleague Abhishek Modi

12. Open Sourced in September 2018
https://github.com/uber/marmaray
Blog Post:
https://eng.uber.com/marmaray-hado
op-ingestion-open-source/

13. High-Level Architecture
Chain of converters
Schema Service
Input
Storage
System
Source
Connector
M3 Monitoring System
Work
Unit
Calculator
Metadata Manager
(Checkpoint store)
Converter1 Converter 2
Sink
Connector
Output
Storage
System
Error Tables
Datafeed Config Store
Spark execution framework

14. Spark job
improvements

15. Storage
improvements

16. Effective data layout (parquet)
● Parquet uses columnar compression
● Columnar compression savings outperform gz or snappy compression
savings
● Align records such that adjacent rows have identical column values
○ Eg. For state column California.
● Sort records with increasing value of cardinality.
● Generalization sometimes is not possible; If it is a framework provide custom
sorting.

17. User Id First Name City State Rider score
abc123011101 John New York City New York 4.95
abc123011102 Michael San Francisco California 4.92
abc123011103 Andrea Seattle Washington 4.93
abc123011104 Robert Atlanta City Georgia 4.95
User Id First Name City State Rider score
cas123032203 Sheetal Atlanta City Georgia 4.97
dsc123022320 Nikki Atlanta City Georgia 4.95
ssd012320212 Dhiraj Atlanta City Georgia 4.94
abc123011104 Robert Atlanta City Georgia 4.95

18. CPU / Runtime
improvements

19. Custom Spark accumulators
● Problem
○ Given a set of ride records; remove duplicate ride records and also count duplicates per state
1. RDD<String> rideRecords =
javaSparkContext.readParquet(some_path);
2. Map<String, Long> ridesPerStateBeforeDup
= rideRecords.map(r ->
getState(r)).countByKey();
3. RDD<String> dedupRideRecords =
dedup(rideRecords);
4. Map<String, Long> ridesPerStateAfterDup =
dedupRideRecords.map(r ->
getState(r)).countByKey();
5. dedupRideRecords.write(some_hdfs_path);
6. Duplicates = Diff(ridesPerStateAfterDup,
ridesPerStateBeforeDup)
7. # spark stages = 5 ( 3 for countByKey)
1. Class RidesPerStateAccumulator extends
AccumulatorV2<Map<String, Long>>
2. RidesPerStateAccumulator
riderPerStateBeforeDup, riderPerStateAfterDup;
3. dedup(javaSparkContext.readParquet(some_pat
h).map(r -> {riderPerStateBeforeDup.add(r);
return r;})).map(r ->
{riderPerStateAfterDup.add(r); return
r;}).write(some_hdfs_path);
4. Duplicates = Diff(ridesPerStateAfterDup,
ridesPerStateBeforeDup)
5. # spark Stages = 2 (no counting overhead!!)

20. Kafka topic
256 Partitions
3Billion messsages per
run
Increasing kafka read parallelism
Kafka
Broker
1 Kafka
consumer
/ partition
(256
tasks)
1 Kafka
consumer per
Kafka partition
Each consumer
reading 12Million
messages
Spark Stage 1
Sparkshuffleservice
Shuffle
Write
(1.3TB)
Spark Stage 2
8192
tasks
Shuffle
Read
(1.3TB)
Kafka
Broker
>1 Kafka consumer
per Kafka partition
Each consumer reading
384K messages
1 Kafka partition split
into 32 virtual
partitions
Spark Stage 2
8192
tasks

21. Increasing kafka read parallelism contd..
2
1

22. ● Why Kryo?
○ Lesser memory footprint than Java serializer.
○ Faster and supports custom serializer
● Bug fix to truly enable kryo serialization
○ Spark kryo config prefix change.
● What all is needed to take advantage of that
○ Set “spark.serializer” to “org.apache.spark.serializer.KryoSerializer”
○ Registering avro schemas to spark conf (sparkConf.registerAvroSchemas())
■ Useful if you are using Avro GenericRecord (Schema + Data)
○ Register all classes which are needed while doing spark transformation
■ Use “spark.kryo.classesToRegister”
■ Use “spark.kryo.registrationRequired” to find missing classes
Kryo serialization

23. Kryo serialization contd..
Data (128)
Data (128)
Data (128)
Data (128)
Data (128)
Avro Schema (4K)
Avro Schema (4K)
Avro Schema (4K)
Avro Schema (4K)
Avro Schema (4K)
Data (128)
Data (128)
Data (128)
Data (128)
Data (128)
Schema identifier(4)
Schema identifier(4)
Schema identifier(4)
Schema identifier(4)
Schema identifier(4)
1 Record = 4228 Bytes 1 Record = 132 Bytes (97% savings)

24. Reduce ser/deser time by restructuring payload
1. @AllArgsConstructor
2. @Getter
3. private class SparkPayload {
4. private final String sortingKey;
5. // Map with 1000+ entries.
6. private final Map<String, GenericRecord>
data;
7. }
8.
1. @Getter
2. private class SparkPayload {
3. private final String sortingKey;
4. // Map with 1000+ entries.
5. private final byte[] serializedData;
6.
7. public SparkPayload(final String sortingKey,
Map<String, GenericRecord> data) {
8. this.sortingKey = sortingKey;
9. this.serializedData =
KryoSerializer.serialize(data);
10. }
11.
12. public Map<String, GenericRecord> getData() {
13. return
KryoSerializer.deserialize(this.serializedData,
Map.class);
14. }
15. }

25. Parallelizing spark’s iterator
jsc.mapPartitions (iterator -> while (i.hasnext) { parquetWriter.write(i.next); })
MapPartitions Stage (~45min) MapPartitions Stage (~25min)
Reading from disk
Fetching record from
Spark’s Iterator
Writing to Parquet
Writer in memory
Parquet columnar
compression
FileSystem write
Reading from disk
Fetching record from
Spark’s Iterator
Writing to Parquet
Writer in memory
Parquet columnar
compression
FileSystem write
Spark’s Thread
New writer
Thread
Memory
buffer
Spark’s Thread

26. Efficiency
improvements

27. Improve utilization by sharing same spark resources
JavaSparkContext.readFromKafka(“topic
1”).writeToHdfs();
JavaSparkContext.readFromKafka(“topic
1”).writeToHdfs();
JavaSparkContext.readFromKafka(“topic
3”).writeToHdfs();
JavaSparkContext.readFromKafka(“topic
2”).writeToHdfs();
JavaSparkContext.readFromKafka(“topic
N”).writeToHdfs();Threadpoolwith#threads=parallelism
needed

28. Improve utilization by sharing same spark resources
Time
ExecutorId
Stage Boundaries
Lot of wastage!!!

29. Improve utilization by sharing same spark resources
Time
ExecutorId
Stage Boundaries

30. Memory
improvements

31. Off heap memory improvements (work in progress)
● Symptom
○ Container killed by YARN for exceeding memory limits. 10.4 GB of 10.4 GB physical memory used. Consider
boosting spark.yarn.executor.memoryOverhead
● Solution as per stack overflow :)
○ Increase “spark.yarn.executor.memoryOverhead”
○ Result - Huge memory wastage.
● Spark memory distribution
○ Heap & off-heap memory (direct & memory mapped)
● Possible solutions
○ Avoid memory mapping or perform memory mapping chunk by chunk instead of entire file
(4-16MB vs 1GB)
● Current vs Target
○ Current - 7GB [Heap(4gb) + Off-heap(3gb)]
○ Target - 5GB [Heap(4gb) + Off-heap(1gb)] - ~28% memory reduction (per container)

32. Thank You!!