Remote storage provides the ability to separate compute and storage, which ushers in a new world of infinitely scalable and cost-effective storage. Remote storage in the cloud built to the HDFS standard has unique features that make it a great choice for storing and analyzing petabytes of data at a time. Customers can have unlimited storage capacity without any limit to the number or size of the files. With such scale, superior I/O performance becomes an increasingly important consideration when performing analysis on this data. For all workloads, a remote storage in the cloud can provide amazing performance when all the different knobs are tuned correctly... Speaker Stephen Wu, Senior Program Manager, Microsoft

1. Apache SparkTM and Apache Hadoop® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

2. stewu@microsoft.com
SparkTM and Hadoop® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

3. SparkTM and HiveTM are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

4. Cloud
elasticity
Scale compute
and storage
independently
No expensive
data centers
Less
management
Global presence Availability
SLAs
SparkTM and Hadoop® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

5. Azure Data Lake/
Azure Storage Blob
Azure Databricks/
Azure HDInsight/
Azure VM
Storage
Compute
Amazon Elastic
MapReduce/
Amazon Elastic Compute
Cloud
Amazon S3
Google Dataproc/
Google Compute
Engine
TM
Google Cloud
Storage
TM
Spark
TM
Hive
TM
MapReduce Storm
TM
Kafka
®
R Server
Azure AWS Google Cloud
TM
Amazon Web Services, the “Powered by AWS” logo, AWS, Amazon Elastic Compute Cloud, Amazon S3 are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries.©2017 Google LLC
All rights reserved. Google and the Google Logo are registered trademarks of Google LLC. SparkTM and Hadoop® are either registered trademarks or trademarks of the Apache Software Foundation in the United States
and/or other countries. All other trademarks are the property of their respective owners.

7. CPU
Intensive
Memory
Intensive
I/O
Intensive
• Machine Learning
• Natural Language
Processing
• PageRank
• Real-time analytics
• Copy
• Data preparation

8. Choose
the right
infrastructure
Use right
YARN settings
Right size your
application

9. Compute
Storage
Utilized
Throughput
Available
Throughput

10. Compute
Storage
Utilized
Throughput
Available
Throughput

11. # of VMs
Application Layer
Memory
Cores
Memory
Cores
Network
Physical Layer
Compute
Compute
Memory
GPU
VM Type VM Size
# of containers Container size
Tasks
RM (YARN)
Layer

12. # of VMs
RM (YARN)
Layer
Application Layer
Memory
Cores
Memory
Cores
Network
Physical Layer
Compute
Compute
Memory
GPU
VM Type VM Size
# of containers Container size
Tasks

13. Increase # of VMs

14. Use VMs with more memory

15. Use VMs with more network bandwidth
Low High

16. Compute
Storage
Utilized
Throughput
Available
Throughput

17. # of VMs
RM (YARN)
Layer
Application Layer
Memory
Cores
Memory
Cores
Network
Physical Layer
Compute
Compute
Memory
GPU
VM Type VM Size
# of containers Container size
Tasks

18. Use smaller YARN containers

19. Increase cores per container

20. Compute
Storage
Utilized
Throughput
Available
Throughput

21. # of VMs
RM (YARN)
Layer
Application Layer
Memory
Cores
Memory
Cores
Network
Physical Layer
Compute
Compute
Memory
GPU
VM Type VM Size
# of containers Container size
Tasks

22. Use all available containers

23. Compute
Storage
Utilized
Throughput
Available
Throughput

24. SparkTM
HiveTM
SparkTM and HiveTM are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

25. Application
Executors
Tasks
Cluster
SparkTM is either a registered trademark or trademark of the Apache Software Foundation in the United States and/or other countries.

26. Default – Four apps
Total resources
64 cores
192 GB
Your App
16 cores
48 GB
SparkTM is either a registered trademark or trademark of the Apache Software Foundation in the United States and/or other countries.

27. Two apps
Total resources
64 cores
192 GB
App #1
32 cores
96 GB
App #2
32 cores
96 GB

28. Your app
size
count
Two apps
Total resources
64 cores
192 GB
App #1
32 cores
96 GB
App #2
32 cores
96 GB

29. Executor-cores
Executor-memory
Num-executorCount
Size
SparkTM is either a registered trademark or trademark of the Apache Software Foundation in the United States and/or other countries.

30. 16 executors
More executors Fewer executorsOut of
memory
8 executors
32 executors
Best practice: set each executor no more than 64GB
App resources
32 cores
96GB

31. More
executors
Fewer
executors
8 executors16 executors 6 executors
Best practice: set cores between 2 and 5
App resources
32 cores
96GB

32. 8
4 cores
6GB
8 executors
16 executors
16
2 cores
6GB
8
2 cores
12GB
Memory underutilized
Both memory and CPU fully utilized
CPU underutilized
App resources
32 cores
96GB
16 executors
8 executors
16 executors
16 executors

33. SparkTM is either a registered trademark or trademark of the Apache Software Foundation in the United States and/or other countries.

34. SparkTM
HiveTM
SparkTM and HiveTM are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

35. Map Stage
Map tasks
Reducer Stage
Input Data
Reduce tasks
Output
HiveTM is either a registered trademark or trademark of the Apache Software Foundation in the United States and/or other countries.

36. 6GB
16 containers
More containers Fewer containersOut of
memory
YARN resources
Memory: 96GB
CPU: 32 cores
12GB
8 containers
3GB
32 containers
Best practice: set to minimum YARN container size

37. 32 containers
Mappers
Reducers
Read
Write
At least as
large as # of
containers
At least as
large as # of
containers

38. 1.6 waves
(best effort)
# of containers # of map tasks
10MB
Input Data: 80MB
Tez.grouping.min-size = 20MB

39. 1.6 waves
(best effort)
Tez.grouping.min-size = 20MB
# of containers
20MB
# of map tasks
Input Data: 80MB
Apache Tez® is either a registered trademark or trademark of the Apache Software Foundation in the United States and/or other countries.

40. 1.6 waves
(best effort)
Set tez.grouping.min-size = 5MB
# of containers
20MB
# of map tasks
Input Data: 80MB
Apache Tez® is either a registered trademark or trademark of the Apache Software Foundation in the United States and/or other countries.

41. 1.6 waves
(best effort)
Set tez.grouping.min-size = 5MB
# of containers
10MB
# of map tasks
Input Data: 80MB

42. ReducerStage
estimate
Hive.exec.reducers.
bytes.per.reducer
hive.tez.max.
partition.factor
Reduce tasks
5 containers:
2512MB 256MB

43. ReducerStage
estimate
Hive.exec.reducers.
bytes.per.reducer
hive.tez.max.
partition.factor
Reduce tasks
2512MB
Set hive.exec.reducers.bytes.per.reducer = 128MB
5 containers:
256MB

44. Set hive.exec.reducers.bytes.per.reducer = 128MB
ReducerStage
estimate
Hive.exec.reducers.
bytes.per.reducer
hive.tez.max.
partition.factor
Reduce tasks
5 containers:
2512MB 128MB

45. HiveTM is either a registered trademark or trademark of the Apache Software Foundation in the United States and/or other countries.