Be ready to big data challenges. The material was composed based on the performance of Leonid Sokolov, Big Data Architect from GreenM. Full article https://medium.com/greenm/scalable-data-pipeline-f5d3c8f7a6d9

1. Scalable Data Pipeline
Be ready to big data challenges with
by Leonid Sokolov
Big Data Architect at greenm.io

2. Agenda
• Data Lake Monsters Recap
• Scale Distributed Data Processing
• Technologies

3. Data Lake Monsters Recap

4. Data Lake Monsters Recap

5. Data Lake Monsters Recap

6. Data Lake Monsters Recap

7. Data Pipeline Challenges
• Complex workflow management
• AWS Athena doesn’t scale well

8. Scale Distributed Data
Processing

9. Basic Data Actions
Extract
Transform
Load

10. Extract Transform Load
Extract Transform Load

11. Extract
Extract Transform Load
Transform Load
Input Process Output Input Process Output Input Process Output

12. Input Process OutputExtract
Database: Partition by Integer Key

13. +1000000
13569933
+ ???
???
Input Process OutputExtract
12569933 13680800900000083645ec0c06727066d249cfd01 ffffffcb586df761f63f561d946ac7c5
Database: Partition by Varchar Key

14. Input Process OutputExtract
Database: Varchar key

15. • Use scalable storage: HDFS, SЗ
• Use multiple files
• Use splittable file formats and compression
Format Splittable
CSV Yes*
JSON Yes**
Parquet Yes
Compression Splittable
gzip No
bzip2 Yes
Snappy No
Input Process OutputExtract
Files
• * CSV is splittable when it is a raw, uncompressed file or using a splittable compression format such as BZIP2
• ** JSON has the same conditions about splittability when compressed as CSV with one extra difference.
When “wholeFile” option is set to true (re: SPARK-18352), JSON is NOT splittable.

16. • Use scalable storage: HDFS, SЗ
• Spark on S3: mapreduce.fileoutputcommitter.algorithm.version = 2
• EMR 5.20.0 or later
• Use multiple files (better the same number as in input)
Input Process OutputExtract

17. Extract Results
0
10
20
30
40
50
60
70
80
Extract Time (minutes)
Before After
• EMR 5.20.0 with 10 instances (c4.4xlarge)
• Input: 3 Databases (MS SQL), ~400GB (Raw Data)
• Output: Parquet(Snappy), ~100GB

18. Extract
Extract Transform Load
Transform Load
Input Process Output Input Process Output Input Process Output

19. Input Process OutputTransform
Volume of Data
Partitions
Data Skew
Volume of Data
Partitions

20. Map Shuffle Reduce
SELECT * FROM Encounters e JOIN Providers p ON e.ProviderId = p.ProviderId
Input Process OutputTransform
Shuffle Join
Shuffle Map

21. Map Reduce
SELECT * FROM Encounters e JOIN Providers p ON e.ProviderId =p.ProviderId
Input Process OutputTransform
Broadcast Join
Broadcast Collect

22. • Use data partitioning, bucketing, sorting
• Broadcast small tables when joining them to big table
• spark.sql.autoBroadcastJoinThreshold= 10485760 (10 MB, default)
• Use COUNT(key) instead of COUNT(DISTINCT key) if possible
• Drop unused data
• Filter/reduce before join
• Cache Datasets used multiple times
Input Process OutputTransform
Reduce Shuffles

23. Transform
0
10
20
30
40
50
60
Transform Time(minutes)
Before After
Results
• EMR 5.20.0 with 10 instances (c4.8xlarge)
• Input: Parquet(Snappy), 50GB
• Output: ORC(ZLib), 19GB

24. Extract Transform Load
Input Process Output Input Process Output Input Process Output

25. Input Process OutputLoad
COPY dm1.FactTable
(
Column1,
Column2,
DateColumnTZ FILLER TIMESTAMPTZ,
DateColumn AS DateColumnTZ AT TIME ZONE 'UTC',
Column4,
...
)
FROM 's3://bucket/prod/datamarts/dm1/FactTable/snapshots/snapshotid=20190418/part-*.orc'
ORC
DIRECT
ABORT ON ERROR;

26. Load Results
0
20
40
60
80
100
Load Time(minutes)
Before After
• Input: ORC(ZLib), 19GB
• Output: Vertica DB (7 Node)

27. Summary
• Build architecture for scale
• Consider tomorrow’s data volume
• Build with failure in mind
• Understand the risks and be ready to respond

28. Technologies
Extract Transform Load
Environment AWS, EMR 5.20.0 AWS, EMR 5.20.0 AWS Batch
Technology Spark 2.4 Spark 2.4 Vertica
Languages Scala + SQL Scala + SQL Python + SQL
Input
Format
Compression
MS SQL, MySQL
Tables
-
S3
Parquet
Snappy
S3
ORC, Parq
Zlib, Snappy
Output
Format
Compression
S3
Parquet
Snappy
S3
ORC ,Parquet
Zlib, Snappy
Vertica
Tables
Native