In rapidly changing conditions, many companies build ETL pipelines using ad-hoc strategy. Such an approach makes automated testing for data reliability almost impossible and leads to ineffective and time-consuming manual ETL monitoring.

1. Scale and Optimize
Data Engineering Pipelines
with Software Engineering Best Practices:
Modularity and Automated Testing
© 2020 / Levi Strauss & Co.
Qiang MENG
Sr. Data Engineer, Levi Strauss & Co.

2. Qiang MENG
• Levi Strauss & Co., Belgium
• Databricks – AWS – Airflow – Spark - Dataiku
• H&M Group, Sweden
• Databricks – Azure – Airflow – Spark - Hive
• Softbank Robotics, France
• AWS – Luigi – Spark
© 2020 / Levi Strauss & Co.

3. © 2020 / Levi Strauss & Co.

4. Levi’s Data, Analytics & AI
© 2020 / Levi Strauss & Co.
Demand Forecast
Loyalty
Price & Promo
Recommendation
0%
10%
20%
30%
40%
50%
60%
70%
Q3 Digital Sales Increase
USA EU AMA
NLP
Allocation
Computer Vision

5. Agenda
I. Hypothesis
• A Data Engineering project in Apache Airflow
II. Optimization
• Software Engineering best practices
III. Auto-testing
• Unit test, Functional test, End to End test, and
Smoky test
© 2020 / Levi Strauss & Co.

6. I. Hypothesis
© 2020 / Levi Strauss & Co.

7. © 2020 / Levi Strauss & Co.
Hypothesis
Data Engineering – Cloud - Airflow - TB-level - Batch

8. © 2020 / Levi Strauss & Co.
Hypothesis
Dev Environments
▪ Local – Dev – Preprod – Prod
Sample Data
▪ ETL in local / Dev
Code Versioning – GIT
• feature - dev - release - hotfix – master
Conventions
• DB, Schema, Table, Column, Pipeline
Container-orchestration
• Docker, K8S, …
Spark clusters
• Databricks, EMR, …
Data Versioning
• Delta Lake, …
CICD

9. Airflow
▪ airflow
• dags
• operators
• …
▪ utils
▪ modules
▪ docs
▪ tests
▪ others
• airflow
• dags
• perators
• …
• others
© 2020 / Levi Strauss & Co.

10. II. Optimization
© 2020 / Levi Strauss & Co.

11. A Big Data ETL

12. Optimization - Airflow DAG

13. Optimization - Airflow DAG
A simple template of tasks – Data Feeds Based
© 2020 / Levi Strauss & Co.

14. © 2020 / Levi Strauss & Co.
Optimization - Airflow Operator
• Default Operators
• Keep as they are
• Customized Operators
• Hive
• Spark
• Deduplication
• Data Validation
• Data Profiling
• …
Do the real work

15. Optimization - Modules and Utils
▪ Modules
▪ Utils

16. Optimization - OOP

17. Optimization - OOP
© 2020 / Levi Strauss & Co.

18. Optimization - Design Patten
© 2020 / Levi Strauss & Co.

19. Optimization - Formatting
© 2020 / Levi Strauss & Co.

20. Optimization - Docs
• Docstring - Sphinx
• Data Catalog – Markdown
© 2020 / Levi Strauss & Co.

21. Docstring
© 2020 / Levi Strauss & Co.

22. Auto Data
Catalog
with
Markdown
© 2020 / Levi Strauss & Co.

23. III. Auto-testing
© 2020 / Levi Strauss & Co.

24. Auto-Testing
• Unit tests
• Integration tests
• End-to-End tests
• Smoky tests
© 2020 / Levi Strauss & Co.

25. Unit Tests
© 2020 / Levi Strauss & Co.

26. Integration Tests
© 2020 / Levi Strauss & Co.

27. End-to-End Tests
© 2020 / Levi Strauss & Co.

28. Smoke Tests
© 2020 / Levi Strauss & Co.

29. READY FOR
THE NEXT
167 YEARS
© 2020 / Levi Strauss & Co.

30. Sign off/Thank You/Questions?
© 2020 / Levi Strauss & Co.
READY FOR THE NEXT 166 YEARS
THANK YOU
© 2020 / Levi Strauss & Co.

31. Feedback
Your feedback is important to us.
Don’t forget to rate
and review the sessions.
© 2020 / Levi Strauss & Co.