In this talk we will be building Airflow pipelines. We’ll look at real business requirements and walk through pipeline design, implementation, testing, deployment and troubleshooting - all that by adhering to idempotency and ability to replay your past data through the pipelines.

1. From business requirements to
working pipelines with Apache Airflow
Derrick Qin - Cloud Data Architect @ DoiT International
Multi-Cloud Engineering Meetup Australia
Tuesday 27 April 2021

2. About the speaker
Derrick Qin
Cloud Data Architect, DoiT International
Data engineering on GCP and AWS

3. Agenda
● Quick overview of Apache Airflow
● A business requirement
● Design data pipeline
● How to write the DAG
● How to write Airflow tasks
● Custom Airflow plugins
● Testing, testing, testing
● CI/CD and troubleshooting
● Q&A
Slido: https://app.sli.do/event/gmgdatnt

4. Quick overview of Apache Airflow

5. Airflow concepts

6. A business
requirement
● An e-commerce website
● Scheduled nudging emails
● Google Cloud
Platform(GCP)

7. Data sources
● Accounts
● Items
● Website activities
● Saved searches
● Purchase history
● ...

8. Design the
pipeline and…
DAG
● Parallel tasks to save time
● Check previous run
● Use built-in plugins
● Branching to handle
different conditions
● Retries
● Readability

9. My Airflow setup

10. How to write the DAG
Let’s check the DAG code!

11. How to write Airflow tasks
● Search built-in plugins(Operator, Sensor and Hooks)
○ official documentation
○ Astronomer Airflow registry
○ Apache Airflow Github repository
○ Google search - S3 to Mongo
● Custom operator/sensor
○ Build it if required
○ Use built-in Hooks
● Custom hook
○ Common interfaces to external platforms and databases
Let’s check the custom plugins code!
https://registry.astronomer.io/

12. Testing, testing, testing
● Data pipelines are normal software
programs
● Unit tests + End-To-End tests to ensure
pipeline quality
Let’s check the test code!

13. How about
CI/CD?
● Data pipelines are normal
software program
● GitOps are helpful

14. Troubleshooting - 1

15. Troubleshooting - 2

16. Troubleshooting - 3

17. What’s next?
● User activities coming from 3rd-party analytics tools? From SFTP?
○ Airflow built-in SFTPToGCSOperator to the rescue!
● Don’t want to spam unsubscribed accounts?
○ Bring in the email preference data source
● Already have all the pipeline built-in in Golang/Java/Php?
○ You can continue doing what you are doing, or consider
○ KubernetesPodOperator