This presentation gives an overview of the Apache Airflow project. It explains Apache Airflow in terms of it's pipelines, tasks, integration and UI. Links for further information and connecting http://www.amazon.com/Michael-Frampton/e/B00NIQDOOM/ https://nz.linkedin.com/pub/mike-frampton/20/630/385 https://open-source-systems.blogspot.com/

1. What Is Apache Airflow ?
● A work flow management platform
● Uses Python based work flows
● Schedule by time or event
● Open source Apache 2.0 license
● Written in Python
● Monitor work flows in UI
● Has a wide range of integration options
● Originally developed at Airbnb

2. What Is Apache Airflow ?
● Uses SqlLite as a back end DB but can use
– MySQL, Postgres, JDBC etc
● Install extra packages using pip command
– Wide variety available, includes
– Many databases, cloud services
– Hadoop eco system
– Security, web services, queues
– Many more

3. Airflow Pipelines
● These are Python based work flows
● Are actually directed acyclic graphs ( DAG's )
● Pipelines use Jinja templating
● Pipelines contain user defined tasks
● Tasks can run on different workers at different times
● Jinja scripts can be embedded in tasks
● Comments can be added in tasks in varying formats
● Inter task dependencies can be defined

4. Airflow Pipelines

5. Airflow Tasks
● Tasks have a lifecycle
● Tasks use operators to execute, depends upon type
– For instance MySqlOperator
● Hooks are used to access external systems i.e. databases
● Worker specific queues can be used for tasks
● Xcom allows tasks to exchange messages
● Pipelines or DAG's allow
– Branching
– Sub DAG's
– Service level agreements ( SLA )
– Triggering rules

6. Airflow Task Stages
● Tasks have life cycle stages

7. Airflow Task Life Cycle

8. Airflow UI
● Airflow UI provides views
– DAG, Tree, Graph, Variables, Gantt Chart
– Task duration, Code view
● Select a task instance in any view to manage
● Monitor and troubleshoot pipelines in views
● Monitor DAG's by owner, schedule, run time etc
● Use views to find pipeline problem areas
● Use views to find bottle necks

9. Airflow UI

10. Airflow Integration
● Airflow Integrates with
– Azure: Microsoft Azure
– AWS: Amazon Web Services
– Databricks
– GCP: Google Cloud Platform
– Cloud Speech Translate Operators
– Qubole
● Kubernetes
– Run tasks as pods

11. Airflow Metrics
● Airflow can send metrics to StatsD
– A network daemon that runs on Node.js
– Listens for statistics, like counters, gauges, timers
– Statistics sent over UDP or TCP
● Install metrics using pip command
● Specify which stats to record i.e.
– scheduler,executor,dagrun

12. Available Books
● See “Big Data Made Easy”
– Apress Jan 2015
●
See “Mastering Apache Spark”
– Packt Oct 2015
●
See “Complete Guide to Open Source Big Data Stack
– “Apress Jan 2018”
● Find the author on Amazon
– www.amazon.com/Michael-Frampton/e/B00NIQDOOM/
●
Connect on LinkedIn
– www.linkedin.com/in/mike-frampton-38563020

13. Connect
● Feel free to connect on LinkedIn
– www.linkedin.com/in/mike-frampton-38563020
● See my open source blog at
– open-source-systems.blogspot.com/
● I am always interested in
– New technology
– Opportunities
– Technology based issues
– Big data integration