Hadoop Summit 2014: Building a Self-Service Hadoop Platform at LinkedIn with Azkaban

8. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved. Azkaban 1.0  Run workflows  Schedule jobs  Job History  Failure notification  Easy to use web UI and visualizations 8

10. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved. Azkaban 2.0  Major re-architecting  Separate executor and web servers  User authentication  Pluggable database drivers – H2 – MySQL  Brand new UI 10

11. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved. Azkaban 2.0  Jobtype plugins – Built-in type: command – Pluggable jobtypes:  Java  Pig  Hive – Non-Hadoop jobtypes:  Teradata  Voldemort  Viewer plugins – extending the Azkaban UI for other tools – HDFS browser – Reportal  LinkedIn-specific code as plugins 11

13. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved. Azkaban 2.5  UI overhauled using Bootstrap  Embedded flows  New self-service tools – Job Summary – Flow Summary – Pig Visualizer  Jobtype-specific plugins  HDFS viewer improvements – Display file schema in addition to content – Parquet file viewer  And more 13

15. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved. Azkaban Today  Workflow manager and scheduler  Integrated runtime environment  Unified front-end for Hadoop tools 15

16. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved. Good News! Success!  1000+ users  Several clusters  2,500 flows executing per day  30,000 jobs executing per day 16

19. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved. Creating Workflows  Add job “type” plugins – hadoopJava – Command – Pig – Hive  Dependencies – Determine the dependency graph  Parameter passing – Parameters can be passed to job 19 type=pig creamy.level=4 chunky.level=4 ... type=hadoopJava jelly.type=grape sugar=HFCS ... type=command bread.type=wheat dependencies=peanutbutter,jelly ... peanutbutter.job bread.job jelly.job

20. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved. Embedded Flows  Embed a flow as a node in another flow. – “flow” job type – Set flow.name to name of the embedded flow – Parameters can be passed to flow 20 peanutbutter jelly bread type=flow flow.name=bread dependencies=coffee,fruit type=hive coffee.decaf=false coffee.cream=true ... type=hadoopJava fruit.type=apple ... coffee.job fruit.job sandwich.job

25. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved.  Finish Current – Finishes current running flows, then stops  Cancel All – Kills all running jobs and finishes immediately  Finish Possible – Finish all possible jobs if their dependencies have met. Then it fails. Running Workflows Failure Options 25

28. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved.  Skip Executions – Prevent concurrent executions  Run Concurrently – Concurrently run the flow  Pipeline – Distance 1: jobA waits until concurrent jobA finishes – Distance 2: jobA waits until concurrent jobA’s children finishes Running Workflows Concurrent Execution Options 28

59. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved. Azkaban Gradle Plugin and DSL  Describe Azkaban flow and deploy with Gradle  Single file (more if you want) to describe all your workflows – Compiles to .job files  Static checker  Valid Groovy code – Add conditionals for deployment to different clusters 59 azkaban { jobConfDir = ‘./jobs’ workflow(‘workflow2’) { pigJob(‘job2’) { script = ‘src/main/pig/count-by-country.job’ parameter ‘inputFile’, ‘/user/foo/sample’ reads ‘/data/databases/foo’, [as: ‘input’] writes ‘/data/databases/bar’, [as: ‘output’] } hiveJob(‘job3’) { query = ‘show tables’ } workflowDepends ‘job2’, ‘job3’ } }

60. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved. Future Roadmap  New visualizers (Hive, Tez, etc.)  Support DSL from other tools  Operationalization tooling  Scalability improvements  Improved plugin interfaces 60

61. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved. Future Discussions  Conditional branching  Hive Metastore browser  Pluggable executors (e.g. YARN)  Persistence storage server  Launching and monitoring long-running YARN applications (Samza, Storm, etc.) 61

62. Data Analytics Infrastructure©2013 LinkedIn Corporation. All Rights Reserved. Main Contributors  David Chen (LinkedIn)  Hien Luu (LinkedIn)  Anthony Hsu (LinkedIn)  Alex Bain (LinkedIn)  Richard Park (RelateIQ)  Chenjie Yu (Tango)  Shida Li (University of Waterloo) 62