Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N

Andreas Neumann Oozie – Workflow for Hadoop

Who Am I? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Oozie Overview ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Oozie Workflow Overview ,[object Object],[object Object],Oozie Hadoop/Pig/HDFS DB WS API Tomcat web-app

Oozie Workflow Directed Acyclic Graph of Jobs start Java Main M/R streaming job decision fork Pig job M/R job join OK OK OK OK end Java Main FS job OK OK ENOUGH MORE

Oozie Workflow Example ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],OK Start Error Start M-R wordcount End Kill

Oozie Workflow Nodes ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Oozie Workflow Application ,[object Object],[object Object],[object Object],[object Object],[object Object]

Running an Oozie Workflow Job ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Monitoring an Oozie Workflow Job ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Oozie Coordinator Overview ,[object Object],[object Object],[object Object],Hadoop Tomcat Oozie Client Oozie Workflow WS API Oozie Coordinator Check Data Availability

Oozie Application Lifecycle Coordinator Job Oozie Coordinator Engine Oozie Workflow Engine 1*f Action 1 WF 2*f Action 2 WF N*f … … Action N … … WF 0*f Action 0 WF action create action start start end A B C

Use Case 1: Time Triggers ,[object Object],00:15 00:30 00:45 01:00

Example 1: Run Workflow every 15 mins ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Use Case 2: Time and Data Triggers ,[object Object],Input Data Exists? 01:00 02:00 03:00 04:00 Hadoop

Example 2: Data Triggers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Use Case 3: Rolling Windows ,[object Object],00:15 00:30 00:45 01:00 01:00 01:15 01:30 01:45 02:00 02:00

Example 3: Rolling Windows ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Use Case 4: Sliding Windows ,[object Object],01:00 02:00 03:00 24:00 24:00 … 02:00 03:00 04:00 +1 day 01:00 +1 day 01:00 … 03:00 04:00 05:00 +1 day 02:00 +1 day 02:00 …

Example 4: Sliding Windows ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Oozie Coordinator Application ,[object Object],[object Object],[object Object]

Running an Oozie Coordinator Job ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Monitoring an Oozie Coordinator Job ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Oozie Web Console: Job Details

Oozie Web Console: Failed Action

Oozie Web Console: Error Messages

What’s Next For Oozie? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

We Need You! ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Thank You! github.com/yahoo/oozie/wiki/How-To-Contribute

Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (10)

Ähnlich wie Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N

Ähnlich wie Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N (20)

Mehr von Yahoo Developer Network

Mehr von Yahoo Developer Network (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N