Hw09 Fingerpointing Sourcing Performance Issues

Jiaqi Tan, Soila Pertet, Xinghao Pan, Mike Kasick, Keith Bare, Eugene Marinelli, Rajeev Gandhi Priya Narasimhan Carnegie Mellon University

Automated Problem Diagnosis ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Challenges in Problem Analysis ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Exploration of Fingerpointing ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Why? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Hadoop Failure Survey ,[object Object],Targeted Failures: 66% Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Hadoop Mailing List Survey ,[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

M45 Job Performance Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University ,[object Object],[object Object],[object Object]

BEFORE : Hadoop Web Console ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

AFTER : Goals, Non-Goals ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Target Hadoop Clusters ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Performance Problems Studied Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University Studied Hadoop Issue Tracker (JIRA) from Jan-Dec 2007 Fault Description Resource contention CPU hog External process uses 70% of CPU Packet-loss 5% or 50% of incoming packets dropped Disk hog 20GB file repeatedly written to Disk full Disk full Application bugs Source: Hadoop JIRA HADOOP-1036 Maps hang due to unhandled exception HADOOP-1152 Reduces fail while copying map output HADOOP-2080 Reduces fail due to incorrect checksum HADOOP-2051 Jobs hang due to unhandled exception HADOOP-1255 Infinite loop at Nameode

Hadoop: Instrumentation Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University JobTracker NameNode TaskTracker DataNode Map/Reduce tasks HDFS blocks MASTER NODE SLAVE NODES Hadoop logs OS data OS data Hadoop logs

How About Those Metrics? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Intuition for Diagnosis ,[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Log-Analysis Approach ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Carnegie Mellon University Priya Narasimhan © Oct 25, 2009

Applying SALSA to Hadoop Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University [ t] Launch Map task : [t] Copy Map outputs : [t] Map task done Map outputs to Reduce tasks on other nodes Data-flow view: transfer of data to other nodes ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Incoming Map outputs for this Reduce task Control-flow view: state orders, durations

Distributed Control+Data Flow ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],<your name here> © Oct 25, 2009 http://www.pdl.cmu.edu/

Intuition: Peer Similarity Oct 25, 2009 Carnegie Mellon University ,[object Object],[object Object],[object Object],Faulty node Normalized counts (total 1.0) Histograms (distributions) of durations of WriteBlock over a 30-second window Normal node Normal node Normalized counts (total 1.0) Normalized counts (total 1.0)

What Else Do We Do? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Putting the Elephant Together Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University TaskTracker heartbeat timestamps Black-box resource usage JobTracker Durations views TaskTracker Durations views JobTracker heartbeat timestamps Job-centric data flows BliMEy: Bli nd Me n and the E lephant Framework [ CMU-CS-09-135 ]

Visualization ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Visualization ( timeseries ) Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University DiskHog on slave node visible through lower heartbeat rate for that node

Visualization( heatmaps ) Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University CPU Hog on node 1 visible on Map-task durations

Visualizations ( swimlanes ) Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University Long-tailed map Delaying overall job completion time

MIROS ,[object Object],[object Object],[object Object],[object Object],[object Object],Jiaqi Tan © July 09 http://www.pdl.cmu.edu/

Current Developments ,[object Object],[object Object],[object Object],[object Object],<your name here> © Oct 25, 2009 http://www.pdl.cmu.edu/

Briefly: Online Fingerpointing ,[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Hard Problems ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

Summary ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Priya Narasimhan © Oct 25, 2009 Carnegie Mellon University

priya@cs.cmu.edu Oct 25, 2009 Carnegie Mellon University

Hw09 Fingerpointing Sourcing Performance Issues

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (7)

Ähnlich wie Hw09 Fingerpointing Sourcing Performance Issues

Ähnlich wie Hw09 Fingerpointing Sourcing Performance Issues (20)

Mehr von Cloudera, Inc.

Mehr von Cloudera, Inc. (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Hw09 Fingerpointing Sourcing Performance Issues

Hinweis der Redaktion