MapReduce

Classification: Restricted
Agenda
Input Split & HDFS Block
Relation between Split & Block
Data locality Optimization
Speculative Execution
MR Flow with Single Reduce
Task
MR flow with multiple Reducers
Input Format & Hierarchy
Output Format & Hierarchy
Meet MapReduce
Word Count Algorithm – Traditional
approach
Traditional approach on a
Distributed System
Traditional approach – Drawbacks
MapReduce Approach
Input & Output Forms of a MR
program
Map, Shuffle & Sort, Reduce Phase
WordCount Code walkthrough
Workflow & Transformation of
Data

In pioneer days they used oxen for heavy pulling, and when on ox couldn’t
budge a log,they didn’t try to grow a larger ox. We shouldn’t be trying for
bigger computers, but for more systems of computers.

Meet MapReduce
• MapReduce is a programming model for distributed processing
• Advantage - easy scaling of data processing over multiple computing nodes
• The basic entities in this model are – mappers & reducers
• Decomposing a data processing application into mappers and reducers
is the task of developer
• once you write an application in the MapReduce form, scaling the
application to run over hundreds, thousands, or even tens of thousands of
machines in a cluster is merely a configuration change

WordCount – Traditional Approach
• Input: do as I say not as I do
• Output:
Word Count
as 2
do 2
I 2
not 1
say 1

WordCount – Traditional Approach
• The program loops through all the documents. For each document, the
words are extracted one by one using a tokenization process. For each
word, its corresponding entry in a multiset called wordCount is
incremented by one. At the end, a display ()function prints out all the
entries in wordCount.
• A multiset is a set where each element also has a count. The word count
we’re trying to generate is a canonical example of a multiset. In practice, it’s
usually implemented as a hash table.
define wordCount as Multiset;
for each document in documentSet {
T = tokenize(document);
for each token in T {
wordCount[token]++;
}
}
display(wordCount);

Traditional Approach – Distributed Processing
define wordCount as Multiset;
for each document in documentSubset {
< same code as in perv.slide>
}
sendToSecondPhase(wordCount);
define totalWordCount as Multiset;
for each wordCount received from firstPhase {
multisetAdd (totalWordCount, wordCount);
}

Traditional Approach – Drawbacks
• Central Storage – bottleneck in bandwidth of the server
• Multiple Storage – handling splits
• Program runs in memory
• When processing large document sets, the number of unique words can
exceed the RAM storage of a machine
• Phase 2 handling by one machine?
• If Multiple machines are used for phase-2, how to partition the data?

Mapreduce Approach
• Has two execution phases – mapping & reducing
• These phases are defined by data processing functions called – mapper &
reducer
• Mapping phase – MR takes the input data and feeds each data element to
the mapper
• Reducing phase – reducer processes all the outputs from the mapper and
arrives at a final result

Input & Output forms:
• In order for mapping, reducing, partitioning, and shuffling (and a few others
that were not mentioned) to seamlessly work together, we need to agree
on a common structure for the data being processed
• InputFormat class is responsible for creating input splits and dividing them
into records()
Input Output
map() <k1, v1> list(<k2, v2>)
reduce() <k2, list(v2)> list(<k3, v3>)

Input & Output forms:
• Input & output forms should be flexible and powerful enough to handle
most of the targeted data processing applications. MapReduce
uses lists and(key/value) pairs as its main data primitives.
• The keys and values are often integers or strings but can also be dummy
values to be ignored or complex object types.

Map Phase

Reduce Phase

Shuffle & sort Phase
• The Default partitioning is Hash-Partitioning

MR - Work flow & Transformation of data
From i/p
files to the
mapper
From the
Mapper to
the
intermediate
results
From
intermediate
results to
the reducer
From the
reducer to
output files

Word Count: Source Code
• Key points to note:
1.In MR, map() processes one record at a time, where as traditional
approaches process one document at a time.
2.The new classes that we have seen (Text, IntWritable, LongWritable etc.,)
have additional serialization capabilities. (Will discuss in detail later)
• Source Code: http://hadoop.apache.org/docs/current/hadoop-mapreduce-
client/hadoop-mapreduce-client-core/MapReduceTutorial.html

Input Split & Hdfs Block
Data Chunk
HDFS Block
(Physical Division)
Input Split
(Logical Division)

Relation Between Input Split & Hdfs Block
1 2 3 4 76 8 1
0
95
File
Lin
es
Block
Bounda
ry
Block
Bounda
ry
Block
Bounda
ry
Block
Bounda
ry
Split Split Split
• Logical records do not fit neatly into the HDFS blocks.
• Logical records are lines that cross the boundary of the blocks.
• First split contains line 5 although it spans across blocks.

Data locality Optimization
• MR job is split into various map &
reduce tasks
• Map tasks run on the input splits
• Ideally, the task JVM would be
initiated in the node where the
split/block of data exists
• While in some scenarios, JVMs might
not be free to accept another task.
• In that case, Task Tracker will be
initiated at a different location.
• Scenario a) Same node execution
• Scenario b) Off-node execution
• Scenario c) Off-rack execution

Speculative execution
• MR job is split into various map & reduce tasks and they get executed in
parallel.
• Overall job execution time is pulled down by the slowest task.
• Hadoop doesn’t try to diagnose and fix slow-running tasks; instead, it tries
to detect when a task is running slower than expected and launches
another equivalent task as a backup. This is
termed speculative execution of tasks.

MapReduce Dataflow With A Single Reduce Task

Map Reduce Dataflow With Multiple Reduce Tasks

MapReduce Dataflow With No Reduce Tasks

Combiner
• A combiner is a mini-reducer
• It gets executed on the mapper output at the mapper side
• Combiner’s output is fed to Reducer
• As the mapper output is further refined using combiner, data that has to be
shuffled across the cluster is minimized
• Because the combiner function is an optimization, Hadoop does not
provide a guarantee of how many times it will call it for a particular map
output record,
if at all
• So, calling the combiner function zero, one, or many times should produce
the same output from the reducer.

Combiner’s Contract
• Only those functions that obey commutative & associative properties can
use combiners.
• Because
max(0, 20, 10, 25, 15) = max(max(0, 20, 10), max(25, 15)) = max(20,
25) = 25
where as,
mean(0, 20, 10, 25, 15) = 14 and
mean(mean(0, 20, 10), mean(25, 15)) = mean(10, 20) = 15
Can a combiner replace a reducer?

Partitioner
• We know that a unique key will always go to a unique reducer.
• Partitioner is responsible for sending key, value pairs to a reducer based on
the key content.
• The default partitioner is Hash-partitioner. It takes mapper output, create a
Hash value for each key and divide it modulo by the number of reducers.
The output of this calculation will determine the reducer that this particular
key would go to

Partitioner
public class HashPartitioner<K, V> extends Partitioner<K, V> {
public int getPartition(K key, V value, int numReduceTasks) {
return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks;
}
}
2%3 = 1
3%3=0
4%3=1
5%3=2
6%3=0
2%4 = 2
3%4=1
4%4=0
5%4=1
6%4=2
7%4=3

Partitioner
Mapper
Mapper
Mapper
Reducer
Reducer
Reducer
Partitione
r
Partitione
r
Partitione
r

InputFormat Hierarchy

• An Input split is a chunk of the input that is processed by a single map. Each
map processes a single split. Each split is divided into records, and the map
processes each record—a key-value pair—in turn. Splits and records are
logical: there is nothing that requires them to be tied to files, for example,
although in their most common incarnations, they are.
• In a database context, a split might correspond to a range of rows from a
table and a record to a row in that range.
• An InputFormat is responsible for creating the input splits and dividing
them into records.

public abstract class InputFormat<K, V> {
public abstract List<InputSplit> getSplits(JobContext context) throws
IOException, InterruptedException;
public abstract RecordReader<K, V> createRecordReader(InputSplit split,
TaskAttemptContext context) throws
IOException, InterruptedException;
}
Client calls getSplits() & map task calls createRecordReader()
• FileInputFormat is the base class for all implementations
of InputFormat that use files as their data source
• It provides two things: a place to define which files are included as the input
to a job, and an implementation for generating splits for the input files. The
job of dividing splits into records is performed by subclasses.

public static void addInputPath(Job job, Path path)
public static void addInputPaths(Job job, String commaSeparatedPaths)
public static void setInputPaths(Job job, Path... inputPaths)
public static void setInputPaths(Job job, String commaSeparatedPaths)

InputFormat
Input Split Input SplitInput SplitInput Split
Record
Reader
Record
Reader
Record
Reader
Record
Reader
Mapper MapperMapperMapper

OutputFormat
Reducer
Output File
Reducer ReducerReducer
RcordWriter RcordWriterRcordWriterRcordWriter
Output FileOutput FileOutput File

OutputFormat Hierarchy

Counters
• Map input records
• Map output records
• Filesystem bytes read
• Launched map tasks
• Failed map tasks
• Killed reduce tasks
• Counters are a useful channel for
gathering statistics about the job:
for quality control or for
application-level statistics.
• Often used for debugging purpose.
• eg: Count number of Good records,
bad records in the input
• Two types – Built-in & Custom
Counters
• Examples of Built-in Counters:

Joins
• Map-side join(Replication): A map-side join that works in situations
where one of the datasets is small enough to cache
• Reduce-side join(Repartition join): A reduce-side join for situations where
you’re joining two or more large datasets together
• Semi-join(A map-side join): Another map-side join where one dataset is
initially too large to fit into memory, but after some filtering
can be reduced down to a size that can fit in memory

Distributed Cache
• Side data can be defined as extra read-only data needed by a job to process
the main dataset
• To make side data available to all map or reduce tasks, we distribute those
datasets using Hadoop’s Distributed Cache mechanism.

Map Join (Using Distributed Cache)

Some Useful Links:
• http://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html
• http://hadoop.apache.org/docs/current/hadoop-mapreduce-
client/hadoop-mapreduce-client-core/MapReduceTutorial.html

Thank you!

MapReduce

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (17)

Ähnlich wie MapReduce

Ähnlich wie MapReduce (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

MapReduce