Zh Tw Introduction To Map Reduce

MapReduce
• Google
• Google
•
• Map Reduce
• Google map reduce

• MapReduce
• Map
– [1,2,3,4] – (*2)  [2,3,6,8]
• Reduce
– [1,2,3,4] – (sum)  10

– (Divide and Conquer)

Copyright 2009 - Trend Micro Inc.

MapReduce
• MapReduce Google
Map Reduce
MapReduce
•
– Map ” ” key/value ” ”
intermediate key/value
– Reduce intermediate key
intermediate values key/value
• MapReduce


•
–
–
–

•

http://www.dbms2.com/2008/08/26/known-applications-of-mapreduce/


MapReduce
•
– map (K1, V1)  list(K2, V2)
– reduce (K2, list(V2))  list(K3, V3)

• grep
– Map: (offset, line)  [(match, 1)]
– Reduce: (match, [1, 1, ...])  [(match, n)]

• MapReduce :


6
Classification Copyright 2009 - Trend Micro Inc.

‧ ➝
‧ ➝


Word Count


MapReduce

• (Distributed Grep)
– (pattern)

• (Distributed Sort)
–

• URL (Count of URL Access Frequency)
– Web URL


MapReduce


Hadoop MapReduce

• Apache Hadoop Google MapReduce
– MapReduce
– Java
– Hadoop (HDFS)
• Yahoo!
• Google, Yahoo!, IBM, Amazon Hadoop
• (Trend Micro) Hadoop MapReduce


Hadoop MapReduce

• Map/Reduce framework
– JobTracker
– TaskTracker
• JobTracker
– Job
– Job JobTracker Job.
• TaskTrackers
• Job

Classification

Hadoop MapReduce
class MyJob {

class Map { // Map
}
class Reduce { // Reduce
}

}
main() {
// job
JobConf conf = new JobConf(“MyJob.class”);
conf.setInputPath(…);
conf.setOutputPath(…);
conf.setMapperClass(Map.class);
conf.setReduceClass(Reduce.class)
// Job
JobClient.runJob(conf);
}
}

•
–
–
–

–
HDFS MapReduce
•
–
–
•
•

, GUID, , ,
1, 123, 131231231, VSAPI, open file
2, 456, 123123123, VSAPI, connect internet


Map
• Mapper map()
• Map : (K1, V1)  list(K2, V2)

map( WritableComparable, Writable,
OutputCollector, Reporter)

• input map()
• OutputCollector collect() method

OutputCollector.collect( WritableComparable,Writable )


Map

class MapClass extends MapReduceBase

implements Mapper<LongWritable, Text, Text, IntWritable> {

private final static IntWritable one = new IntWritable(1);

private Text hour = new Text();

public void map( LongWritable key, Text value,
OutputCollector<Text,IntWritable> output, Reporter reporter) throws
IOException {

String line = ((Text) value).toString();

String[] token = line.split(quot;,quot;);

String timestamp = token[1];

Calendar c = Calendar.getInstance();

c.setTimeInMillis(Long.parseLong(timestamp));

Integer h = c.get(Calendar.HOUR);

hour.set(h.toString());

output.collect(hour, one)

}}} Copyright 2007 - Trend Micro Inc.

Reduce
• Reducer reduce() method
• Reduce : (K2, list(V2))  list(K3, V3)

reduce (WritableComparable, Iterator,
OutputCollector, Reporter)

• OutputCollector collect() method

OutputCollector.collect( WritableComparable,Writable )


Reduce

class ReduceClass extends MapReduceBase implements Reducer< Text,
IntWritable, Text, IntWritable> {

IntWritable SumValue = new IntWritable();

public void reduce( Text key, Iterator<IntWritable> values,

OutputCollector<Text, IntWritable> output, Reporter reporter)

throws IOException {

int sum = 0;

while (values.hasNext())

sum += values.next().get();

SumValue.set(sum);

output.collect(key, SumValue);

}}


• JobConf
– Mapper Reducer Inputformat OutputFormat Combiler Petitioner

–
–
–
• map reduce
•

• JobClient JobConf

JobClient.submitJob(conf);
JobClient.setJobEndNotificationURI(URI);


Main Function
Class MyJob{
public static void main(String[] args) {
JobConf conf = new JobConf(MyJob.class);
conf.setJobName(”Caculate feedback log time distributionquot;);
// set path
conf.setInputPath(new Path(args[0]));
conf.setOutputPath(new Path(args[1]));
// set map reduce
conf.setOutputKeyClass(Text.class); // set every word as key
conf.setOutputValueClass(IntWritable.class); // set 1 as value
conf.setMapperClass(MapClass.class);
conf.setCombinerClass(Reduce.class);
conf.setReducerClass(ReduceClass.class);
onf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
// run
}}


1.
– javac -classpath hadoop-*-core.jar -d MyJava
MyJob.java
2.
– jar –cvf MyJob.jar -C MyJava .
3.
– bin/hadoop jar MyJob.jar MyJob input/ output/


• bin/hadoop jar MyJob.jar MyJob input/ output/

Classification

Web Console
http://172.16.203.132:50030/

Classification

Hadoop MapReduce
• Mapper ?
– Mapper Input Input Hadoop
Mapper
– JobConf setNumMapTasks(int) Hadoop
Mapper Hadoop

• Reducer ?
– JobConf JobConf.setNumReduceTasks(int)
Reducer
– Reducer Reducer
MapReduce Map Reduce

Classification

Non-Java Interface
• Hadoop Pipes
– MapReduce C++ API
– C++ java
• Hadoop Streaming
– MapReduce


• Google MapReduce
– http://labs.google.com/papers/mapreduce.html
– http://code.google.com/edu/submissions/mapreduce/listing.html
– http://code.google.com/edu/submissions/mapreduce-minilecture/
listing.html
• Hadoop
– http://hadoop.apache.org/core/


• Eclipse MapReduce (IBM
)
– Eclipse Hadoop
–
• http://code.google.com/edu/parallel/tools/hadoopvm/hadoop-
eclipse-plugin.jar
• Hadoop (Google )
– VMware Hadoop

VMware Google
–
• http://code.google.com/edu/parallel/tools/hadoopvm/
index.html


Zh Tw Introduction To Map Reduce

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Zh Tw Introduction To Map Reduce

Ähnlich wie Zh Tw Introduction To Map Reduce (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Zh Tw Introduction To Map Reduce