- Provides dynamic mechanism for controlling number of splitters, mappers, reducers - Claimed to improve utilization of servers - Kalman Filter found to be more effective by authors in prediction algorithms -Full article post on http://www.hadoopsphere.com/

1. hadoopsphere.com
Adaptive MapReduce for real
time applications
hadoopsphere

2. Source
• AMREF, an Adaptive MapReduce
Framework designed for real time data
intensive applications. (published in the
paper Fan Zhang, Junwei Cao, Xiaolong
Song, Hong Cai, Cheng Wu: AMREF: An
Adaptive MapReduce Framework for Real
Time Applications. GCC 2010: 157-162.)
• Full article on http://www.hadoopsphere.com
hadoopsphere.com 2

3. Adaptive Splitter
the ‘Adaptive splitter’ would
• in stage 1 distribute the input file evenly to
the mappers
• in stage 2, different mappers with different
processing capacity would have different
length of input files
• in stage 3, a new input file is distributed to
the three mappers according to their
processing capacity.
hadoopsphere.com 3

4. Adaptive Splitter – Stage 1
split () map()
split () map()
split () map()
hadoopsphere.com 4

5. Adaptive Splitter – Stage 2
split () map()
split () map()
split () map()
hadoopsphere.com 5

6. Adaptive Splitter – Stage 3
split () map()
split () map()
split () map()
hadoopsphere.com 6

7. Adaptive Mapper
map() reduce()
map() reduce()
split ()
map() reduce()
map()
An adaptive mapper is added dynamically if it is observed that there is an overburden on
the other mappers
hadoopsphere.com 7

8. Adaptive Reducer
map() reduce()
map() reduce()
map() reduce()
map() reduce()
When the output of mappers are too fast for the number of reducers, an adaptive
reducer is added in parallel to produces output
hadoopsphere.com 8

9. Comments
• Provides dynamic mechanism for controlling
number of splitters, mappers, reducers
• Claimed to improve utilization of servers
• Kalman Filter found to be more effective by
authors in prediction algorithms
• Full article post on
http://www.hadoopsphere.com/2012/10/adapting-m
hadoopsphere.com 9