Past Present and Future of Data Processing in Apache Hadoop

Data Processing with Hadoop
Looking Back, Looking Ahead

Arun C. Murthy
Founder & Architect
@acmurthy (@hortonworks)

Page 1

Hello!
• Founder/Architect at Hortonworks Inc.
– Lead - Map-Reduce/YARN/Tez
– Formerly, Architect Hadoop MapReduce, Yahoo
– Responsible for running Hadoop MapReduce as a service for all
of Yahoo (~50k nodes footprint)

• Apache Hadoop, ASF
– Frmr. VP, Apache Hadoop, ASF (Chair of Apache Hadoop PMC)
– Long-term Committer/PMC member (full time for 7 years)
– Release Manager for hadoop-2.x

© Hortonworks Inc. 2013 Page 2

Once upon a time …

… long, long ago, there was a kingdom we shall call
Apache Hadoop
http://2.bp.blogspot.com/-hIp99urgxCk/UAsSFo4i8YI/AAAAAAAAAFg/IzjNDwrBBVg/s1600/magickingdo


Hadoop begat …

… a two-headed monster on every node in the kingdom;
each belonged to a different clan and answered to a
different master
http://4.bp.blogspot.com/_C7CsfdqySYc/TNSKvIwiFcI/AAAAAAAAAbs/2FSU2TV_rRA/s1600/Two-Headed+Monster+-+With+Identifiers+-+Jan+19,+2009_0.jpg


Knights of Bytes - HDFS

… stored data uncompromisingly in directories/files, nary a
care about contents
http://whoiscraigmoser.com/Images/identity/knight.png


Prince of Processing - MapReduce

He ruled with an iron fist by mapping,
and then by mercilessly reducing data http://media.comicvine.com/uploads/14/144886/2868181-sauron.jpg


Peace Reigned

… for a while with the odd change in the direction of the wind

http://www.get-covers.com/wp-content/uploads/2012/07/Peace.jpg


Slowly, but surely …

Human beings define reality through misery and suffering.
- Agent Smith
http://api.ning.com/files/*oWmhl7LBlXuodD2itWUUtOautEVfD*pbBn57L8ThCyYIykiTuzkO4lJY1bwaNbJF7GecTDwsVj3EFHpDM-F1y-UW4b3Xsvh/matrix_revolutions_agent_smith_04.bmp



Human beings define reality through misery and suffering.
- Agent Smith
http://api.ning.com/files/*oWmhl7LBlXuodD2itWUUtOautEVfD*pbBn57L8ThCyYIykiTuzkO4lJY1bwaNbJF7GecTDwsVj3EFHpDM-F1y-UW4b3Xsvh/matrix_revolutions_agent_smith_04.bmp



… people of the kingdom clamored for more.
A palpable sense of greed & expectation.
http://sidoxia.files.wordpress.com/2011/11/wall-st-greed-st1.jpg


Signs of Distress

SQL said some, others said Machine Learning,
still others said Real-Time Event Processing
http://www.truth-seeker.info/wp-content/uploads/2012/11/distress.jpg


A Meeting at the Summit

MapReduce is dead!
Err… not quite.
We need more options! We need more!
True…
http://4.bp.blogspot.com/-
oqr1t6avx6g/TW55kUnmQvI/AAAAAAAAMMk/q9Jc87MSG4g/s400/arab%2Bleague%2Bround%2Btable%2B%2Bbig%2Bgood%2B2011.bmp


A Meeting at the Summit

A common thread YARN running through all applications…
Long live the King!

http://whipup.net/wp-content/images/2008/08/yarn.gif


The Edict

Henceforth, in the Kingdom of King YARN…
MapReduce has been relegated to the status
of, merely, one of the applications!
http://www.napavintners.org/images/winery_Labels/EdictWines-800HW.jpg


Reign of King YARN

King YARN came to throne
with promises to return power
to all applications
equally, lower performance
taxes and resource
management…

http://images.fineartamerica.com/images-medium-large/the-coronation-the-crown-that-queen-everett.jpg


Oh the Shame!

Well, at least, Prince
MapReduce still had
powerful allies like
Highness
Hive, Powerful
Pig, Cheery
Cascading…

http://www.gibbsmagazine.com/MPj03414090000%5B1%5D.jpg


Things get worse before better

Unfortunately, things got a lot worse for the Prince MapReduce…
http://www.deviantart.com/download/144412184/Smile__Tomorrow_will_be_worse__by_daGrevis.jpg


Knight Tez

He did MapReduce, and so much more…
Smartly aligned himself to Kingdom YARN.
http://twomorrows.com/alterego/media/08shiningknight.gif


Knight Tez

Long term alliances of MapReduce with
Hive, Pig, Cascading etc. broke up…

… they decided to throw their
lot with Knight Tez!
http://informatica.upg-ploiesti.ro/62689/img/partners.jpg
http://www.officialpsds.com/images/thumbs/broken-glass-psd44132.png


Happily ever after…

(nothing cute to say)


On a more serious note…


Every season has a flavor…

SQL-on-Hadoop is the new black!

SQL-on-Hadoop will be solved within
the existing ecosystem


Looking ahead

What will it be next year?

Real-time event processing?
Machine Learning?


Play to our strengths

Invest in the Apache Hadoop platform
and the ecosystem (Hive et al).


Seriously…
Technical Details


Hadoop MapReduce – The System


Hadoop MapReduce – The Paradigm

m m0 m1 m2 m3 m4

r r0 r1 r2


Hadoop YARN

Node
Node
Manager
Manager

Container App Mstr
App Mstr

Client

Resource Node
Node
Resource
Manager
Manager Manager
Manager
Client
Client

App Mstr Container
Container

MapReduce Status Node
Node
MapReduce Status
Manager
Manager
Job Submission
Job Submission
Node Status
Node Status
Resource Request
Resource Request Container Container

Tez - Core Ideas

Task <Input, Processor & Output>

Input Processor Output

Task

Tez Task - <Input, Processor, Output>

YARN ApplicationMaster to run DAG of Tasks


Pig/Hive-MR versus Pig/Hive-Tez

SELECT a.state, COUNT(*)
FROM a JOIN b ON (a.id = b.id)
GROUP BY a.state

I/O Synchronization
I/O Pipelining
Barrier

Pig/Hive - MR Pig/Hive - Tez


Pig/Hive-MR versus Pig/Hive-Tez
SELECT a.state, COUNT(*), AVERAGE(c.price)
FROM a
JOIN b ON (a.id = b.id)
JOIN c ON (a.itemId = c.itemId)
GROUP BY a.state

Job 1

Job 2

I/O Synchronization
Barrier

Job 3

I/O Synchronization
Barrier

Single Job

Job 4

Pig/Hive - MR Pig/Hive - Tez

Thank You!
Questions (surely) & Answers (maybe)

@acmurthy


Past Present and Future of Data Processing in Apache Hadoop

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Past Present and Future of Data Processing in Apache Hadoop

Ähnlich wie Past Present and Future of Data Processing in Apache Hadoop (10)

Mehr von DataWorks Summit

Mehr von DataWorks Summit (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Past Present and Future of Data Processing in Apache Hadoop