Presentation by Dr. Kostas Tzoumas at the Stratosphere Summit [1] (Nov. 14-15, 2013) introducing the Stratosphere Platform for Big Data Analytics. Check out http://stratosphere.eu for more information. [1] http://stratosphere.eu/events/2013/summit.html

1. Ask “what,”
not “how”
Kostas Tzoumas

2. Data is an important asset
video & audio streams, sensor data, RFID, GPS, user online
behavior, scientific simulations, web archives, ...
Volume
Handle petabytes of data
Velocity
Handle high data arrival rates
Variety
Handle many heterogeneous data sources
Veracity
2
Handle inherent uncertainty of data

3. Data
Analysis
3

4. Four “I”s for Big Analysis
text mining, interactive and ad hoc analysis, machine
learning, graph analysis, statistical algorithms
Iterative
Model the data, do not just describe it
Incremental
Maintain the model under high arrival rates
Interactive
Step-by-step data exploration on very large data
Integrative
4
Fluent unified interfaces for different data models

5. Map
Reduce
MapReduce
and Hadoop
Co
(Romeo, (1,1,1))
(art, (1,1))
(thou, (1,1))
(What, 1)
(art, 1)
(thou, 1)
(hurt, 1)
Data written
to disk
(Romeo, 3)
(art, 2)
(thou, 2)
(wherefore, 1)
(What, 1)
(hurt, 1)
40
(Romeo, 1)
(wherefore, 1)
(art, 1)
(thou, 1)
(Romeo, 1)
Reduce
Match
Reduce
“What, art thou
hurt?”
Map
“Romeo, Romeo,
wherefore art thou
Romeo?”
Map
Cross
(Romeo, 1)
(wherefore, 1)
(What, 1)
(hurt, 1)
Data shuffled
over network
5

6. SQL analytics
with Hadoop
Pitfalls:
Reduce
Reduce
Reduce
Map
Map
Map
Lacking in
declarativity
6
HDFS-based
data exchange
Sort the only
grouping operator
Hadoop engine
tailored to simple
aggregations

7. SQL
MapReduce
BigAnalytics
BigSQL
NoMapReduce

8. Advanced
Analytics
Analytics that model the data to reveal hidden
relationships, not just describe the data.
E.g., machine learning, statistics, graph analysis
Increasingly important from a market perspective.
Very different than SQL analytics: different languages and
access patterns (iterative vs. one-pass programs).
Hadoop toolchain poor; R, Matlab, etc not parallel.
8

9. Use case in
all verticals
Media and Communications
Example: Risk management, analytics on
phone call logs, risk management,
sentiment analysis, clickstream and call
analysis
Manufacturing
Example: Data-driven quality
control and assurance, demand
forecasting, sales and operation
planning, process optimization
Travel and tourism
Example: Improve personalized customer
experience in hotels, estimate no-show in
flights, route planning
Retail
Example: Improve campaign ROI
by optimizing advertising channels,
market basket analysis, fraud
detection, social trend analysis,
product recommendation
Social and e-commerce
Example: Targeted customer experience,
explore new business models, real-time
recommendations, social graph analysis,
game analytics
9

10. Big data lives in Hadoop. Hadoop clusters
offer very low effective storage cost, and are
becoming a data vortex, attracting crossdepartmental data.
Companies want to perform advanced and
predictive analytics to maximize ROI of their
data assets by modeling the data, not just
describing it.
How do we bring
advanced analytics to
the world of big data?
10

11. What,
not how
Recipe for success:
declarativity
User specifies what information to
extract out of the data, not how
the system extracts the information.
This is what relational databases
pioneered in the 70s resulting in a
vibrant research community and a
billion dollar industry.
11
Big data consumers in
the future
people with data
analysis skills
systems
programming
experts
Big data
consumers now

12. Desiderata for next-gen big
data platforms: Usability
10 million
Excel users
3 million
R users
70,000
Hadoop
users
12
“the market faces
certain challenges
such as unavailability
of qualified and
experienced work
professionals, who can
effectively handle the
Hadoop architecture.”

13. Desiderata for next-gen big
data platforms: Performance
Stratosphere!
Hadoop!
0!
100!
200!
300!
400!
500!
600!
700!
Performance difference from days to minutes enables
real time decision making and widespread use of data
within the organization.
13

14. How to lift
declarativity from
the closed world of
relational algebra to
the open world of
advanced analytics.
14

15. Step 1: Specify
//"get"the"customers"with"their"debit"
val"debits:((String,(Double)(=(sql(
(((("SELECT&customerId,&debit&FROM&customer_accounts;")
//"get"the"number"of"warned"invoices"in"the"last"
//"12"and"6"months
val"warnings:((String,(Int,(Int)(=(sql
"""""SELECT&R12.customerId,&R12.cnt,&R6.cnt
&&&&&&&&&&&&FROM&(…)&R12&LEFT&OUTER&JOIN&(…)&R6
&&&&&&&&&&&&&&ON&(R6.customerId&=&R12.customerId);")
//"number"of"contracts"a"customer"has
val"numContracts(:((String,(Int)(=(sql(
(((("SELECT&customerId,&numContracts&FROM&customers;")
//"join"the"data"into"one"data"point
case"class"DataPoint(x:(Vector,(y:(Double)
val(dataPoints(=(numContracts(
((join(warnings
((where({_._1}(isEqualTo({_._1}
((join(debits
((where({_._1}(isEqualTo({_._1}
""map({((x,y,z)(=>(DataPoint(Vector(x._2,(y._2,(y._3),
(((((((((((((((((((((((((((((if((z._2(>(X)(1(else(0)(}
//"run"regression"with"dimensionality"3"for"40"iterations
val(weights:(Vector(=(logRegression(3,(dataPoints,(40)
Unify data and
programming models in
a declarative abstraction.
SQL for extracting
enterprise data from
databases.
General-purpose
programming for feature
extraction and
normalization.
Statistical libraries for
advanced analysis.
15

16. First step for
declarative
analytics
Scala: functional and object-oriented JVM language,
excellent basis for domain-specific language
development. Coolest kid in the block ☺
Feels like a scripting language, but is not restricted
to a fixed data model like Pig, Hive, etc.
Scala’s extensible compiler architecture is a good
match for implementing optimizers.
16

17. Step 2: Optimize
Data characteristics change
Each color is a differently written
program that produces the same result but has very
different performance depending on small changes
in the data set and the analysis requirements
Query optimizers: the
enabling technology for SQL
data warehousing and BI
Successful industrial
application of artificial
intelligence
Data characteristics change
(a) Complex Plan Diagram
Currently, no other system
can optimize non-relational
data analysis programs.
17
(b) Reduced Plan Diagram

18. filter
MATC
aggregate
project
Enumeration Algorithm: 4 5 6 7 8r0
0 123
r0 = this Read Set
UDF Code Analysis
0 123456 78
= this
r1 = @parameter0 // Iter
ter0 //
project Iterator
r2 = @parameter1 // Coll
ter1 // Collector
Use a combination of compiler and
• Checks reorder conditions and switches successive operators
database technology to lift optimization
Supported Transformations:
• Filter push-down
beyond relational algebra. Derive
• Join reordering
• Invariant group transformations
properties of user-defined functions via
• Non-relational operators are integrated
code analysis and use these to mimic a
relational database optimizer.
r0 = this
r1 = @parameter0 // Record
/ Reco
Prerequisites:
r2 = @parameter1 // Collector
/ Coll
r1 = @parameter0 // Record
/ Reco
• Descents data //filter
r2 = @parameter1 ow recursively top-down
/ Collector
Coll
$r5 = r1.getField(8)
8)
Control-Flow, Def-Use, Use-Def lists
$r6 = r0.date_lb
$i0 = $r5.compareTo($r6)
o($r6)
• Fixed API to access records
$d0 = r1.getField(6) r0 = this
r1
/ Reco
r3 = r1.next() = @parameter0 // Record
r0.extendedprice = $d0 = @parameter0 // Record = r
r2 = @parameter1 // Collector
/ Coll
r1
/ Reco
d0
r3.getField(4)
)
$d1 = r1.getField(7) r2 = @parameter1 // Collector
/ Coll
$d0 = r1.getField(6)
goto 2
r0.discount = $d1
r0.extendedprice = $d0
$r5 = r1.getField(8)
8)
$d1
$r6 = r0.date_lb
1: r3 = r1.next()= r1.getField(7)
$r7 = r0.revenue // PactRecord
r0.discount = $d1
$i0 = $r5.compareTo($r6)
o($r6)
$d1 = r3.getField(4)
Extracted Information:
$d2 = r0.extendedprice $i0 < 0 goto 1
if
$r9 = r1.getField(8)
8)
d0 = d0 + $d1 $r7 = r0.revenue // PactRecord
$d3 =
• Field sets= r0.date_ub write accesses on records r0.discount
track read and
$d2 = r0.extendedprice
$r10
$r9 = r1.getField(8)
8)
$d4 = 0 - $d3
2: $z0 = r1.hasNext()
$d3 = r0.discount
• Upper and$r9.compareTo($r10)
$i1 = lower output cardinality bounds
$r10 = r0.date_ub
$d5 = $d2 * $d4
$d4 1
if $z0 != 0 goto = 0 - $d3
if $i1 >= 0 goto 1
$i1 = $r9.compareTo($r10)
$d5 = $d2 * $d4
$r7.setValue($d5)
if $i1 >= 0 goto 1
r3.setField(4, d0)
4,
Safety:
$r7.setValue($d5)
r1.setNull(6)
)
r2.collect(r1)
r2.collect(r3)
r1.setNull(6)
)
• All record access instructions are detected
r2.collect(r1)
r1.setNull(7)
r1.setNull(7)
• Supersets of actual Read/Write sets are returned
r1.setNull(8)
1: return
r1.setNull(8)
1: return
if $i0 < 0 goto 1
• Supersets allow fewer but always safe transformations
$r8 = r0.revenue
nds
0 123456 78
Details Set[HPS+12] Bounds
Write in Ou
et Out-Card
[0,1]
0 123456 78
0 123456 78
[0,1]
Data Flow Transformations
Reorder Conditions:
output
[0,1]
0 123456 78
[0,1]
0 123456 78
Physical Optimization
output
0 123456 78
0 12345678
0 12345678
0 123456 78
5
0 123456 78
0
0 123456 78
0 123456 78
0 123456 78
5
0 12345678
MATCH
5
0 123456 78
0 123456 78
0 123456 78
0
0 123456 78
aggregate
0 123456 78
0 123456 78
0 12345678
0 123456 7
supplier8
Interesting Properties:
5
0 123456 78
0
0 123456 78
0 123456 78
0 123456 78
0 123456 78
• Checks reorder conditions and switches successive operators
REDUCE
join
0 12345678
REDUCE
aggregate
MAP
supplier
project
5
0 123456 78
0 123456 78
0 123456 78
Details in [HPS+12]
0 123456 78
0 123456 78
0 123456 78
0 123456 78
0 123456 78
0 123456 78
0 123456 78
REDUCE
aggregate
0 123456 78
0 123456 78
0 123456 78
MAP
lineitem
filter
Partition
project
output
lineitem
Execution Plan Selection:
Details in [BEH+10]
• Chooses execution strategies for 2nd-order functions
Local Forward
MATCH
Hybrid-Hash
• Chooses shipping strategies to distribute data
[WK09]
Warneke, Kao,
• Strategies known from parallel databases
Local Forward
Parallel Execution
REDUCE
0 12345678
0 12345678
0 12345678
0 12345678
0 12345678
0 12345678
Partition
supplier
MAP
filter
0 123456 78
MAP
• Massively parallel execution of
18 DAG-structured data ows
• Sequential processing tasks
•
•
R
0 123456 78
0 12345678
0 12345678
0 12345678
0 12345678
MAP
0 123456 78
Pipeline
Local Forward
0 12345678
0 12345678
•
lineitem
MAP
Pipeline
•
0 123456 78
Execution Engine:
•
0 123456 78
0 123456 78
lineitem
E
filter
COMBINE 3 4 5 6 7 8
0 12
0 1
Part-Sort2 3 4 5 6 7 8
MAP
Local Forward
project
0 123456 78
lineitem
0 123456 78
Partition
REDUCE
0 123456 78
• Exploits UDF annotations for size estimates
• Cost model combines network, disk I/O and CPU costs
Parallel Execution
lineitem
Pa
0 123456 78
aggregate
supplier
MAP
0 123456 78
0 123456 78
hysical Optimization
[0,1]
5
REDUCE 2 3 4 5 6 7 8
0 1
Sort 0 1 2 3 4 5 6 7 8 supplier
0 123456 78
project
0 123456 78
0 123456 78
supplier
0 123456 78
0 123456 78
MAP
Cost-based Plan Selection:
0 123456 78
0 123456 78
Local Forward
5
0 123456 78
0 123456 78
filter
lineitem
Local Forward
MATCH
join
MATCH
5
0
0
Hybrid-Hash 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8
0 123456 78
0 123456 78
0 123456 78
0 123456 78
0 123456 78
0 123456 78
0 123456 78
0 123456 78
MAP
0
0 123456 78
0 123456 78
• Sorting, Grouping, Partitioning
MAP
MAP
MAP
supplier
filter
filter
project
• Invariant group project
transformations
• Property preservation reasoning with write sets
• Non-relational operators are integrated
filter
0 123456 78
0 12345678
5
0 123456 78
0 123456 78
0
• Filter push-down 1 2 3 4 5 6 7 8
0 123456 78
• Join reordering MAP
r3.setField(4, d0)
4,
r2.collect(r3)
MAP
0 12345678
0 12345678
• Chooses shipping strategies to distribute data
join
Enumeration Algorithm:
• Descents data ow recursively top-down
• Strategies known from parallel databases
MATCH
Supported Transformations:
2: $z0 = r1.hasNext()
if $z0 != 0 goto 1
0 123456 78
5
0 123456 78
0
0 123456 78
0 123456 78
0 123456 78
0 123456 78
5
0 123456 78
0 123456 78
0 123456 78
0 12345678
0 123456 78
0 12345678
0 12345678
0 123456 78
0 123456 78
1: r3 = r1.next()
$d1 = r3.getField(4)
d0 = d0 + $d1
output
0 12345678
0 12345678
0 123456 78
0 12345678
5
0 123456 78
project
goto 2
output
output
0 12345678
0 12345678
REDUCE
MATCH
Execution Plan Selection: aggregate
join
2. Preservation of groupsREDUCE
for grouping operators
nd-order functions
MATCH
• Chooses executionMATCH
strategies for 2
• Groups must remain unchanged or be completely removed
join
aggregate
join
0 123456 78
MAP
r3 = r1.next()
d0 = r
r3.getField(4)
)
output
output
1. No Write-Read / Write-Write con icts on 0 1 2 3 4 5 6 7 8
record elds
0 12345678
0 12345 78
• Similar to con ict detection 6in optimistic 1concurrency control
0 2345678
[0,1]
0
r0 = this
r1 = @parameter0 // Iter 0 1 2 3 4 5 6 7 8
ter0 // Iterator
r2 = @parameter1 // Coll 0 1 2 3 4 5 6 7 8
ter1 // Collector
$r8 = r0.revenue
r1.setField(4, $r8)
r2.collect(r1)
1)
r1.setField(4, $r8)
r2.collect(r1)
1)
Details in [HPS+12] and [HKT12]
5
0 123456 78
aggregate
r0 = this
• Static Code Analysis Framework provides
join
0 123456 78
Local Forward
output
output
output
lineitem
MATCH
Hybrid-Hash
REDUCE
Sort
MATCH
Hybrid-Hash
supplier
REDUCE
Sort
MATCH
Hybrid-Hash
supplier
REDUCE
Sort
D
supplier
[H

19. Step 3:
Execute
one pass
dataflow
many pass
dataflow
MapReduce
Impala, ...
Stratosphere
Text
✔
✔
✔
Aggregation
✔
✔
✔
ETL
✔
✔
✔
SQL
Hive is too
slow
✔
✔
Advanced
analytics
Mahout is slow
and low level
Madlib is
too slow
✔
map
reduce
A fast, massively parallel
database-inspired backend.
Truly scales to diskresident large data sets.
Built-in support for iterative
programs: predictive and
advanced analytics (machine
learning, graph processing,
stats) are all iterative.
19

20. Stratosphere is an award-winning open-source platform:
15 man-years of R&D,150k LOC, 3 million € behind it.
IBM Faculty
Award
HP Open
Innovation
Award
Stratosphere is the only Hadoop-compatible nextgeneration big data analytics platform developed in
Europe that you can download and use right now.
20

21. Stratosphere
Sky in Java
Sky in Scala
Monitoring tools, e.g., Hue
Visualization and reporting tools, e.g., Datameer
Other HLLs:
SQL, R, ...
Compiler and optimizer
Hadoop
MapReduce,
Impala, ...
Runtime engine
Hadoop storage and cluster management: HDFS,Yarn
21

22. www.stratosphere.eu/
downloads
22

23. www.stratosphere.eu/quickstart
23

24. val(input(=(TextFile(textInput)
val(words(=(input
"".flatMap(
{(line(=>(line.split(“(“)(}
val(counts(=(words
((.groupBy(
((((((({(word(=>(word(}
((.count()(
val(output(=(counts
.write"(wordsOutput,
((((((((RecordDataSinkFormat()()
val(plan(=(new(ScalaPlan(Seq(output))
24

25. Help us shape the future of
Big Data and the
Stratosphere platform!
We are looking for contributions and pilot customers:
github.com/stratosphere/stratosphere/wiki/Starter-Jobs
Try out Stratosphere and give us feedback
Work with us to implement your use case
www.stratosphere.eu
www.github.com/stratosphere
Contact kostas.tzoumas@tu-berlin.de
Visit
Tweet
#StratoSummit