Apache Hive and its HiveQL interface has become the de facto SQL interface for Hadoop. Apache Hive was originally built for large-scale operational batch processing and it is very effective with reporting, data mining, and data preparation use cases. These usage patterns remain very important but with widespread adoption of Hadoop, the enterprise requirement for Hadoop to become more real time or interactive has increased in importance as well. Enabling Hive to answer human-time use cases (i.e. queries in the 5-30 second range) such as big data exploration, visualization, and parameterized reports without needing to resort to yet another tool to install, maintain and learn can deliver a lot of value to the large community of users with existing Hive skills and investments. To this end, we have launched the Stinger Initiative, with input and participation from the broader community, to enhance Hive with more SQL and better performance for these human-time use cases. We believe the performance changes we are making today, along with the work being done in Tez will transform Hive into a single tool that Hadoop users can use to do report generation, ad hoc queries, and large batch jobs spanning 10s or 100s of terabytes. Presenter(s): Alan Gates, Co-founder, Hortonworks and Apache Pig PMC and Apache HCatalog PPMC Member, Author of "Programming Pig" from O'Reilly Media Owen O’ Malley, Co-founder, Hortonworks, First committer added to Apache Hadoop and Founding Chair of the Apache Hadoop PMC

1. Stinger Tech Deep
Dive
Page 1
Alan Gates, Arun Murthy, Owen O’Malley

2. Hadoop Summit
Page 2Architecting the Future of Big Data
• June 26-27, 2013- San Jose Convention Cntr
• Co-hosted by Hortonworks & Yahoo!
• Theme: Enabling the Next Generation
Enterprise Data Platform
• 90+ Sessions and 7 Tracks:
• Community Focused Event
– Sessions selected by a Conference Committee
– Community Choice allowed public to vote for
sessions they want to see
• Training classes offered pre event
– Apache Hadoop Essentials: A Technical
Understanding for Business Users
– Understanding Microsoft HDInsight and Apache
Hadoop
– Developing Solutions with Apache Hadoop –
HDFS and MapReduce
– Applying Data Science using Apache Hadoop
• HUG member discount 10%
Use code: 13DiscHUG10
hadoopsummit.org

3. Stinger Deep Dive Overview
Page 3
© 2013 Hortonworks, DO NOT SHARE, HORTONWORKS
CONFIDENTIAL
• Hive Performance Near Term
• Support for Analytics
• ORCFile
• Tez
• Hive Performance Longer Term

4. Hive Performance Near Term
Page 4
© 2013 Hortonworks, DO NOT SHARE, HORTONWORKS
CONFIDENTIAL
• Enable star joins
– Where possible do in single map only task
– When not possible push larger tables to separate tasks
• Collapse adjacent jobs where possible
– Hive has lots of M->MR type plans, collapse these to MR
– Collapse adjacent jobs on sufficiently similar keys when feasible
– join followed by group
– join followed by order
– group followed by order
– Much of this ongoing in the community, we are shepherding
• Remove client side hash table generation
– Need to prove this is worth the effort short term

5. © Hortonworks Inc. 2013
TL;DR
• TPC-DS Query 27, Scale=200, 10 EC2 nodes (40 disks)
1501.895
1176.479
631.027
39.985
0
200
400
600
800
1000
1200
1400
1600
Query 27
Text
RCFile
Partitioned RCFile
Partitioned RCFile + Optimizations

6. © Hortonworks Inc. 2013
TL;DR - II
• TPC-DS Query 82, Scale=200, 10 EC2 nodes (40 disks)
3257.692
2862.669
255.641
71.114
0
500
1000
1500
2000
2500
3000
3500
Query 82
Text
RCFile
Partitioned RCFile
Partitioned RCFile + Optimizations

7. © Hortonworks Inc. 2013
Before

8. © Hortonworks Inc. 2013
After

9. Support for Analytics - OVER
Page 9
© 2013 Hortonworks, DO NOT SHARE, HORTONWORKS
CONFIDENTIAL
• OVER clause
– PARTITION BY, ORDER BY, ROWS BETWEEN/FOLLOWING/PRECEDING
– Will work with current aggregate functions
– Adding new aggregates/window functions we believe are useful
– Condor: RANK, LEAD, ROW_NUMBER, LAG, LEAD, FIRST_VALUE, LAST_VALUE
– Ducati: NTILE, DENSE_RANK, CUME_DIST, PERCENT_RANK, PERCENT_CONT,
PERCENT_DISC
– Ongoing work in the community, will be shepherding it
SELECT salesperson, AVG(salesprice) OVER
(PARTITION BY region, ORDER BY date
RANGE 10 ROWS PRECEEDING)
FROM sales;

10. Support for Analytics Continued
Page 10
© 2013 Hortonworks, DO NOT SHARE, HORTONWORKS
CONFIDENTIAL
• CUBE BY, ROLLUP in Hive 0.10
• Subqueries in WHERE
– Non-correlated only
– [NOT] IN supported
– Plan to optimize to fit in memory as hash table when feasible, join when not
• Datatypes
– datetime
– char() and varchar()
– add precision and scale to decimal and float
– aliases for standard SQL types (BLOB = binary, CLOB = string, integer = int, real/
number = decimal)

11. ORCFile – Optimized RCFile
Page 11
© 2013 Hortonworks, DO NOT SHARE, HORTONWORKS
CONFIDENTIAL

12. © Hortonworks Inc. 2012
Top Level
Page 12

13. © Hortonworks Inc. 2012
File Structure
Page 13

14. © Hortonworks Inc. 2012
Stripe Structure
Page 14

15. © Hortonworks Inc. 2012
File Layout
Page 15

16. © Hortonworks Inc. 2012
Integer Column Serialization
Page 16

17. © Hortonworks Inc. 2012
String Column Serialization
Page 17

18. © Hortonworks Inc. 2012
Compression
Page 18

19. © Hortonworks Inc. 2012
Projection and Predicate Filtering
Page 19

20. © Hortonworks Inc. 2012
Example File Sizes
Page 20

21. © Hortonworks Inc. 2012
Final notes
Page 21

22. © Hortonworks Inc. 2012
Comparison
Page 22
RC File Trevni ORC File
Hive Type Model N N Y
Separate complex columns N Y Y
Splits found quickly N Y Y
Default column group size 4MB 64MB* 250MB
Files per a bucket 1 > 1 1
Store min, max, sum, count N N Y
Versioned metadata N Y Y
Run length data encoding N N Y
Store strings in dictionary N N Y
Store row count N Y Y
Skip compressed blocks N N Y
Store internal indexes N N Y

23. © Hortonworks Inc. 2012
Page 23
Tez

24. © Hortonworks Inc. 2012: DO NOT SHARE. CONTAINS HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION
Tez (fka MRX)
• Low level data-processing execution engine
• Use it for the base of MapReduce, Hive, and Pig
• Enables pipelining of jobs
• Removes task and job launch times
• Hive and Pig jobs no longer need to move to the end
of the queue between steps in the pipeline
• Does not write intermediate output to HDFS
– Much lighter disk and network usage
• Built on YARN
Page 24

25. © Hortonworks Inc. 2012: DO NOT SHARE. CONTAINS HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION
Tez- Core Idea
Task with pluggable Input, Processor & Output
Page 25
YARN ApplicationMaster to run DAG of Tez Tasks
Input Processor
Tez Task
Output

26. © Hortonworks Inc. 2012: DO NOT SHARE. CONTAINS HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION
Tez – Blocks for building tasks
MapReduce ‘Map’
Page 26
MapReduce ‘Reduce’
HDFS
Input
Map
Processor
Tez Task
Sorted
Output
Shuffle
Input
Reduce
Processor
Tez Task
HDFS
Output
Intermediate ‘Reduce’
for
Map-Reduce-Reduce
Shuffle
Input
Reduce
Processor
Tez Task
Sorted
Output

27. © Hortonworks Inc. 2012: DO NOT SHARE. CONTAINS HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION
Tez – More tasks
Special Pig/Hive ‘Map’
Page 27
In-memory Map
HDFS
Input
Map
Processor
Tez Task
Pipeline
Sorter
Output
HDFSIn
put
Map
Processor
Tez Task
In-
memory
Sorted
Output
Special Pig/Hive
‘Reduce’
Shuffle
Skip-
merge
Input
Reduce
Processor
Tez Task
Sorted
Output

28. © Hortonworks Inc. 2012: DO NOT SHARE. CONTAINS HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION
Hive/MR versus Hive/Tez
Page 28
I/O Synchronization
Barrier
I/O Pipelining
Hive - MR Hive - Tez
SELECT a.state, COUNT(*)
FROM a JOIN b ON (a.id = b.id)
GROUP BY a.state

29. © Hortonworks Inc. 2012: DO NOT SHARE. CONTAINS HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION
Hive/MR versus Hive/Tez
Page 29
Hive - MR Hive - Tez
SELECT a.state, COUNT(*), AVERAGE(c.price)
FROM a
JOIN b ON (a.id = b.id)
JOIN c ON (a.itemId = c.itemId)
GROUP BY a.state

30. © Hortonworks Inc. 2012: DO NOT SHARE. CONTAINS HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION
FastQuery: Beyond Batch with YARN
Page 30
Tez Generalizes Map-Reduce
Simplified execution plans process
data more efficiently
Always-On Tez Service
Low latency processing for
all Hadoop data processing

31. © Hortonworks Inc. 2012: DO NOT SHARE. CONTAINS HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION
Tez Service
• Hive Query Startup Expensive
– Job launch & task-launch latencies are fatal for short queries (in
order of 5s to 30s)
• Solution
– Tez Service
– Removes task-launch overhead
– Removes job-launch overhead
– Hive
– Submit query-plan to Tez Service
– Native Hadoop service, not ad-hoc
Page 31

32. © Hortonworks Inc. 2012: DO NOT SHARE. CONTAINS HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION
More details
• Tez initiation
– MRX runtime for running existing MR applications
• Tez Improvements
– In-memory sort/shuffle
– Specialized sorts for Hive (integer sorts etc.)
– Specialized merges for Hive (shallow key-space merges)
• Tez on MRX
– Change Hive to use MR+ from Tez
– Focus on map-side joins
• Initiated Apache Incubator Project
Page 32

33. Hive Performance Longer Term -
Vectorization
Page 33
© 2013 Hortonworks, DO NOT SHARE, HORTONWORKS
CONFIDENTIAL
• Rewrite operators to work on arrays of Java scalars
• aka vectorization – ala MonetDB
• Operates on blocks of 1K or more records
• Each block contains an array of Java scalars, one for each column
• Avoids many function calls
• Size to fit in L1 cache, avoid cache misses
• Generate code for operators on the fly to avoid branches in code, maximize
deep pipelines of modern processers
• Will require complete rewrite of operators and execution pipeline
– Need to figure out how to integrate this into Hive, separate branch, separate layer
in another project, or integrate in existing pipeline
• Requires conversion of all column values to Java scalars – no objects allowed
– Integrates nicely with up coming ORC work, other input types will need conversion
on reading
• Want to write this in a way it can be shared by Pig, Cascading, MR
programmers

34. © Hortonworks Inc. 2013
CPU intensive code