How is it that one system can query terabytes of data, yet still provide interactive query support? This talk will discuss two of the underlying technologies that allow Apache Hive to support fast query response, both on-premise in HDFS and in cloud object stores such as S3 and WASB. LLAP was introduced in Hive 2.6. It provides standing processes that securely cache Hive’s columnar data and can do query processing without ever needing to start tasks in Hadoop. We will cover LLAP’s architecture, intended uses cases, and performance numbers for both on-premise and in the cloud. The second technology is the integration of Hive with Apache Druid. Druid excels at low-latency, interactive queries over streaming data. Its method of storing data makes it very well suited for OLAP style queries. We will cover how Hive can be integrated with Druid to support real-time streaming of data from Kafka and OLAP queries.

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved
Fast SQL for Big Data
Apache Hive and Apache Druid
Alan Gates
Hortonworks Co-founder, Apache Hive PMC member
@alanfgates

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
7000 analysts, 80ms average latency, 1PB data.
250k BI queries per hour
On demand deep reporting in the cloud over
100TB in minutes.

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved
• Ran all 99 TPCDS queries
• Total query runtime have improved multifold in each release!
Benchmark Journey
TPCDS 10TB scale on 10 node cluster
HDP 2.5
Hive1
HDP 2.5
LLAP
HDP 2.6
LLAP
25x 3x 2x
HDP 3.0
LLAP
2016 20182017
ACID
tables

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
Hive LLAP - MPP Performance at Hadoop Scale
Deep
Storage
Hadoop Cluster
LLAP Daemon
Query
Executors
LLAP Daemon
Query
Executors
LLAP Daemon
Query
Executors
LLAP Daemon
Query
Executors
Query
Coordinators
Coord-
inator
Coord-
inator
Coord-
inator
HiveServer2
(Query
Endpoint)
ODBC /
JDBC
SQL
Queries In-Memory Cache
(Shared Across All Users)
HDFS and
Compatible
S3 WASB Isilon

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved
Aggressive Caching

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved
Caching in LLAP
• Find grained (by row group and column) and compact (dictionary encoding, RLE)
• Important in environment with PBs of data but common queries only touch 100s of GB
• Prioritized – indexes cached with higher priority
• Off heap to avoid GC
• Supports spill to SSD – important in the cloud
• Uses LRFU replacement algorithm to avoid large scans purging the cache

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved
Query Result Cache
Returns results directly from storage (e.g.
HDFS) without actually executing the query
If the same query has run before and the
underlying data has not changed
Important for dashboards, reports etc.
where repetitive queries are common
Uses transactions to determine when
underlying data has changed
Without
cache
With
cache

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved
Metastore Cache
• With query execution time being < 1 sec, compilation time starts to dominate
• Metadata retrieval is often significant part of compilation time. Most of it is in RDBMS
queries.
• Cloud RDBMS As a Service is often slower, and frequent queries leads to throttling.
• Metadata cache speeds compilation time by around 50% with on prem MySQL.
Significantly more improvement with cloud RDBMS.
• Cache is consistent in single metastore setup, eventually consistent with HA setup.
Consistent HA setup support is in the works.

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved
New in Hive 3:
Materialized Views

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved
Possible Workflow
1. Create materialized view using Hive tables
• Stored by Hive or Druid
2. User or dashboard sends queries to Hive
• Hive rewrites queries using available materialized views
• Execute rewritten query
Dashboards, BI tools
CREATE MATERIALIZED VIEW `ssb_mv`
STORED AS 'org.apache.hadoop.hive.druid.DruidStorageHandler'
ENABLE REWRITE
AS
<query>;
DBA, recommendation system
①
②
Data
Queries

11. 11 © Hortonworks Inc. 2011–2018. All rights reserved
Materialized View-based Rewriting Example
• Materialized view definition
CREATE MATERIALIZED VIEW mv AS
SELECT <dims>,
lo_revenue,
lo_extendedprice * lo_discount AS d_price,
lo_revenue - lo_supplycost
FROM
customer, dates, lineorder, part, supplier
WHERE
lo_orderdate = d_datekey
and lo_partkey = p_partkey
and lo_suppkey = s_suppkey
and lo_custkey = c_custkey;
• Query
SELECT sum(lo_extendedprice * lo_discount)
FROM
lineorder, dates
WHERE
lo_orderdate = d_datekey
and d_year = 2013
and lo_discount between 1 and 3;
• Materialized view-based rewriting
SELECT SUM(d_price)
FROM mv
WHERE
d_year = 2013
and lo_discount between 1 and 3;
supplier
part
dates
customerlineorder
d_year lo_discount <dims> d_price
2013 2 ... 7.55
2014 4 ... 432.60
2013 2 ... 34.45
2012 2 ... 2.05
… … ... …
mv contents
sum
42.0
…
Query results

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved
Materialized View - Maintenance
• Partial table rewrites are supported
• Typical: Denormalize last month of data only
• Rewrite engine will produce union of latest and historical data
• Updates to base tables
• Invalidates views, but
• Can choose to allow stale views (max staleness) for performance
• Can partial match views and compute delta after updates
• Incremental updates
• Common classes of views allow for incremental updates
• Others need full refresh

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved
Optimizer Improvements

14. 14 © Hortonworks Inc. 2011–2018. All rights reserved
SELECT * FROM
( SELECT AVG(ss_list_price) B1_LP,
COUNT(ss_list_price) B1_CNT ,COUNT(DISTINCT
ss_list_price) B1_CNTD
FROM store_sales
WHERE ss_quantity BETWEEN 0 AND 5 AND
(ss_list_price BETWEEN 11 and 11+10 OR
ss_coupon_amt BETWEEN 460 and 460+1000 OR
ss_wholesale_cost BETWEEN 14 and 14+20)) B1,
( SELECT AVG(ss_list_price) B2_LP,
COUNT(ss_list_price) B2_CNT ,COUNT(DISTINCT
ss_list_price) B2_CNTD
FROM store_sales
WHERE ss_quantity BETWEEN 6 AND 10 AND
(ss_list_price BETWEEN 91 and 91+10 OR
ss_coupon_amt BETWEEN 1430 and 1430+1000 OR
ss_wholesale_cost BETWEEN 32 and 32+20)) B2,
. . .
LIMIT 100;
TPCDS SQL query 28 joins 6 instances of store_sales table
Shared Scan - 4x Improvement!
RS RS RS RS RS
Scan
store_sales
Combined OR’ed B1-B6 Filters
B1 Filter B2 Filter B3 Filter B4 Filter B5 Filter
Join

15. 15 © Hortonworks Inc. 2011–2018. All rights reserved
• Dramatically improves performance of very selective joins
• Builds a bloom filter from one side of join and filters rows from other side
• Skips scan and further evaluation of rows that would not qualify the join
Dynamic Semijoin Reduction - 7x Improvement for q72
SELECT …
FROM sales JOIN time ON
sales.time_id = time.time_id
WHERE time.year = 2014 AND
time.quarter IN ('Q1', 'Q2’)
Reduced scan on sales

16. 16 © Hortonworks Inc. 2011–2018. All rights reserved
Statistics (not new)
• Statistics collection can be set to automatic or manual
• Used extensively in join selection
• Without statistics much of the optimizer will not be used

17. 17 © Hortonworks Inc. 2011–2018. All rights reserved
• Memory exhaustion due to under
provisioning
• Excessive runtime (future)
• Excessive spilling (future)
• Query fails because of stats estimation
error
• Runtime sends observed statistics back
to coordinator
• Statistics overrides are created at
session, server or global level
• Query is replanned and resubmitted
Symptoms Solution
Optimizer Is Learning from Planning Mistakes

18. 18 © Hortonworks Inc. 2011–2018. All rights reserved
Apache Druid

19. 19 © Hortonworks Inc. 2011–2018. All rights reserved
Druid Capabilities
• Streaming ingestion capability
• Data Freshness – analyze events as they occur
• Fast response time (ideally < 1sec query time)
• Arbitrary slicing and dicing
• Multi-tenancy – 1000s of concurrent users
• Scalability and Availability
• Rich real-time visualization with Superset
Apache Druid is a distributed, real-time, column-oriented
datastore designed to quickly ingest and index large amounts
of data and make it available for real-time query.

20. 20 © Hortonworks Inc. 2011–2018. All rights reserved
Druid: Fast Facts
Most Events per Day
30 Billion Events / Day
(Metamarkets)
Most Computed Metrics
1 Billion Metrics / Min
(Jolata)
Largest Cluster
200 Nodes
(Metamarkets)
Largest Hourly Ingestion
2TB per Hour
(Netflix)

21. 21 © Hortonworks Inc. 2011–2018. All rights reserved
Hive and Druid, Better Together
Technology Strengths Issues
Hive SQL 2011, JDBC/ODBC
Fast scans
ACID
Not optimized for slice and dice and drill down
(OLAP cubing) operations
Druid Dimensional aggregates support OLAP cubes
Timeseries queries
Realtime ingestion of streaming data
Lacks SQL interface
No joins
Problem: You don't want two systems to manage and load data into
Solution: For data that fits best in Druid, load it in Druid and access it with Hive
• Hive supports push down of queries to Druid, optimizer knows what to push and what to run in Hive
• Enables SQL and JDBC/ODBC access to data in Druid
• Enables join of historical and realtime data
• Enables Hive support of slice & dice, drill down for OLAP cubing
• Can also create materialized views in Hive and store them in Druid

22. 22 © Hortonworks Inc. 2011–2018. All rights reserved
Druid Connector
Realtime Node
Realtime Node
Realtime Node
Broker HiveServer2
Instantly analyze kafka data with milliseconds latency

23. 23 © Hortonworks Inc. 2011–2018. All rights reserved
Druid Connector - Joins Between Hive and Realtime Data in Druid
Bloom filter pushdown greatly reduces data transfer
Send promotional email to all customers from CA who purchased more than $1000 worth of merchandise today.
create external table sales(`__time` timestamp, quantity int, sales_price double,customer_id bigint, item_id int, store_id int)
stored by 'org.apache.hadoop.hive.druid.DruidStorageHandler'
tblproperties ( "kafka.bootstrap.servers" = "localhost:9092", "kafka.topic" = "sales-topic",
"druid.kafka.ingestion.maxRowsInMemory" = "5");
create table customers (customer_id bigint, first_name string, last_name string, email string, state string);
select email from customers join sales using customer_id where to_date(sales.__time) = date ‘2018-09-06’
and quantity * sales_price > 1000 and customers.state = ‘CA’;

24. 24 © Hortonworks Inc. 2011–2018. All rights reserved
Tips for Optimizing Hive

25. 25 © Hortonworks Inc. 2011–2018. All rights reserved
Making Your Queries Blaze in Hive 3
• Use a columnar format
• We recommend ORC; ORC or Parquet much better for DW queries than row oriented formats
• Use the right tool for the right job, all in Hive
• LLAP for BI queries
• Tez for ETL/batch
• Druid for ROLAP and realtime ingestion
• Do not use MapReduce as your Hive engine, it is very slow
• Keep statistics current on your data
• Define materialized views for common joins and aggregations
• Turn on ACID – it enables query cache and materialized view partial rewrites

26. 26 © Hortonworks Inc. 2011–2018. All rights reserved
SOLUTIONS: Heuristic recommendation engine
Fully self-serviced query and storage optimization

27. 27 © Hortonworks Inc. 2011–2018. All rights reserved
Questions?