This document provides an overview of Cloudera's SQL-on-Hadoop technologies and how to choose the right tool for different jobs. It discusses Hive for batch processing, Impala for interactive SQL and analytics, and SparkSQL for machine learning applications. The document also summarizes performance benchmarks that show Impala outperforming other SQL-on-Hadoop engines in terms of throughput, latency, and scalability. Finally, it briefly outlines new features in Cloudera 5.5 including improvements to Impala, Hive, SparkSQL, and the introduction of Kudu and RecordService.

1. 1© Cloudera, Inc. All rights reserved.
Choosing the Right Tool for the
Right Job
Overview of Cloudera’s SQL-on-Hadoop Technologies

2. 2© Cloudera, Inc. All rights reserved.
§ The information in this document is proprietary to Cloudera. No part of this document may be reproduced, copied or transmitted in any form for
any purpose without the express prior written permission of Cloudera.
§ This document is a preliminary version and not subject to your license agreement or any other agreement with Cloudera. This document contains
only intended strategies, developments and functionalities of Cloudera products and is not intended to be binding upon Cloudera to any particular
course of business, product strategy and/or development. Please note that this document is subject to change and may be changed by Cloudera at
any time without notice.
§ Cloudera assumes no responsibility for errors or omissions in this document. Cloudera does not warrant the accuracy or completeness of the
information, text, graphics, links or other items contained within this material. This document is provided without a warranty of any kind, either
express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose or non-infringement.
§ Cloudera shall have no liability for damages of any kind including without limitation direct, special, indirect or consequentialdamages that may
result from the use of these materials. The limitation shall not apply in cases of gross negligence.

3. 3© Cloudera, Inc. All rights reserved.
Cloudera is Built for Production Success
Hadoop delivers:
• One place for unlimited data
• Unified, multi-framework data access
Cloudera delivers:
• Leading Performance
• Enterprise Security
• Data Management
• Simple Administration
Security and Administration
Unlimited Storage
Process Discover Model Serve
Deployment
Flexibility
On-Premises
Appliances
Engineered Systems
Public Cloud
Private Cloud
Hybrid Cloud
A modern data platform plus what the enterprise requires.

4. 4© Cloudera, Inc. All rights reserved.
One Platform, Many Workloads
Batch, Interactive,
and Real-Time.
Leading performance and
usability in one platform.
• End-to-end analytic workflows
• Access more data
• Work with data in new ways
• Enable new users
Security and Administration
Process
Ingest
Sqoop, Flume,
Kafka
Transform
MapReduce,
Hive, Pig, Spark
Discover
Analytic Database
Impala
Search
Solr
Model
Machine Learning
SAS, R, Spark,
Mahout
Serve
NoSQL Database
HBase
Streaming
Spark Streaming
Unlimited Storage HDFS, HBase
YARN, Cloudera Manager,
Cloudera Navigator

5. 5© Cloudera, Inc. All rights reserved.
Choosing the Right SQL Engine
Know Your Audience, Know Your Use Case
Batch
Processing
BI and
SQL Analytics
Procedural
Development
SQLOR
Impala

6. 6© Cloudera, Inc. All rights reserved.
Hive
Batch Processing
• User:
• SQL-based ETL developers
• Designed for:
• Handful of concurrent, very long-
running batch jobs
• Strengths:
• Custom file formats
• Very long-running ETL, data preparation,
or batch processing
• Massive ETL sorts with joins
• Existing Hive jobs

7. 7© Cloudera, Inc. All rights reserved.
Impala
BI and Analytics
User:
• Data Analysts
• BI Users
Designed for:
• Interactive SQL for large number of BI
users and analysts
Strengths:
• Multi-user scale
• Interactive latency
• Compatibility (BI tools, ANSI SQL, and
vendor-specific SQL)
• Usability

8. 8© Cloudera, Inc. All rights reserved.
SparkSQL
Machine Learning Applications
User:
• Data Engineers
• Data Scientists
Designed for:
• Ease of development for Spark
developers
• Handful of concurrent Spark jobs
Strengths:
• Ease of embedding SQL into Java or Scala
applications
• SQL for common functionality in
developer flow (eg. aggregations, filters,
samples)

9. 9© Cloudera, Inc. All rights reserved.
SQL-on-Hadoop Benchmark
Impala, Presto, Stinger, SparkSQL
Versions:
• Impala 1.4.0
• Presto 0.74
• Stinger Phase 3 (Final) => Hive 0.13.0
• SparkSQL 1.1
• Benchmark Details
• Based on industry standards (TPC)
• Repeatable (https://github.com/cloudera/impala-tpcds-kit)
• Methodical testing with multiple runs on
same hardware
• Help competing software do well
• SQL-92 join style for engines without CBO
• JVM tuning for Presto
• Run on optimal file formats for each
Full Details: http://blog.cloudera.com/blog/2014/09/new-benchmarks-for-sql-on-hadoop-impala-1-4-widens-the-
performance-gap/

10. 10© Cloudera, Inc. All rights reserved.
Impala Multi-User Performance Over 10x Faster
with Just 10 Users
0
50
100
150
200
250
300
350
Impala Spark SQL Presto Hive-on-Tez
Time (in seconds)
Single User vs 10 User Response Time/Impala
Times Faster
(Lower bars = better)
Single User, 5
10 Users, 11
Single User, 25
10 Users, 120
10 Users, 302
10 Users, 202
Single User, 37
Single User, 77
5.0x
10.6x
7.4x
27.4x
15.4x
18.3x

11. 11© Cloudera, Inc. All rights reserved.
Impala Enables Over 8.7x Throughput
More Work Done in Less Time
2333
266
106
175
0
500
1000
1500
2000
2500
Impala Spark SQL Presto Hive-on-Tez
Queries per Hour
Query Throughput/Impala Throughput Times More Than
(Higher bars = better)
8.7x 22.0x 13.3x

12. 12© Cloudera, Inc. All rights reserved.
Performance Benchmark Takeaways
• Impala unlocks BI usage directly on Hadoop
• Meets BI low-latency and multi-user requirements
• Advantage expands from 5x for single-user to >10x with just 10 users
• Hive is designed (and still great) for batch processing
• Most Impala customers use Hive for data preparation
• Hive is the most commonly used ETL framework
• Spark SQL enables easier Spark application development
• Enables mixed procedural Spark (Java/Scala) and SQL job development
• Mid-term trends will further favor Impala’s design approach for latency and concurrency
• More data sets move to memory (HDFS caching, in-memory joins, Intel joint roadmap)
• CPU efficiency will increase in importance
• Native code enables easy optimizations for CPU instruction sets
• Intel joint roadmap support these opportunities

13. 13© Cloudera, Inc. All rights reserved.
IBM Research Validation
• VLDB academic paper compares Impala and Hive (both MR and Tez) for SQL-on-Hadoop
• http://www.vldb.org/pvldb/vol7/p1295-floratou.pdf
• Impala’s significantly more efficient than Hive/Tez or Hive/MR
• Impala’s lead due to CPU efficiency, I/O manager, and overall
architecture that resembles a shared-nothing parallel database
• Parquet more efficient than ORC
• Additional Notes:
• Impala 1.4 and higher is significantly faster on selective joins than Impala 1.2.2 used in the paper
• Impala 2.0 has disk-based joins and aggregations
• Paper compares single-user only. Multi-user would perform even better
“Impala’s database-like architecture
provides significant performance gains,
compared to Hive’s MapReduce or Tez-
based runtime”
“The Parquet format skips data more efficiently
than ORC, which tends to pre-fetch
unnecessary data, especially when a table
contains a large number of columns”

14. 14© Cloudera, Inc. All rights reserved.
Major new SQL features in Cloudera 5.5
• Impala
• Reliability (particularly with concurrency and scale)
• Nested types
• Column-level security
• Additional functions
• Hive
• Quality
• S3 support
• CM monitoring
• Navigator lineage
• SparkSQL (with DataFrames)
• Now supported in CDH 5.5 (recommend HiveContext)
• Thriftserver and JDBC not ready for support
• Navigator Optimizer (beta)
• Helps assess and offload workloads onto Hadoop

15. 15© Cloudera, Inc. All rights reserved. 15
Kudu Fills a Critical Gap: Fast Analytics on Fast Changing Data
Fast Scans, Analytics
and Processing of
Stored Data
Fast On-Line
Updates &
Data Serving
Unchanging
Fast Changing
Frequent Updates
HDFS
HBase
Arbitrary Storage
(Active Archive)
Append-Only
Fast Analytics
(on fast-changing or
frequently-updated data)
Real-Time
Kudu
Kudu fills the Gap
Modern analytic
applications often
require complex data
flow & difficult
integration work to
move data between
HBase & HDFS
Analytic
Gap
Pace of Analysis
Pace of Data

16. 16© Cloudera, Inc. All rights reserved.
Current Security Architecture: Inconsistency = Limited
Access
Policy B
Impala
(column-level)
Policy A
Impala
...than others.
Some engines support
more granular restrictions...
Unified, Granular
Policy Enforcement
Challenge: Hadoop access engines respect policies differently, forcing reliance on lowest common denominator file- or
table-based policies, or restricted access. Policy management only solves part of the problem.
Solution: RecordService is a new high-performance security layer that centrally enforces access control policy.
Complementing Apache Sentry, which provides unified policy definition, it delivers unified row- and column-based
security, and dynamic data masking, to every Hadoop access path.
Benefits:
● Security: Fine-grained permissions and enforcement across Hadoop, building on Sentry.
● Interoperability: Developers don’t need to be aware of on-disk formats; transparently swap components.
RecordService: Unified Access Control Enforcement
Spark
(table-level)
RecordService
(policy enforcement)
Spark
Sentry
(policy definition)
Sentry
(policy definition)
...

17. 17© Cloudera, Inc. All rights reserved.
Thank You