1.
Large-scale, interactive
ad-hoc queries over
different datastores with
Apache Drill
Michael Hausenblas, Chief Data Engineer, MapR Technologies
JAX London, 2013-10-29

2.
http://www.flickr.com/photos/kevinomara/2866648330/ licensed under CC BY-NC-ND 2.0
Which
workloads do
you
encounter in
your
environment?

3.
Batch processing
Apache Pig
Cascalog
… for recurring tasks such as large-scale data mining, ETL
offloading/data-warehousing  for the batch layer in Lambda
architecture

4.
OLTP
… user-facing eCommerce transactions, real-time messaging at
scale (FB), time-series processing, etc.  for the serving layer in
Lambda architecture

5.
Stream processing
… in order to handle stream sources such as social media feeds
or sensor data (mobile phones, RFID, weather stations, etc.) 
for the speed layer in Lambda architecture

6.
Search/Information Retrieval
… retrieval of items from unstructured documents (plain
text, etc.), semi-structured data formats (JSON, etc.), as
well as data stores (MongoDB, CouchDB, etc.)

7.
But what about
interactive
ad-hoc query
at scale?
http://www.flickr.com/photos/9479603@N02/4144121838/ licensed under CC BY-NC-ND 2.0

8.
Interactive Query (?)
Impala
low-latency

9.
Use Case: Marketing Campaign
• Jane, a marketing analyst
• Determine target segments
• Data from different sources

10.
Use Case: Logistics
• Supplier tracking and performance
• Queries
– Shipments from supplier ‘ACM’ in last 24h
– Shipments in region ‘US’ not from ‘ACM’
SUPPLIER_ID
NAME
REGION
ACM
ACME Corp
US
GAL
GotALot Inc
US
BAP
Bits and Pieces Ltd
Europe
ZUP
Zu Pli
Asia
{
"shipment": 100123,
"supplier": "ACM",
“timestamp": "2013-02-01",
"description": ”first delivery today”
},
{
"shipment": 100124,
"supplier": "BAP",
"timestamp": "2013-02-02",
"description": "hope you enjoy it”
}
…

11.
Use Case: Crime Detection
•
•
•
•
Online purchases
Fraud, bilking, etc.
Batch-generated overview
Modes
– Explorative
– Alerts

12.
Requirements
•
•
•
•
•
Support for different data sources
Support for different query interfaces
Low-latency/real-time
Ad-hoc queries
Scalable, reliable

13.
And now for something completely different …

14.
Google’s Dremel
“
Dremel is a scalable, interactive ad-hoc
query system for analysis of read-only
nested data. By combining multi-level
execution trees and columnar data layout,
it is capable of running aggregation
queries over trillion-row tables in
seconds. The system scales to thousands of
CPUs and petabytes of data, and has
thousands of users at Google.
…
“
http://research.google.com/pubs/pub36632.html
Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar, Matt
Tolton, Theo Vassilakis, Proc. of the 36th Int'l Conf on Very Large Data Bases (2010), pp. 330339

15.
Google’s Dremel
multi-level execution trees
columnar data layout

16.
Google’s Dremel
nested data + schema
column-striped representation
map nested data to tables

17.
Google’s Dremel
experiments:
datasets & query performance

18.
Back to Apache Drill …

19.
Apache Drill–key facts
•
•
•
•
•
•
Inspired by Google’s Dremel
Standard SQL 2003 support
Plug-able data sources
Nested data is a first-class citizen
Schema is optional
Community driven, open, 100’s involved

20.
High-level Architecture

21.
Principled Query Execution
• Source query—what we want to do (analyst
friendly)
• Logical Plan— what we want to do (language
agnostic, computer friendly)
• Physical Plan—how we want to do it (the best
way we can tell)
• Execution Plan—where we want to do it

22.
Principled Query Execution
Source
Query
SQL 2003
DrQL
MongoQL
DSL
Parser
parser API
Logical
Plan
query: [
{
@id: "log",
op: "sequence",
do: [
{
op: "scan",
source: “logs”
},
{
op: "filter",
condition:
"x > 3”
},
Optimizer
Topology
CF
etc.
Physical
Plan
Execution
scanner API

23.
Wire-level Architecture
• Each node: Drillbit - maximize data locality
• Co-ordination, query planning, execution, etc, are distributed
• Any node can act as endpoint for a query—foreman
Drillbit
Drillbit
Drillbit
Drillbit
Storage
Process
Storage
Process
Storage
Process
Storage
Process
node
node
node
node

24.
Wire-level Architecture
• Curator/Zookeeper for ephemeral cluster membership info
• Distributed cache (Hazelcast) for metadata, locality
information, etc.
Curator/Zk
Drillbit
Drillbit
Drillbit
Drillbit
Distributed Cache
Distributed Cache
Distributed Cache
Distributed Cache
Storage
Process
Storage
Process
Storage
Process
Storage
Process
node
node
node
node

25.
Wire-level Architecture
• Originating Drillbit acts as foreman: manages query execution,
scheduling, locality information, etc.
• Streaming data communication avoiding SerDe
Curator/Zk
Drillbit
Drillbit
Drillbit
Drillbit
Distributed Cache
Distributed Cache
Distributed Cache
Distributed Cache
Storage
Process
Storage
Process
Storage
Process
Storage
Process
node
node
node
node

26.
Wire-level Architecture
Foreman turns into
root of the multi-level
execution tree, leafs
activate their storage
engine interface.
node
Curator/Zk
node
node

27.
On the shoulders of giants …
•
•
•
•
•
•
•
•
•
•
•
•
•
Jackson for JSON SerDe for metadata
Typesafe HOCON for configuration and module management
Netty4 as core RPC engine, protobuf for communication
Vanilla Java, Larray and Netty ByteBuf for off-heap large data structures
Hazelcast for distributed cache
Netflix Curator on top of Zookeeper for service registry
Optiq for SQL parsing and cost optimization
Parquet (http://parquet.io)/ ORC
Janino for expression compilation
ASM for ByteCode manipulation
Yammer Metrics for metrics
Guava extensively
Carrot HPC for primitive collections

28.
Key features
•
•
•
•
Full SQL – ANSI SQL 2003
Nested Data as first class citizen
Optional Schema
Extensibility Points …

29.
Extensibility Points
•
•
•
•
Source query  parser API
Custom operators, UDF  logical plan
Serving tree, CF, topology  physical plan/optimizer
Data sources &formats  scanner API
Source
Query
Parser
Logical
Plan
Optimizer
Physical
Plan
Execution

30.
User Interfaces
• API—DrillClient
– Encapsulates endpoint discovery
– Supports logical and physical plan submission,
query cancellation, query status
– Supports streaming return results
• JDBC driver, converting JDBC into DrillClient
communication.
• REST proxy for DrillClient

31.
User Interfaces

32.
LET’S GET OUR HANDS DIRTY…

33.
Demo
• Install
• Preparation
$ wget http://people.apache.org/~jacques/apache-drill-1.0.0m1.rc3/apache-drill-1.0.0-m1-binary-release.tar.gz
$ tar -zxf apache-drill-1.0.0-m1-binary-release.tar.gz
$ export
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_11.jdk/Contents/Ho
me
$ export DRILL_LOG_DIR=$PWD
$ ./bin/drillbit.sh start

34.
Demo: submitting physical plan in a 3-node cluster
Test 1: Scan JSON doc
$ bin/submit_plan -f sample-data/physical_json_scan_test1.json -t physical -zk 127.0.0.1:2181
Test 2: Scan Parquet doc
$ bin/submit_plan -f sample-data/parquet_scan_union_screen_physical.json -t physical -zk 127.0.0.1:2181

35.
Demo: SQL on single node
$ ./bin/sqlline -u jdbc:drill:schema=parquet-local
0: jdbc:drill:schema=parquet-local> SELECT _MAP['N_REGIONKEY'] as regionKey, _MAP['N_NAME'] as name
FROM "sample-data/nation.parquet" WHERE cast(_MAP['N_NAME'] as varchar) < 'M';

36.
Demo: DIY
https://github.com/mhausenblas/apache-drill-sandbox/

37.
Useful Resources
• Getting Started guide
https://github.com/vrtx/incubatordrill/blob/getting_started/docs/getting_started.rst
• Demo HowTo
https://cwiki.apache.org/confluence/display/DRILL/De
mo+HowTo
• How to build/install Apache Drill on Ubuntu 13.04
http://www.confusedcoders.com/bigdata/apachedrill/how-to-build-apache-drill-on-ubuntu-13-04

38.
BE A PART OF IT!

39.
Status
• Heavy development by multiple organizations
(MapR, Pentaho, Microsoft, Thoughtworks,
XingCloud, etc.)
• Currently more than 100k LOC
• M1 Alpha available via
http://www.apache.org/dyn/closer.cgi/incubator/drill/drill-1.0.0-m1-incubating/

40.
Kudos to …
•
•
•
•
•
•
•
Julian Hyde, Pentaho
Lisen Mu, XingCloud
Tim Chen, Microsoft
Chris Merrick, RJMetrics
David Alves, UT Austin
Sree Vaadi, SSS
Srihari Srinivasan,
ThoughtWorks
• Alexandre Beche, CERN
• Jason Altekruse, MapR
•
•
•
•
•
•
•
•
•
Ben Becker, MapR
Jacques Nadeau, MapR
Ted Dunning, MapR
Keys Botzum, MapR
Jason Frantz
Ellen Friedman
Chris Wensel, Concurrent
Gera Shegalov, Oracle
Ryan Rawson, Ohm Data
http://incubator.apache.org/drill/team.html

41.
Contributing
Contributions appreciated—not only code drops …
• Test data & test queries
• Use case scenarios (textual/SQL queries)
• Documentation

42.
Engage!
• Follow @ApacheDrill on Twitter
• Sign up at mailing lists (user | dev)
http://incubator.apache.org/drill/mailing-lists.html
• Standing G+ hangouts every Tuesday at 5pm GMT
http://j.mp/apache-drill-hangouts
• Keep an eye on http://drill-user.org/