Presentation to the Apache Drill Meetup in Sunnyvale, CA on 2012/9/13. Framing the debate about Drill's goals in terms of a "typical" modern DBMS architecture; and also introducing the Optiq extensible query optimizer.

1. Why is data independence
(still) so important?
Julian Hyde @julianhyde
http://github.com/julianhyde/optiq
http://github.com/julianhyde/optiq-splunk
Apache Drill Meeting
2012/9/13

2. Data independence
This is my opinion about data management systems in general. I don't
claim that it is the right answer for Apache Drill.
I claim that a logical/physical separation can make a data management
system more widely applicable, therefore more widely adopted,
therefore better.
What “data independence” means in today's “big data” world.

3. About me
Julian Hyde
Database hacker (Oracle, Broadbase, SQLstream, LucidDB)
Open source hacker (Mondrian, olap4j, LucidDB, Optiq)
@julianhyde
http://github.com/julianhyde

4. http://www.flickr.com/photos/torkildr/3462606643

5. http://www.flickr.com/photos/sylvar/31436961/

6. “Big Data”
Right data, right time
Diverse data sources / Performance / Suitable format
Volume / Velocity / Variety
Volume – solved :)
Velocity – not one of Drill's goals (?)
Variety – ?

7. Variety
Variety of source formats (csv, avro, json, weblogs)
Variety of storage structures (indexes, projections, sort
order, materialized views) now or in future
Variety of query languages (DrQL, SQL)
Combine with other data (join, union)
Embed within other systems, e.g. Hive
Source for other systems, e.g. Drill | Cascading > Teradata
Tools generate SQL

8. Use case: Optiq* at Splunk
SQL interface on NoSQL system
“Smart” JDBC driver – pushes processing down to Splunk
* Truth in advertising: I am the author of Optiq.

9. Expression tree SELECT p.“product_name”, COUNT(*) AS c
FROM “splunk”.”splunk” AS s
JOIN “mysql”.”products” AS p
ON s.”product_id” = p.”product_id”
WHERE s.“action” = 'purchase'
GROUP BY p.”product_name”
Splunk ORDER BY c DESC
Table: splunk
Key: product_name
Key: product_id Agg: count
Condition: Key: c DESC
action =
'purchase'
scan
join
MySQL filter group sort
scan
Table: products

10. Expression tree SELECT p.“product_name”, COUNT(*) AS c
FROM “splunk”.”splunk” AS s
(optimized) JOIN “mysql”.”products” AS p
ON s.”product_id” = p.”product_id”
WHERE s.“action” = 'purchase'
GROUP BY p.”product_name”
Splunk ORDER BY c DESC
Condition:
Table: splunk action =
'purchase' Key: product_name
Agg: count
Key: c DESC
Key: product_id
scan filter
MySQL
join group sort
scan
Table: products

11. Conventional DBMS architecture
JDBC client
JDBC server
SQL parser /
validator Metadata
Query
optimizer
Data-flow
operators
Data Data

12. Drill architecture
DrQL client
DrQL parser /
validator
?
Metadata
Data-flow
operators
Data Data

13. Optiq architecture
JDBC client
JDBC server
Optional SQL parser / Metadata
validator SPI
Core Query Pluggable
optimizer rules
3rd 3rd
Pluggable party party
ops ops
3rd party 3rd party
data data

14. Analogy: Compiler architecture
front end C++ C Fortran
middle end Optimizations
back end x86 ARM Fortran

15. Conclusions
Clear logical / physical separation allows a data
management system to handle a wider variety of data,
query languages, and packaging.
Also provides a clear interface between the sub-teams
working on query language and operators.
A query optimizer allows new operators, and alternative
algorithms and data structures, to be easily added to
the system.

16. Extra material follows...

17. Writing an adapter
Driver – if you want a vanity URL like “jdbc:drill:”
Schema – describes what tables exist
Table – what are the columns, and how to get the data.
Operators (optional) – non-relational operators, if any
Rules (optional, but recommended) – improve efficiency by changing the
question
Parser (optional) – additional source languages