Apache Druid basics and history. Where it fits. What it's best for. How is it different from other similar tech.

1. Apache Druid
The rise of operational analytic data stores
Rommel Garcia
rommel.garcia@imply.io

2. Who am I?
Rommel Garcia
Director, Solutions Engineering at
Author: Virtualizing Hadoop
11 years working on distributed, scalable systems
2

3. Agenda
● How Druid was born
● Druid Architecture
● Use Cases
● Data Modeling
● The Future
3

4. The Genesis
4

5. The Former Team @ Metamarkets
5
Vadim Ogievetsky Gian Merlino Fangjin Yang

6. Their Challenges
● Scale: when data is large, we need a lot of servers
● Speed: aiming for sub-second response time
● Complexity: too much fine grain to precompute
● High dimensionality: 10s or 100s of dimensions
● Concurrency: many users and tenants
● Freshness: load from streams
6

7. What They Tried
● Relational
● NoSQL
● Hadoop
7

8. Architecture
8

9. Key features
● Low latency ingestion from Kafka
● Bulk load from Hadoop
● Can pre-aggregate data during ingestion
● “Schema light”
● Ad-hoc queries
● Exact and approximate algorithms
● Can keep a lot of history (years are ok)
9

10. Druid: Decoupled, Distributed, Redundant
10

11. Druid: Ingest/Query Mechanism
11
Indexer
Indexer
Indexer
Files
Historical
Historical
Historical
Streams
Segments
Broker Broker
Queries

12. Segments
● Fundamental storage unit in Druid
● Immutable once created
● No contention between reads and writes
● One thread scans one segment
12

13. Segments & Time
13

14. Powered by Druid
14
Source: http://druid.io/druid-powered.html

15. What is Druid?
● “high performance”: low query latency, high ingest rates
● “analytics”: counting, ranking, groupBy, time trend
● “data store”: the cluster stores a copy of your data
● “event-driven data”: fact data like clickstream, network flows,
user behavior, digital marketing, server metrics, IoT
15

16. New class of data store
● Column oriented
● High concurrency
● Scalable to 100s of servers, millions of messages/sec
● Partition key for query pruning
● May or may not have secondary indexes
● Query through SQL
● Rapid queries on denormalized data
16

17. New class of data store
● “Operational analytics” or “big OLAP” data stores
● Examples
○ Apache Druid [incubating] (open source community)
○ Scuba (from Facebook)
○ Pinot (from LinkedIn)
○ Doris, formerly Palo (from Baidu)
○ ClickHouse (from Yandex)
17

18. Use Cases
18

19. Use cases
● Clickstreams, user behavior
● Application performance management
● Network flows
● IoT
● Digital marketing
● OLAP / business intelligence
19

20. Data Modeling
20

21. Optimized For A Reason
● denormalized
● roll up or to-not-roll-up
● Query time vs ingest time aggregation
● no joins
● lookups for slowly changing dimensions
21

22. The Future
22

23. Druid roadmap
● Parallel loading of data files without Hadoop
● Automatic compaction
● Smaller, faster compression (FastPFOR, etc)
● Subtotals, SQL “grouping sets”
● SQL standard null handling
● Vectorized query engine
● Garbage-free expression engine
● More to come!!
23

24. Try this at home
24

25. Download
Druid community site (current): http://druid.io/
Druid community site (new): https://druid.apache.org/
Imply distribution: https://imply.io/get-started
25

26. Contribute
26
https://github.com/apache/incubator-druid

27. Stay in touch
27
@druidio
http://druid.io/community