This slide for 2nd druid meetup@Seoul. http://www.meetup.com/Druid-Seoul/events/233026535/

1. DRUID REAL WORLD
DRUID MEETUP@SEOUL
YOU SUN JEONG (JERRYJUNG@SK.COM)

2. DRUID MEETUP@SEOUL
WELCOME TO DRUID WORLD
THE JOURNEY OF 9 MONTHS

3. DRUID MEETUP@SEOUL
DRUID OVERVIEW
REALTIME
BROKER HISTORICAL

4. DRUID MEETUP@SEOUL
ARCHITECTURE - BATCH INGESTION
HDFS
HISTORICAL
NODE
HISTORICAL
NODE
HISTORICAL
NODE
BROKER
NODE
Segments
Queries

5. DRUID MEETUP@SEOUL
ARCHITECTURE - STREAMING INGESTION
REALTIME
NODE
HISTORICAL
NODE
HISTORICAL
NODE
HISTORICAL
NODE
BROKER
NODE
Segments
Queries
Streaming

6. DRUID MEETUP@SEOUL
ARCHITECTURE - LAMBDA
REALTIME
NODE
HISTORICAL
NODE
HISTORICAL
NODE
HISTORICAL
NODE
BROKER
NODE
Segments
Queries
Streaming
HDFS

7. DRUID MEETUP@SEOUL
REAL WORLD IS CRUEL

8. DRUID MEETUP@SEOUL
PROBLEMS
▸ For Data Scientist
Arbitrary and Interactive
exploration of time series data
▸ Scalability and Performance
Ad-hoc query on trillions of events
▸ Characteristics of the data
Dynamic Columns
Numeric data with Array Type

9. DRUID MEETUP@SEOUL
DATA LAKE
http://www.kdnuggets.com/2015/09/data-lake-vs-data-warehouse-key-differences.html

10. DRUID MEETUP@SEOUL
STAT FUNCTION FOR DATA SCIENTIST
▸ HISTOGRAM
(MEDIAN)
▸ MEAN
▸ STDDEV
▸ RANGE
▸ AREA
▸ MIN/MAX/SUM

11. DRUID MEETUP@SEOUL
DATA EXPLODING
// column size 100 ~ 1000
KEYS = [“a”,”b”,”c”……]
VALUES = [3.25,45.443,103.2…..]
“a” = 3.25
“b” = 45.443
“c” = 103.2
.
.
.

12. DRUID MEETUP@SEOUL
REAL WORLD ARCHITECTURE
DATA
NODE #1
DATA
NODE #70
OVERLORD
MIDDLE
MANAGE
#1
COORDI
NATOR
MYSQL
HA
PROXY
MEMCACHED
#2
BROKER
NODE
#1
BROKER
NODE
#1
MEMCACHED
#3
MEMCACHED
#1
HISTORICAL
NODE #1
HISTORICAL
NODE #70
MIDDLE
MANAGE
#50
ZK1
ZK2
ZK3

13. DRUID MEETUP@SEOUL
BUT, WE DID IT
ORC INGESTION
VIRTUAL COLUMNS
JDBC FIREHOSE
STATS-EXTENTIONS
OPTIMIZATION
QUERY

14. DRUID MEETUP@SEOUL
NO PAIN, NO GAIN

15. DRUID MEETUP@SEOUL
MAY THE FORCE BE WITH YOU

16. Q&A
THANK YOU
DRUID MEETUP@SEOUL