在資料滿滿的年代,你/妳是否常常被要求要設計出能夠實時的收集、串流式的處理、及時的展現與秒級反應的巨量資料分析系統呢? 你/妳是否曾經陷落在機場出関檢查的長長人龍陣中, 看著別人優雅地快速穿過“快速通関”時眼神飄過來的那種“你的不舒服是我的那種舒暢”。接續著2017在DataCon 的 Streaming enrichment 之後, 我們接著介詔與現場展示 Druid 加 Superset 這兩個神兵利器幫大家開通的”資料的快速通道”。我們將以最簡單的架構來展現 Druid 為何如此快速的關鍵點，並利用 Superset 現場拉 UI 來看資料與做資料的視覺化。想知道如何建立屬於自己的“資料的快速通道”嗎？想知道如何處理 TB 的等級的資料量的同時又能讓使用者享受暢快的 Dashboard 嗎？

1. Druid + Superset
( )
@
2018 – Q2

3. • 5 /
• 3 MES/SFC
• 3 ERP (SAP-MM )/EAI Leader
• 2 End-To-End Business Integration Analyst
• 1 OEM Sales Team Leader
• 4 /
• 5 /
• IT (witlab) –
• 226 Finisher

5. Source:http://www.dreamwerkztechnologies.com/dwz-inventory-management/ Source: https://logicyyc.com/2017/04/27/industry-4-0/

6. Data Collection
and Storage
Data
Processing
Event
Processing
Data
Analysis
(Latency)
(Throughput)
(Cost)

7. 10
25
18
!

8. 10
25
18

9. 10
25
18
!

10. “Lamda “

11. Lamda Batch Layer
)

12. •
•
•

15. (
•
•
• /
•
• )
• ,
•
•
•
• )
• /

17. •
•
•
•
•
•
•
•
•

18. Lamda Speed Layer
Streaming)

19. –
Kafka Indexing
Service

20. O
• r L -
T
• P
u - D
• T A /
B
• Br d Br
• B i /

21. • Critical for interactive user experience
• Avg query times ~500ms
• 90% percentile under 1 sec
• 99% percentile under 10 sec
• Handle 1000’s of concurrent queries

25. R S - -
• - - a rs
• i / e - c
• tP rs l s
• b - / / / / p &A L
• D / -u
• p /,bd L h /, / - ,/ - o
hnz

27. Shipment

28. Shipment Raw
(Dimensions) (Measures)(Event time)

29. – Kafka Databus
Kafka Indexing
Service
1

30. Shipment
• , : S
• J ( (
• J ) -23 415
• 8 81 680

31. – Shipment ingestion
Kafka Druid
Kafka Indexing
Service
2
2
2

32. Druid kafka index service ingestion
Kafka Druid

33. (Event time) Hour” (Sum / Count / Max /
Min)
Druid - (Pre-Aggregation)

34. Druid (Pre-Aggregation)
• SQL
• select sum("raw_count"),count(*) from shipment
• No Pre-agg : 34,214,400 , 34,214,400 => 433.22 MB
• Pre-agg by minute : 34,214,400 , 23,126,204 => 243.97 MB
• Pre-agg by hour : 34,214,400 , 383,088 => 8.03 MB
• Pre-agg by day : 34,214,400 , 33,384 => 0.65 MB

35. – Shipment ingestion
Kafka Druid
Kafka Indexing
Service
3

39. c
•
• t 9
• A i
~
• r k
• ~~ e
• 9s M ) / ( 3 , s ) / 3
• 9s7 7 G ) / ( 3 ,
• G 9 ms: 0
• 9 9 ) / 3p
• G r an
• ar a

40. &
• )
(

41. ) (

42. ) (

43. ) (

44. ) (

45. ) (

46. ) (

47. ) (

48. ?

49. Source: https://devilsadvocatepaper.com/2016/04/06/the-end/