Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.

1. Ananth Packkildurai
November 28, 2017
1
Measuring Slack API performance using Druid

2. Public launch: 2014 800+ employees across
7 countries worldwide
HQ in San Francisco
Diverse set of industries
including software/technology, retail, media,
telecom and professional services.
About Slack

3. An unprecedented adoption rate

4. Agenda
1. A bit history.
2. Druid infrastructure & usecases
3. Challenges.

5. A bit history

6. March 2016
5 350+ 2M
Data Engineers Slack employees Active users

7. October 2017
10 800+ 6M
Data Engineers Slack employees Active users

8. Data usage
1 in 3 per
week
500+
tables
400k
access data
warehouse
Tables Events per sec

9. It is all about Slogs

10. Well, not exactly

11. Slog

12. Slog

13. Druid infrastructure & usecases

14. What can go wrong?

15. We want more...

16. Performance & Experimentation
● Engineering & CE team should be able to detect the performance
bottleneck proactively.
● Engineers should be able to see their experimentation performance in
near real-time.

17. Near Real time Pipeline

18. Keep the load in DW Kafka predictable.
More comfortable to upgrade and verify newer Kafka version.
Smaller Kafka cluster is relatively more straightforward to operate.
Why Analytics Kafka

19. Druid Architecture

20. Middle manager Autoscale based on number of running tasks.
Historical node autoscale based on the segment size.
Fault tolerance deployment for overlord & Coordinator
Brokers autoscale and load balanced by ELB.
Druid Architecture

21. Challenges

22. Cascading failures

23. Forward Index fields

24. SQL

25. Bridge the gap between batch and
realtime tables.

26. Thank You!
26