1. Effective
Monitoring
with

2. @alq
CTO at
Datadog

3. An application
through the naked eye

4. An application
through a monitoring
tool

5. OODA Loop (simplified)
Observe
Orient
Act
Decide

6. OODA Loop (simplified)
Observe
Orient
Act
Decide

7. M
on
ito
g rin
To
ol
OODA Loop (simplified)
Observe
Orient
Act
Decide

8. Yo
u
M
on
ito
g rin
To
ol
OODA Loop (simplified)
Observe
Orient
Act
Decide

9. Yo
u
M
on
ito
g rin
To
ol
OODA Loop (simplified)
Orient
Yo
u
Observe
Act
Decide

10. M
on
ito
g rin
To
ol
Yo
u
Yo
u
Yo
u
OODA Loop (simplified)
Observe
Act
Orient
Decide

11. Observations need to
be...
1.Timely
2.Correct
3.Comprehensive

12. Observations need to
be...
1.Timely
2.Correct
3.Comprehensive

13. Observations need to
be...
1.Timely
2.Correct
3.Comprehensive
Else

14. Observations need to
be...
1.Timely
2.Correct
3.Comprehensive
Else
Garbage
In, Garbage
Out

15. Timely
Initial
assumptions
Initial set of
metrics
Contact
with reality
Revised
assumptions
Revised set
of metrics

16. M
N in
ot ut
we es
ek
s
Timely
Initial
assumptions
Initial set of
metrics
Contact
with reality
Revised
assumptions
Revised set
of metrics

17. Comprehensive
Resources
Resources
Resources
Resources
Resources
Work
Value

18. Comprehensive
Easy to collect
generic
but not actionable
Resources
Resources
Resources
Resources
Resources
Work
Value

19. Comprehensive
Easy to collect
generic
but not actionable
Resources
Resources
Resources
Resources
Resources
Work
Value
Harder to collect,
custom
but most actionable

20. statsD
Easy

21. statsD
Easy
Timely

22. statsD
Easy
Timely
Comprehensive

23. How statsD works
pageviews:100|
c@0.25
latency:320|ms
backlog:333|g
uniques:765|s
Client libraries talk to a
simple UDP server...
...using a simple text protocol

24. statsD types
Type
Definition
Example
Gauges
Absolute values
Queue size
Counters
Per-second rates
Page views
Histograms
Gauge summary
Page Latency
Timers
Gauge distribution
Page Latency
Sets
Counters of unique
things
Unique visitors

25. statsD problems
Type
Gauges
Counters
Histograms
Timers
Sets
Definition
Problem
Absolute values
Latest value wins.
Gauge deltas???
Per-second rates
Rates, not counts (!
= rrdtool)
Gauge summary
Assumes normal
distribution
Can measure much
Gauge distribution
more than time
Counters of unique
things
:-)

26. #1 pitfall: “Counters”
http://dtdg.co/tokyo-counters

27. How we use statsD
http://dtdg.co/tokyo-dog

28. Essential: Tagging
http://dtdg.co/tokyo-tags

29. How to get started
• statsD https://github.com/etsy/statsd
• client libraries https://github.com/etsy/statsd/wiki
(my company) 1-stop shop http://www.datadoghq.com

30. ありがとうございました。
質問？@alq
Thank you very much!
Questions？ @alq