Presentation given at Icinga Camp Stockholm in October 2016

1. How to make your monitoring
shut up
Observations on what to monitor and when to alert

2. Hello!
@moegyver@EttLejon
(tho I hardly use twitter…)
(easier to find me as lejonet
@Freenode/OFTC IRC)

3. What is your name?
Ops/DevOps
What is your quest?
A healthy and functioning server
environment
What is… your favourite time to
be woken up by an alert?
Erh.. Never.. Erh office hou
aaaaaaaaaah!

4. Example Checklist for monitoring vs alerting
1. What am I actually monitoring, service or details of
part of a service (hardware, connectivity, etc)?
If the answer isn’t a service, then it might not be fit
for alerting
2. Will I get fired for not responding directly to this?
(aka is it business critical)
If the answer to that is yes, maybe its fit for
alerting
3. Is what I want to check possibly covered by
another alert or monitoring check? E.g.
connectivity is also answered by a check for if / in
the webroot can be requested, in combination with
answering if the webserver is properly configured

5. And for the love of all that is compiling, keep all the monitoring and alerting in _ONE_
system!
(if you need/want to use more than one, atleast aggregate the end result to _ONE_
system)

6. Arguing about how to monitor/alert or with
what takes away from the fact that you’re
not that interested in that the monitoring is
done but what information it gives you.
AKA
Even someone pressing F5 in their browser
to verify that the site is up is more
informative than: Server has 1337 MB RAM
free, / is 42% free and there is 1
(management ;) ) zombie process

7. Alerting

8. alert(“a service wants
attention”)

9. TMI!
DISK CRITICAL - free space: / 7002 MB (9% inode 60%):
/data 16273093 MB (26% inode 99%)

10. ???
DISK CRITICAL - free space: / down to 9%

11. ✓
DISK CRITICAL - root will be out of space in 2 h

12. Define sane SLOs

17. Always fix your alerts!

20. A bad fix is better than
no fix

22. Once upon a time...

23. License:CC BY-ND 2.0, Wisconsin Department of Natural Resources

25. How did we get there?

26. Let’s do this!

27. Let’s do this!

28. Let’s do this!

31. License: CC-SA, Marco Mayer
2 years and counting...

32. You are never done done

34. Qs?