This talk explains how we use Prometheus, ansible to resolve incidents in an unattended way. We use for that amqp and Jenkins.

1. Incident Resolution as Code
Julien Pivotto (@roidelapluie)
Config Management Camp
February 4th, 2019

2. user{name="roidelapluie"} 1
I like Open Source
I like monitoring
I like automation
... and all of that is my daily job at inuits

3. Monitoring
Creative Commons Attribution 2.0 https://www.flickr.com/photos/24375810@N06/3719090065

4. Creative Commons Attribution ShareAlike 2.0 https://www.flickr.com/photos/grendelkhan/400428874

5. Creative Commons Public Domain https://pxhere.com/en/photo/265717

6. Creative Commons Attribution 2.0 https://www.flickr.com/photos/51809988@N06/5229933669

7. Traditional Monitoring
It works - OK
It does not work - CRITICAL
It kinda works - WARNING
I don't know - UNKNOWN

9. Creative Commons Public Domain https://pxhere.com/fr/photo/952999

10. Creative Commons Attribution 2.0 https://www.flickr.com/photos/wwarby/2460655511

11. Creative Commons Attribution-Share Alike 3.0 Unported
https://commons.wikimedia.org/wiki/File:CUPE3903-picketLine-20180504.jpg

12. Real world
It works ; it does not work ; it kinda works ; it maybe
works ; no one uses it ; it is broken ; some things
are broken ; it should work but it does not ; where
are my users? help me...

13. The Technical bias
By looking at technical service, we miss the
actual point
Are we serving our users correctly?
Just looking at the traffic light will not tell you
about the traffic jams.

14. Observability
Creative Commons Attribution 2.0 https://www.flickr.com/photos/24375810@N06/3719090065

15. Metrics
Creative Commons Attribution-Share Alike 2.0 https://www.flickr.com/photos/tillwe/11892564676/

16. Metric
Name
Labels (Key-Value Pairs)
Value (Number)
Timestamp
Fetched at a high frequency

17. Business Metrics
Creative Commons Attribution 2.0 https://www.flickr.com/photos/89228431@N06/11323330056

18. CPU usage is no money
Creative Commons Attribution-ShareAlike 2.0
https://www.flickr.com/photos/nox_noctis_silentium/3960497840

19. What are business metrics?
how you fullfil your customers' requests
quality and level of business service

20. Where are we?
Creative Commons Attribution-ShareAlike 2.0 https://www.flickr.com/photos/hernanpc/6259950189

21. What do we have?
metrics that tell us if business works
DB, Frontends, balancers, queing systems...
They don't come from the troublesome
component!

22. Recap on High Availability

23. Traditional High Availability
LinuxHA, pacemaker, corosync
"health check" script
Restart, promote, balance traffic elsewhere

24. High Availability Nowadays
Multiple workers
Health exposed by the app
Load balancer balances to healthy nodes
Unhealty nodes are restarted automatically

25. When HA is not enough...
the processes are not "really" crashing
the component that has issues does not really
know about them (metrics available from DB ,
frontends, clients..)
there is no HA in place... (but still need 24x7
availability)

26. Async cases
Backlog can be delayed by a few hours
But not a complete week-end ...

27. Alerting
alert is fired = someone to take action
Runbooks to follow, depending on the alert
knowledge is built, then runbook == (ansible)
playbook

28. An ideal world
Creative Commons Attribution 2.0 https://www.flickr.com/photos/athomeinscottsdale/3247600886/

29. Ideally...
Memory leaks are fixed (quickly)
Multi master, redundant, in service discovery
You build it, you run it
Full control over 3rd parties (and their bugs..)

30. Ironically
Developers often just payed for features
Ops not working closely with devs
No "bugfix money" for stuff that do not happen
really often
Code base is 20y old and "it will be
decommisioned soon"

31. What's next?
Creative Commons Attribution 2.0 https://www.flickr.com/photos/janitors/15795816662/

33. if Frontends says backend responds slowly
then Restart the backend

34. if Lots of write errors towards NFS
then Balance traffic to another datacenter

35. Challenges
How to prevent concurrent ansible runs?
How to avoid large scale failure? e.g. errors in
playbook?
How to make sure it works?

37. In pratice

38. Prometheus
Time series collection database
Computing rules, creating alerts
View on all the components
Open Source

39. Alertmanager
Receives alerts from prometheus
Propagates the alerts via webhooks
Open Source (part of Prometheus)

40. Ansible
Open Source orchestration/cfgmgmt
Acts on multiple servers
Knows your infra

41. Webhook that call Ansible
How/where to get the credentials? Where to
run Ansible?
Duration of the ansible run?
Which server to act upon?
Concurrent playbooks?

43. Queues
Alerts as JSON
Labels: env, playbook, limit
alert2amqp
AMQP protocol
Apache Artemis message broker (activemq
family)

44. Consumer
amqp2jenkins
Reads from queue
Launch a Jenkins job that will run ansible
Stop processing if a Jenkins job is failing
Ansible jobs take 2 input

46. Recap
In an ideal world we do not need this. Bugs are
fixed, techno is up to date, infra and apps are
reduntant.

47. Achievement
Metrics from 1, 2, X sources generate alerts that
are triggering automated resolution within minutes
towards different systems.
Common incidents get solved more quickly than
with people intervention.
People are woken up less often for known issue
with clear runbook.

48. Safeties
Needs monitoring to be up
Needs the last ansible run to be green
Whitelist upfrond
Discards "old alerts"
No concurrent execution
Alerts someone if not resolved in time

49. Use Cases
Must happen infrequently
Must not be predictable
Must not do more harm
Must impact daily work or on call

50. Code
https://github.com/roidelapluie/alert2amqp
https://github.com/roidelapluie/amqp2jenkins

51. Julien Pivotto
roidelapluie
roidelapluie@inuits.eu
Inuits
https://inuits.eu
info@inuits.eu
Contact