Talk on using puppet to configure icinga monitoring

1. Automating Monitoring
with Puppet
Chris Mague
Moovweb
May 23, 2012 1

2. Where I Want to be.....
+
2

3. What I'll Settle For.....
3

4. Requirements
Rock solid stability
Automated node addition (discovery)
Scales horizontally
Service dependency models
Easy to write plugins
Promotes sane workflows
Unified front end view
Flexible configuration
4

5. Tool Stack
5

6. What????
but #monitoringsucks and #ihatenagios
How could you?
6

7. In defense of Nagios
Been around since 1996
Has Service dependencies
Easy to write plugins
Easy-ish to troubleshoot
ROCK SOLID
7

8. Valid attacks on Nagios
No automated discovery
It's complicated to setup
Text files – really?
Front end won't win any beauty contests
Development is slow
Stats collection is a PITA
8

9. Solutions
Use Icinga!
Use Puppet to auto configure
Stats – leave it to graphite. It's really good at that
Big boys and girls learn their tools
9

10. Icinga
Fork of Nagios
Configurations are compatible
More solid architecture ( core, API, Web, IDODB )
Nice front end, nice mobile front end
Can use NRPE
10

11. High Level View
11

12. Configure Icinga Servers
using Puppet Standard Types
12

13. Things to configure with Standard Types
icinga.cfg (file) => icinga main config file
Apache icinga.conf (file) => http access to each server
cgiauth.cfg (file) => cgi access
cgi.cfg (file) => options, users
templates.cfg (file) got lazy => use for basic classes
idomod.cfg (template) => template for hostname to DB
13

14. Configure Icinga using Nagios Types
14

15. Puppet Nagios Types
nagios_command nagios_hostgroup
nagios_contact nagios_service
nagios_contactgroup nagios_servicedependency
nagios_host nagios_serviceescalation
nagios_hostdependency nagios_serviceextinfo
nagios_hostescalation nagios_servicegroup
nagios_hostextinfo nagios_timeperiod
15

16. Configuring Hosts
16

17. Overview
17

18. Detailed Overview
18

19. Store Configs
Store puppet info in a DB
Retrieve information from DB
Share info across nodes
Use thin_storeconfigs
Set up on puppet master
19

20. Exporting Nagios_host Resources
Export = Save to DB
Use facter for dynamic data
PRO TIP: use ENC
PRO TIP: use targets
PRO TIP: hostgroups
PRO TIP: use tags
20

21. PRO TIP: Use your ENC
21

22. PRO TIP: use targets
Use cfg_dir in icinga.cfg
Create a unique file per host or
service
Addition and removal are now
super easy
Also default dirs are in a
horrible place /etc/nagios
22

23. PRO TIP: hostgroups
Add machines to a hostgroup
Add services to a hostgroup
New machines inherit all of the
services associated with a
hostgroup
23

24. PRO TIP: use tags
Tags allow you to filter
resources so that you only
realize those resources that
you need
24

25. Configuring Services/Commands
25

26. Icinga Services
OR 'Stuff I want to monitor'
Associate with a hostgroup
Use a target
26

27. Icinga Commands
OR 'What actually gets run'
Use Macros to set paths in resource.cfg
27

28. Dependencies
28

29. PRO TIP: Dependencies
Unreliable services
Cut down on the number of alerts
Tell me what's really wrong
Route alerts accordingly
29

30. Nagios_servicedependencies
30

31. NRPE
31

32. NRPE
Runs on client
Secured via SSL
Has ACLs
Runs as nobody
Can run commands
Useful for other things...
32

33. Configuring NRPE
33

34. NRPE Checks
34

35. Plugins
35

36. exchange.nagios.org
36

37. Writing Plugins
Write in any language
Output 1 line to stdout
NRPE/Icinga/Nagios all use exit
codes to determine status
Run by hand to check
37

38. Workflows
38

39. Watching Monitoring
39

40. Scheduling Downtime
40

41. Filtering
41

42. Alerting
42

43. #monitoringisawesome
REMOVE unreliable checks
Just MONITOR – don't bolt on - especially stats
TIER your monitoring
Use timeperiods for sanity
Delegate responses
Use dependencies to pin down problems quickly
Work smart
43

44. 44

45. Resources
Icinga
http://icinga.org
Puppet
http://docs.puppetlabs.com/references/latest/type.html#nagioscommand
NRPE
http://nagios.sourceforge.net/docs/3_0/addons.html
IRC
##infra-talk, #icinga, #puppet
Contact
christianmague@gmail.com, @maguec, #gaijin (freenode), http://blog.mague.com
Thanks
45