Daniel Wittenburg' presentation on a reference story for a German Health Insurance Company. The presentation was given during the Nagios World Conference North America held Sept 27-29th, 2011 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/nwcna

1. Scaling Nagios At A Giant Insurance Company Daniel Wittenberg [email_address] https://github.com/dwittenberg2008/nagios

3. Worked on lots of different technologies and solutions including HA, SAN/iSCSI, Forensics/Security, Backups/Disaster Recovery, Performance Tuning, Capacity Planning, Monitoring/Trending, Networking/Protocol Analysis, Virtualization and Cloud Computing.

4. Consulted and worked in many industries include insurance, banking, accounting, construction, embedded hardware design, printing/publishing early education, higher education, and ISP/hosting providers.

6. Operating System

7. Nagios Core

8. Plugins

9. Other Add-ons

10. Event Brokers

11. Other Software

12. Performance Monitoring

13. General

14. Overview

19. Strip down the running services

20. Create ramdisks in Nagios RC script

21. - first one for status.dat, checkresults, temp_file

22. - second one for pnp processing directory, moved there by nagios perf command

23. - nagios.rc on github for full rc script ramdisk=`mount |grep "/var/nagios/ramcache"` if [ "$ramdisk"X == "X" ]; then mkdir -p -m 755 /var/nagios/ramcache mount -t tmpfs -o size=128m tmpfs /var/nagios/ramcache mkdir -p -m 755 /var/nagios/ramcache/checkresults chown -R nagios:nagios /var/nagios/ramcache fi

25. ulimit -a

26. Renice daemons and services

27. daemon -15 --user=$user $exec -d $config

28. perfdata_file_run_cmd =/bin/nice -n 20 /usr/libexec/pnp4nagios/process_perfdata.pl

29. puppet runs also re-niced (/etc/sysconfig/puppet – NICELEVEL=19)

32. Re-write RC script to optimize restart time

33. Don't allow restart/stop if config broken

34. Limit use of macros (resources.cfg)

36. Keep logging level low in all core and plugins/brokers

37. Keep comments limited, delete if X # or Y days old

38. status_update_interval=20 (default is 10)

39. (how often to update the status.dat in seconds)

40. enable_environment_macros=0 (default is 1)

41. (pass macros as ENV variables)

43. check_logfiles

44. check_hpasm / check_dell_sensors / check_dell_omreport

45. check_oracle_health – check_mysql_health

46. check_ps.sh (re-written for perf data, correct calculations)

47. nagios_auto_service

48. Return perf data whenever possible

49. Many other custom and one-up plugins

51. Run from xinetd to allow larger/faster connections

52. MUST use instances = UNLIMITED

53. Recommend per_source = UNLIMITED

56. Mod-Gearman (under evaluation)

57. MK Livestatus

58. Performance Data Splunker (custom)

59. Log separator (reduces grepping for messages) (custom)

63. iostat

64. top

65. iptraf

66. sar

67. strace

68. esxtop (if have to use VM)

70. Hosts are tied to a hostgroup

71. Hostgroups are tied to a servicegroup

72. Services are tied to a servicegroup

73. host.cfg -> hostgroups -> servicegroups ← services

74. This allows for easy drop-in and removal of hosts, but also requires at least 1 host be assigned to a management server

75. Limitations – harder to make per-server per-service customizations

76. Hosts are built/assigned from control panel (round-robin distribution)

77. Parents built automatically from topology database, updated nightly, ESX hourly

78. Parent's only ping once a day unless there are problems, uses fping

79. Some alerts do trigger eventhandlers – auto-fix as much as possible

81. Nagios configtest, livestatus connectivity

82. PNP4Nagios directory size (keeping up on processing)

83. Performance (cpu/memory) usage on certain processes

84. Puppet update time to make sure doesn't get behind

85. DB Response times (oracle/mysql/postgresql)

86. Apache Stats

87. Custom app status (user accounts, response times, loads, etc.)

88. Various SNMP/WMI values (most network related stats)

90. MK Livestatus - http://mathias-kettner.de/checkmk_livestatus.html

91. LivestatusSlave - http://nagios.larsmichelsen.com/livestatusslave/

92. PNP4Nagios - http://docs.pnp4nagios.org/pnp-0.6/start

93. ConSol Labs - http://labs.consol.de/

94. Puppet - http://puppetlabs.com/

95. DNX - http://dnx.sourceforge.net/

96. Cacti Template (Base) - http://forums.cacti.net/about33806.html

97. Questions ? Suggestions ?