Superb Supervision of Short-lived Servers with Sensu

1. Paul O’Connor paul.oconnor@yelp.com Superb Supervision of Short-lived Servers with Sensu

2. Yelp’s Mission Connecting people with great local businesses.

3. Short-Lived Servers

4. Short-Lived Servers • Servers in auto-scaling groups

5. Short-Lived Servers • Servers in auto-scaling groups • Short batch servers existing while the batch runs

6. Short-Lived Servers • Servers in auto-scaling groups • Short batch servers existing while the batch runs • Ensuring latest image build

7. Why Sensu?

8. Why Sensu? • Designed to be pluggable / extensible • Arbitrary check metadata

9. Why Sensu? • Designed to be pluggable / extensible • Arbitrary check metadata • Simple model • Components do exactly one thing

10. Why Sensu? • Designed to be pluggable / extensible • Arbitrary check metadata • Simple model • Components do exactly one thing • Ruby • Not afraid to extend (or fork!)

12. https://laur.ie/blog/2014/02/why-ill-be-letting-nagios-live-on-a-bit-longer-thank-you-very-much/ “Sensu has so many moving parts that I wouldn’t be able to sleep at night unless I set up a Nagios instance to make sure they were all running.”

13. How to use Sensu

14. How to use Sensu • Don’t use all of this!

15. How to use Sensu • Don’t use all of this! • Standalone checks only • Default in Puppet module

16. Sensu Data Flow • Sensu client runs checks locally on each machine • Results are published to RabbitMQ Cluster

17. Sensu Data Flow • Sensu client runs checks locally on each machine • Results are published to RabbitMQ Cluster • Sensu Server in H/A Cluster • Processes check results from RabbitMQ • Invokes appropriate handlers • Writes state to Redis

18. Sensu Data Flow • Sensu client runs checks locally on each machine • Results are published to RabbitMQ Cluster • Sensu Server in H/A Cluster • Processes check results from RabbitMQ • Invokes appropriate handlers • Writes state to Redis • Redis + Redis Sentinel • 2+ instances in each cluster • Read by the Sensu API

19. Sensu Data Flow • Sensu client runs checks locally on each machine • Results are published to RabbitMQ Cluster • Sensu Server in H/A Cluster • Processes check results from RabbitMQ • Invokes appropriate handlers • Writes state to Redis • Redis + Redis Sentinel • 2+ instances in each cluster • Read by the Sensu API • Every layer is behind HAProxy

20. Mutually Assured Monitoring • Multiple independent Sensu clusters per data centre/environment • 2+ RabbitMQ Servers • 2+ Redis Servers • 2+ Sensu Server/API Servers

21. Mutually Assured Monitoring • Multiple independent Sensu clusters per data centre/environment • 2+ RabbitMQ Servers • 2+ Redis Servers • 2+ Sensu Server/API Servers • Each cluster monitors each other

22. • /etc/sensu/conf.d/checks/$check_name.json Machine Readable Config

23. • /etc/sensu/conf.d/checks/$check_name.json • One check per file Machine Readable Config

24. Machine Readable Config • /etc/sensu/conf.d/checks/$check_name.json • One check per file • Extensible with arbitrary metadata

25. Machine Readable Config • /etc/sensu/conf.d/checks/$check_name.json • One check per file • Extensible with arbitrary metadata • Hash merge

26. Machine Readable Config • /etc/sensu/conf.d/checks/$check_name.json • One check per file • Extensible with arbitrary metadata • Hash merge • Never edit by hand!

27. Let Puppet Do The Work • Puppet is already working in the environment

28. Let Puppet Do The Work • Puppet is already working in the environment • It knows everything about every node in the environment

29. Let Puppet Do The Work • Puppet is already working in the environment • It knows everything about every node in the environment • Puppet is human readable

30. monitoring_check { ‘check_disk_slash’: page => true, check_every => ‘5m’, alert_after => ‘30m’, realert_every => 10, runbook => ‘http://wiki/check_disk/slash', command => ‘/usr/lib/nagios/plugins/check_disk -c 10% - K 10 -p /‘, } monitoring_check

35. sensu::check • monitoring_check wraps this

36. sensu::check • monitoring_check wraps this • Writes a JSON file for each check

37. sensu::check • monitoring_check wraps this • Writes a JSON file for each check • Comment safe

38. { "checks": { "check_disk_slash": { "standalone": true, "handlers": [ "default" ], "command": "/usr/lib/nagios/plugins/check_disk -c 5% -K 5 -p /", "dependencies": [ ], "interval": 300, "timeout": 300, "alert_after": 300, "realert_every": "10", "runbook": "http://wiki/disk-slash", "team": "operations", "irc_channels": "operations-notifications", "notification_email": "undef", "ticket": false, "project": false, "page": true, "tip": "Try: sudo du -h -x --max-depth=1 /", "tags": [ ] } } }

43. • Same as Nagios checks • Simple text output • Posix Exit Codes Check Scripts

44. Check Scripts • Same as Nagios checks • Simple text output • Posix Exit Codes • Result sent to Sensu Server, along with check definition • Includes all custom metadata • Custom handlers process the extra data

45. • base Handlers

46. • base • JIRA Handlers

47. Handlers • base • JIRA • email

48. Handlers • base • JIRA • email • irc

49. Handlers • base • JIRA • email • irc • pagerduty

50. How Do The Checks Get Executed? • Each machine runs the client

51. How Do The Checks Get Executed? • Each machine runs the client • Client is managed entirely by Puppet

52. Situational Awareness

53. Single Source of Truth • DNS is canonical source for sensu servers

54. # Use DNS to detect if this server is a sensu server $local_sensu_server_ips_array = gethostbyname2array("sensu.local-$ {::habitat}.yelpcorp.com") $array_intersection = intersection($::all_ipaddresses, $local_sensu_server_ips_array) # If one of our ipaddresses are in the dns entries, we must be a sensu server! if size($array_intersection) > 0 { $is_sensu_server = true $local_sensu_server_array = gethostbyaddr2array($local_sensu_server_ips_array) Single Source of Truth • DNS is canonical source for sensu servers • Configure things in one place

58. Automatic Monitoring • Cron Jobs - check if a job was completed successfully

59. if $staleness_threshold { $actual_command = "/nail/sys/bin/success_wrapper '${reporting_name}' $ {command}" cron::staleness_check { $reporting_name: threshold => $staleness_threshold, params => $staleness_check_params, user => $user, } } else { $actual_command = $command } Automatic Monitoring • Cron Jobs - check if a job was completed successfully • cron::d

62. define cron::staleness_check( $threshold, $params, $user, ) { $threshold_s = cron_human_time_to_seconds($threshold) # Check whether we are fresh five times per threshold, not to exceed 1 hour if $threshold_s / 5 > 3600 { $check_every = 3600 } else { $check_every = $threshold_s / 5 } $check_title = "${name}_staleness" $overrides = { 'command' => "/nail/sys/bin/cron_staleness_check ${name} ${threshold_s}", 'check_every' => $check_every, 'needs_sudo' => true, 'alert_after' => '2m', } $check_data = { "$check_title" => merge( $params, $overrides ) } create_resources('monitoring_check', $check_data) file { "/nail/run/success_wrapper/${name}": ensure => 'file', owner => $user, mode => '640', } -> Monitoring_check[$check_title] }

68. Automatic Remediation • Make the computer try something before paging

69. Automatic Remediation • Make the computer try something before paging • Try it repeatedly if necessary monitoring_check { ‘check_syslogd’: page => true, check_every => ‘5m’, alert_after => ‘10m’, realert_every => 10, runbook => ‘http://wiki/syslogd', command => ‘/usr/lib/nagios/plugins/check_proc syslogd‘, remediation_action => ‘/etc/init.d/syslogd start’, remediation_retries => 1 }

70. Server Maintenance • Don’t alert on call if someone is working on a server

71. function is_sensu_cli_available { which sensu-cli >/dev/null return $? } deregister () { if is_sensu_cli_available ; then # By this run level, sensu-client should be stopped, but we can stop anyway. service sensu-client stop 2>/dev/null sensu-cli client delete `/opt/puppet-omnibus/bin/facter fqdn` fi } case "$1" in start) echo "$0 does nothing on start." stop) # Refuse to run on any runlevel except 0. if runlevel | grep -q 0; then deregister fi ;; esac rc0.d

76. function is_sensu_cli_available { which sensu-cli >/dev/null return $? } function silence_for { if is_sensu_cli_available; then if sensu-cli stash show "silence/${fqdn}" | grep -q "total items"; then echo "Silence already detected for ${fqdn}. Not replacing the existing silence" else echo "Automatically silencing this host, ${fqdn} for $1 seconds in sensu..." sensu-cli silence "${fqdn}" --expire "$1" --reason "Reboot initiated." fi else echo "sensu-cli is unavailable" fi } case "$1" in start) echo "Sensu silence does nothing on start" exit 0 ;; stop) # Refuse to run on any runlevel except 6. echo "Sensu silenced by reboot " | logger -s -t sensu-silence silence_for 1800 2>&1 | logger -s -t sensu-silence ;; esac rc6.d

81. Cluster Checks • Assert some % of machines are healthy • Use to reduce alert noise

82. Cluster Checks • Assert some % of machines are healthy • Use to reduce alert noise • If a cluster becomes unavailable, you want someone to be paged

83. Cluster Checks • Assert some % of machines are healthy • Use to reduce alert noise • If a cluster becomes unavailable, you want someone to be paged • If one machine becomes unavailable, it’s not a problem - open a JIRA ticket to get it fixed in core hours

84. @YelpEngineering fb.com/YelpEngineers engineeringblog.yelp.com github.com/yelp

Superb Supervision of Short-lived Servers with Sensu

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Superb Supervision of Short-lived Servers with Sensu

Ähnlich wie Superb Supervision of Short-lived Servers with Sensu (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Superb Supervision of Short-lived Servers with Sensu