Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Wolfgang Alper - Zabbix Meets OPS Control / Rundeck | ZabConf2016

Hier ansehen

1 von 35 Anzeige
1 von 35 Anzeige

Wolfgang Alper - Zabbix Meets OPS Control / Rundeck | ZabConf2016

Herunterladen, um offline zu lesen

Zabbix is an excellent tool to do network monitoring and to alert if something bad happens. But Zabbix can do more. An underestimated feature of Zabbix is its ability to perform actions in addition to simple notifications. However, this requires to precisly setup those actions within zabbix, which is not always an easy task and might duplicate existing work. So what if Zabbix actually worked in concert with an external taskrunner / jobscheduler that is build to do exactly this: run a task or action against a host and report its outcome? Zabbix would perform the same well defined steps that an ops member would perform in case of certain failures using this kind of tool. A well know example of this kind of software is "Rundeck" which is licensed under the Apache License Version 2.0.

Zabbix is an excellent tool to do network monitoring and to alert if something bad happens. But Zabbix can do more. An underestimated feature of Zabbix is its ability to perform actions in addition to simple notifications. However, this requires to precisly setup those actions within zabbix, which is not always an easy task and might duplicate existing work. So what if Zabbix actually worked in concert with an external taskrunner / jobscheduler that is build to do exactly this: run a task or action against a host and report its outcome? Zabbix would perform the same well defined steps that an ops member would perform in case of certain failures using this kind of tool. A well know example of this kind of software is "Rundeck" which is licensed under the Apache License Version 2.0.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Andere mochten auch (19)

Anzeige

Wolfgang Alper - Zabbix Meets OPS Control / Rundeck | ZabConf2016

  1. 1. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meets Ops / Rundeck Integrate Zabbix trigger based actions into a taskrunner like rundeck Contact: Wolfgang Alper Email: wolfgang.alper@intellitrend.de www.intellitrend.de IntelliTrend IT-Services GmbH Otto-Brenner-Strasse 119 D-33607 Bielefeld Germany
  2. 2. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Real world example Customer ticketsystem uses a “search service”. When the search service fails: • Login to server running the search service. • Remove corrupt indices. • Rebuild search indices. We use Zabbix for monitoring the search service :-)
  3. 3. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Typical scenario Zabbix alerts Problem start Zabbix: fire trigger Zabbix: send email (create ticket) Ops Team: escalate Problem persists? Ops Team: run default action Opt: Acknowledge triggered event in zabbix
  4. 4. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Advanced Scenario Zabbix alerts and executes a command Problem start Zabbix: escalate Problem persists?Opt: Acknowledge triggered event in zabbix Zabbix: fire trigger Zabbix: send email (create ticket) Zabbix: execute remote command
  5. 5. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Reaction on trigger based event Zabbix action / Ops action Ops Team: run default action Command 1 Command 2 ... Command n Command 1 Command 2 ... Command n Zabbix: execute remote command If default action will not solve the problem, command output from default action would be helpful for further escalation. ● Commands are similar / identical ● Command execution requires a valid user account on target system
  6. 6. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Ops team Command 1 Command 2 ... Command n Non standard tasks - Direct access: Taskrunner Default / frequent tasks - Taskrunner: Different ways to react:
  7. 7. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Advanced Ops team options What is a taskrunner?
  8. 8. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck What is a taskrunner ● Executes commands on remote hosts. ● Defines jobs as a reusable sequences of commands. ● Provides central storage of credentials / keys. ● Preserves output from executed commands. ● Maintains a history of job executions. ● Separates access to job executions from direct access to remote hosts.
  9. 9. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck What is a taskrunner Executing a task through “Rundeck” (http://rundeck.org)
  10. 10. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Taskrunner Rundeck – Job definition
  11. 11. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Taskrunner Rundeck – Job result
  12. 12. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Taskrunner Rundeck – Log output
  13. 13. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Taskrunner Rundeck – Job activity ● Execution Id ● Date, Duration ● Who executed ● Command output available
  14. 14. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Execute via taskrunner Execute job through taskrunner by Ops Ops Team: run default action Command 1 Command 2 ... Command n ● Taskrunner executes pre defined set of commands against target host(s). ● Taskrunner preserves result state and output of executed commands. Taskrunner ● Ops Team selects host on taskrunner. ● Ops Team selects job on taskrunner. ● Ops Team executes job on taskrunner. Remote Host
  15. 15. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix: execute remote command Zabbix meet Ops / Rundeck Execute via taskrunner Execute job through taskrunner by Zabbix ● Taskrunner executes pre defined set of commands against target host(s). ● Taskrunner preserves result state and output of executed commands. Taskrunner ● Zabbix selects host on taskrunner. ● Zabbix selects job on taskrunner. ● Zabbix executes job through taskrunner. Command 1 Command 2 ... Command n Remote Host
  16. 16. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix - Rundeck Integration Zabbix <-> Rundeck
  17. 17. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix - Rundeck Soft migration from standard Ops actions to Zabbix remote commands. Easier overall maintenance of actions, no duplicated sets of commands. Less error prone definition of remote commands. Easier testing of Zabbix remote commands. Preserved output from executed remote actions. Central location for access credentials and keys. Advantages
  18. 18. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix - Rundeck ● Map Zabbix host to Rundeck nodes. ● Map Zabbix trigger names to Rundeck job names. ● On trigger event, pass related host and trigger from Zabbix to Rundeck. ● Bonus: Return job execution status from Rundeck as an event acknowledgment in Zabbix. ... Zabbix API, Rundeck API, Middleware, ... Requirements
  19. 19. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix - Rundeck Zabbix hosts Rundeck nodes Zabbix triggers Rundeck jobs Zabbix event Rundeck executionZabbix action calls rundeck job - Rundeck API - Bonus: Rundeck acknowledes event in Zabbix Map triggers to jobs - Zabbix Action - Map hosts to nodes - Zabbix API -
  20. 20. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix - Rundeck Hosts and Hostgroups Integration
  21. 21. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix – Rundeck: Hosts Zabbix as node(host) provider for Rundeck Zabbix API Rundeck calls middleware as a node provider Get hosts, hostgroups, interfaces and meta information from Zabbix
  22. 22. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix - Rundeck Trigger and Action Integration
  23. 23. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix – Rundeck: Trigger Execute Rundeck job from Zabbix - simple Zabbix Trigger Rundeck-API Select specific Zabbix action by trigger name Zabbix Action calls Rundeck API Call Rundeck API with hardcoded jobname in action Action B: Action A:
  24. 24. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix – Rundeck: Trigger Execute Rundeck job from Zabbix - simple Zabbix Trigger Rundeck-API Select specific Zabbix action by trigger name Zabbix Action calls Rundeck API Call Rundeck API with hardcoded jobname in action Action B: Action A:
  25. 25. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix – Rundeck: Trigger Zabbix Trigger A Rundeck-API Zabbix Action A Zabbix Trigger B Zabbix Action B Zabbix Trigger C Zabbix Action C Zabbix Trigger ... Zabbix Action ... Simple implementation Each trigger requires its own Zabbix action
  26. 26. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix - Rundeck Trigger and Action Integration We can do better ;-)
  27. 27. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix – Rundeck: Trigger Execute Rundeck job from Zabbix - advanced Zabbix Trigger Select default Rundeck Zabbix action and pass trigger description Zabbix actions call Rundeck API through middleware Rundeck-API Zabbix Action calls middleware
  28. 28. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix – Rundeck: Trigger Execute Rundeck job from Zabbix - advanced Zabbix Trigger Select default Rundeck Zabbix action and pass trigger description Zabbix actions call Rundeck API through middleware Rundeck-API Zabbix Action calls middleware
  29. 29. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix – Rundeck: Trigger Zabbix Trigger A Rundeck-APIZabbix Action RD Zabbix Trigger B Zabbix Trigger C Zabbix Trigger ... All trigger share one Zabbix action Middleware extracts job name from trigger desc.
  30. 30. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix – Rundeck To execute a rundeck job form a zabbix trigger event: ● Add “[RD]” to trigger name (optional). ● Add “[RD:<Rundeck-Job-Name>]” to trigger description. ● Done :-) Summary
  31. 31. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Integration Zabbix - Rundeck Bonus: Automatic acknowledgment through executed tasks
  32. 32. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Automatic acknowledgment ● Remote actions can take longer than a few seconds, however Ops team should “see” that there is a remote action and the initial state. ● Ops Team should “see” the current execution ID of the job or get a http link to it. ● Zabbix event should show the acknowledge as the user “rundeck”. Overall – Acknowledgment from Rundeck jobs give more transparency of what is going on. To consider
  33. 33. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Automatic acknowledgment - Design Zabbix Trigger A Rundeck-API Zabbix Action RD Zabbix Trigger B Zabbix Trigger C Zabbix Trigger ... Middleware extracts job name from trigger desc. Zabbix Acknowledges +Middleware Ack. event via Zabbix API +Get job execution ID from Rundeck API +Pass event ID +API User “rundeck”
  34. 34. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meet Ops / Rundeck Automatic acknowledgment - Result Zabbix event info: Remote action Zabbix Event: Zabbix event info: Acknowledgment info with link to rundeck job execution
  35. 35. © Copyright 2016 IntelliTrend GmbH ● Otto-Brenner-Straße 119 ● 33607 Bielefeld ● Germany ● Tel: +49 521 557716-0 ● www.intellitrend.de Zabbix meets Ops / Rundeck Thank You! Contact: Wolfgang Alper Email: wolfgang.alper@intellitrend.de www.intellitrend.de IntelliTrend IT-Services GmbH Otto-Brenner-Strasse 119 D-33607 Bielefeld Germany

×