Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Ryan Armstrong - Monitoring More Than 6000 Devices in Zabbix | ZabConf2016

Wird geladen in …3
×

Hier ansehen

1 von 22 Anzeige
1 von 22 Anzeige

Ryan Armstrong - Monitoring More Than 6000 Devices in Zabbix | ZabConf2016

Herunterladen, um offline zu lesen

Ryan will describe a Skunkworks project executed by Kinetic IT at the Department of Education to deliver an autonomous infrastructure monitoring solution for over 6000 devices distributed across WA. The team were given opportunity to experiment with DevOps practices such as Scrum product development, Infrastructure As Code and Continuous Integration to determine where the value lay and which practices should be adopted at greater scale.

Ryan will describe a Skunkworks project executed by Kinetic IT at the Department of Education to deliver an autonomous infrastructure monitoring solution for over 6000 devices distributed across WA. The team were given opportunity to experiment with DevOps practices such as Scrum product development, Infrastructure As Code and Continuous Integration to determine where the value lay and which practices should be adopted at greater scale.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (19)

Ähnlich wie Ryan Armstrong - Monitoring More Than 6000 Devices in Zabbix | ZabConf2016 (20)

Anzeige

Weitere von Zabbix (20)

Anzeige

Ryan Armstrong - Monitoring More Than 6000 Devices in Zabbix | ZabConf2016

  1. 1. Monitoring 7000+ hosts in Zabbix Automate all the things
  2. 2. Commercial in Confidence | 3 Today we’re talking about… • Zabbix Deployment • Automation
  3. 3. Commercial in Confidence | 4 About me • Kinetic IT are a Managed Service Provider in Australia https://www.kineticit.com.au/ • I work as a Senior Engineer in the Tools and Automation team at WA Department of Education • Blog: http://cavaliercoder.com/ • GitHub: https://github.com/cavaliercoder/ • Twitter: https://twitter.com/cavaliercoder • LinkedIn: https://www.linkedin.com/in/ryanarmstrongwa
  4. 4. Commercial in Confidence | 5 WA Department of Education FACT: Germany can fit into WA 7 times! Managing over 120,000 devices for approx. 400,000 people in 800 schools With 1500 heterogeneous servers Hub-spoke topology Across 2.7 million km²
  5. 5. Commercial in Confidence | 6 Production stats 7000+ 1.5M 450K 2K Hosts Items Triggers NVPS
  6. 6. Commercial in Confidence | 7 Zabbix topology • Physical Zabbix server Dell R720 64GB memory • Physical PostgreSQL server Dell R720 256GB memory o Local SSD in RAID 10 o SAN attached storage for Tier 2 and archival data o PostgreSQL partitioning • Virtual Web server for users • 2nd Virtual Web server with high memory and long session optimisations for batch processing
  7. 7. Commercial in Confidence | 8 • Spend time on adding value! Why automate? So we can spend time adding value!
  8. 8. Commercial in Confidence | 9 Why automate? • We want to spend our time: o Improving monitoring quality and coverage o Improving Event Management o Improving BI and capacity planning o Enhancing the user experience • Not: o Adding/removing users and hosts o Keeping dev/test/prod in sync
  9. 9. Commercial in Confidence | 10 Build Pipeline • Provision Dev in Vagrant • Build and configure with Puppet Enterprise • Store all code in GitLab • Package custom binaries with RPM and WiX
  10. 10. Commercial in Confidence | 11 Build Pipeline • Puppet keeps Vagrant, dev, test and production identical • Rebuild on a whim (Cattle, not pets)
  11. 11. Commercial in Confidence | 12 • Nightly builds o Team City o Vagrant o Cucumber Build Pipeline
  12. 12. Commercial in Confidence | 13 Host and User Management • Hourly cron jobs • Modelled ‘views’ from data sources • Remediate differences via Zabbix API
  13. 13. Commercial in Confidence | 14 User Management • INI file configuration • Define LDAP groups • Script queries AD/LDAP • Sync users/groups via Zabbix API
  14. 14. Commercial in Confidence | 15 Host Management • XML configuration of data sources and views • Query hosts in CMDB via SQL • Sync hosts, groups, templates and trigger dependencies via Zabbix APi
  15. 15. Commercial in Confidence | 16 SNMP Template Generator • Parses MIB files • Generates Zabbix Template XML files • https://github.com/cavaliercoder/mib2zabbix $ mib2zabbix.pl --enable-items --oid 1.3.6.1.2.1.2.2 > template.xml
  16. 16. Commercial in Confidence | 17 > Export-CounterSetToZabbixTemplate -CounterSet LogicalDisk,PhysicalDisk | Out-File template.xml PDH Template Demonstration • PowerShell Module • Query performance counter API • Generates Zabbix template XML • https://github.com/cavaliercoder/ZabbixTemplates
  17. 17. Commercial in Confidence | 18 Agent Stress Test • Multi-threaded Go binary • Queries the Zabbix Agent • Test for performance and stability • https://github.com/cavaliercoder/zabbix_agent_bench
  18. 18. Commercial in Confidence | 19 Modules, why bother? • Better performance • Simpler packaging • Smaller footprint
  19. 19. # test built-in item $ zabbix_agent_bench -key agent.ping # test simple fork $ zabbix_agent_bench -key "system.run[/bin/echo 1]" # test simple script $ zabbix_agent_bench -key "system.run[/usr/bin/python –c 'print 1']" # test module item $ zabbix_agent_bench -key dummy.ping # regression test multiple keys for CI/CD $ zabbix_agent_bench –iterations 1 –strict –keys appkeys.conf
  20. 20. Commercial in Confidence | 21 References Kinetic IT https://www.kineticit.com.au/ ZabbixTemplates - Powershell module https://github.com/cavaliercoder/ZabbixTemplates mib2zabbix – SNMP Template generator https://github.com/cavaliercoder/mib2zabbix zabbix_agent_bench – Agent benchmark tool https://github.com/cavaliercoder/zabbix_agent_bench WiX MSI template for Zabbix Windows Agent https://github.com/cavaliercoder/zabbix-msi PostgreSQL agent module https://github.com/cavaliercoder/libzbxpgsql
  21. 21. Many thanks

×