Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Oleg Ivanivskyi - Lessons Learned While Being On-Site | ZabConf2016

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige

Hier ansehen

1 von 56 Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Andere mochten auch (20)

Anzeige

Ähnlich wie Oleg Ivanivskyi - Lessons Learned While Being On-Site | ZabConf2016 (20)

Weitere von Zabbix (10)

Anzeige

Aktuellste (20)

Oleg Ivanivskyi - Lessons Learned While Being On-Site | ZabConf2016

  1. 1. Lessons learned while being on-site
  2. 2. About me
  3. 3. Consultant - (from Latin: consultare "to discuss") is a professional who provides expert advice in a particular area such as IT
  4. 4. Consultant - (from Latin: consultare "to discuss") is a professional who provides expert advice in a particular area such as IT Zabbix consultant = Zabbix expert
  5. 5. Initial Zabbix installation Environment review and performance tuning Migration and upgrade High availability, distributed setup, scalable solution Other projects requiring expertise in Zabbix (e.g. DB partitioning, architecture planing, DB recovery, etc.) Areas:
  6. 6. Graph 8% 10% 14% 32% 24% 12% Initial installation Envonment review and performance tuning Migration and upgrade HA Cluster Partitioning Other
  7. 7. 7 Lessons I Learned
  8. 8. Lesson 1: Expectations prior and during the engagement
  9. 9. Lesson 1: Expectations prior and during the engagement What setup do you need? HA cluster? DB replication? Proxies? Custom templates? Performance tuning?
  10. 10. Lesson 1: Expectations prior and during the engagement What setup do you need? HA cluster? DB replication? Proxies? Custom templates? Performance tuning? It’s important to understand what exactly do you need to ensure the solution reflects it!
  11. 11. Lesson 2: failing to properly document requirements
  12. 12. Example Agenda for a consultancy session: - install Zabbix - configure some devices monitoring - configure reports and notifications
  13. 13. Example Agenda for a consultancy session: - install Zabbix - configure some devices monitoring - configure reports and notifications My estimate: stand-alone installation, adding devices, show available reports, configure email/sms notifications
  14. 14. Example Agenda for a consultancy session: - install Zabbix - configure some devices monitoring - configure reports and notifications My estimate: stand-alone installation, adding devices, show available reports, configure email/sms notifications His estimate: HA cluster for Zabbix, custom templates, push notifications, custom printable reports
  15. 15. Lesson 3: sometimes, even a tiny change can deliver big savings in terms of cost and performance
  16. 16. Lesson 3: sometimes, even a tiny change can deliver big savings in terms of cost and performance problems
  17. 17. Example A customer decided to tune PHP parameters
  18. 18. Example A customer decided to tune PHP parameters Issue: it is not possible to open list of items because of “ERROR: Zabbix has received an incorrect request. Field “itemids” is mandatory”
  19. 19. Example A customer decided to tune PHP parameters Issue: it is not possible to open list of items because of “ERROR: Zabbix has received an incorrect request. Field “itemids” is mandatory” Reason: incorrect value for a PHP parameter
  20. 20. Example A customer decided to tune PHP parameters Issue: it is not possible to open list of items because of “ERROR: Zabbix has received an incorrect request. Field “itemids” is mandatory” Reason: incorrect value for a PHP parameter Solution: use default value for that PHP parameter
  21. 21. Example A customer decided to improve Zabbix agent configuration (use HostnameItem instead of Hostname)
  22. 22. Example A customer decided to improve Zabbix agent configuration (use HostnameItem instead of Hostname) Issue: active agent auto registration generates duplicate hosts
  23. 23. Example A customer decided to improve Zabbix agent configuration (use HostnameItem instead of Hostname) Issue: active agent auto registration generates duplicate hosts Reason: Zabbix agent sends full DNS name instead of short as host name
  24. 24. Example A customer decided to improve Zabbix agent configuration (use HostnameItem instead of Hostname) Issue: active agent auto registration generates duplicate hosts Reason: Zabbix agent sends full DNS name instead of short as host name Solution: custom script and Zabbix API to clean up DB
  25. 25. Lesson 4: most customers don't call for help before the system fails
  26. 26. Lesson 4: most customers don't call for help before the system fails Instead they wait try to fix it themselves enlist the assistance of the local computer geek/staff
  27. 27. Lesson 5: do not overestimate your DBA skills
  28. 28. Example DBA decided to improve Zabbix official schema
  29. 29. Example DBA decided to improve Zabbix official schema Result: Zabbix server restart takes hours because of a slow query
  30. 30. Example DBA decided to improve Zabbix official schema Result: Zabbix server restart takes hours because of a slow query Reason: missing index is the root cause
  31. 31. Example DBA decided to improve Zabbix official schema Result: Zabbix server restart takes hours because of a slow query Reason: missing index is the root cause Solution: add missing index and register new feature request to share your ideas
  32. 32. Example A customer decided to set up partitioning himself
  33. 33. Example A customer decided to set up partitioning himself Result: Zabbix has no partition to write history/trends. Downtime!
  34. 34. Example A customer decided to set up partitioning himself Result: Zabbix has no partition to write history/trends. Downtime! Reason: partitioning script does not work for some reason
  35. 35. Example A customer decided to set up partitioning himself Result: Zabbix has no partition to write history/trends. Downtime! Reason: partitioning script does not work for some reason Solution: make sure you know what you're doing! Test your solution and set up monitoring
  36. 36. Example A customer decided to set up partitioning himself
  37. 37. Example A customer decided to set up partitioning himself Result: some history tables are huge
  38. 38. Example A customer decided to set up partitioning himself Result: some history tables are huge Reason: partitioning is not configured for history_log, history_str and history_log tables. Housekeeper is disabled for them also.
  39. 39. Example A customer decided to set up partitioning himself Result: some history tables are huge Reason: partitioning is not configured for history_log, history_str and history_log tables. Housekeeper is disabled for them also. Solution: configure partitioning for the tables
  40. 40. Note, changes like these are easy to make early on - but very hard to carry out when you need to make database changes in a live environment storing huge amount of data
  41. 41. Lesson 6: do not store trash in Zabbix DB
  42. 42. Example A customer has partitioning configured for Zabbix DB. History storage period is 6 months and 1 year for trends.
  43. 43. Example A customer has partitioning configured for Zabbix DB. History storage period is 6 months and 1 year for trends. Result: DB size is 2.3TB (storage is almost full, DB is slow)
  44. 44. Example A customer has partitioning configured for Zabbix DB. History storage period is 6 months and 1 year for trends. Result: DB size is 2.3TB (storage is almost full, DB is slow) Reason: there were old partitions for 2013 and 2014
  45. 45. Example A customer has partitioning configured for Zabbix DB. History storage period is 6 months and 1 year for trends. Result: DB size is 2.3TB (storage is almost full, DB is slow) Reason: there were old partitions for 2013 and 2014 700GB or 30% of useless data in Zabbix DB!
  46. 46. Example A customer has partitioning configured for Zabbix DB. History storage period is 6 months and 1 year for trends. Result: DB size is 2.3TB (storage is almost full, DB is slow) Reason: there were old partitions for 2013 and 2014 700GB or 30% of useless data in Zabbix DB! Solution: improve partitioning script and delete old partitions
  47. 47. Example A customer has "healthcheck" triggers on 46 hosts
  48. 48. Example A customer has "healthcheck" triggers on 46 hosts Result: 92 triggers generate 55 000 events daily
  49. 49. Example A customer has "healthcheck" triggers on 46 hosts Result: 92 triggers generate 55 000 events daily 1.5M monthly!
  50. 50. Example A customer has "healthcheck" triggers on 46 hosts Result: 92 triggers generate 55 000 events daily 1.5M monthly! Reason: the triggers are too sensitive
  51. 51. Example A customer has "healthcheck" triggers on 46 hosts Result: 92 triggers generate 55 000 events daily 1.5M monthly! Reason: the triggers are too sensitive Solution: use less sensitive triggers/analyze history
  52. 52. Lesson 7: monitoring is crucial - the more coverage you have, the quicker you can identify issues that arise
  53. 53. Use Zabbix to monitor hardware, software and services
  54. 54. Use Zabbix to monitor hardware, software and services Zabbix (App and Team) will help you:
  55. 55. Use Zabbix to monitor hardware, software and services Zabbix (App and Team) will help you: troubleshoot issues even avoid issues
  56. 56. Every day is a school day Thank you

×