Icinga Camp Berlin 2018 - Automated Configuration with Icinga Director

#IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
AUTOMATED MONITORING
IN HETEROGENEOUS ENVIRONMENTS

THOMAS GELF
● Principal Consultant @netways
● Lead Architect @icinga
SELF-INTRODUCTION

HI BERLIN!
WHO IS…?

DISCLAIMER
NOT EVERYTHING SHOULD BE TAKEN LITERALLY.

AUTOMATION
IMPORT → SYNC → DEPLOYMENT. JOBS.

AUTOMATION
OVERVIEW

AUTOMATION
● Any Data Source. Shipped with Director or from a Module
● Fills "deduplicted" tables, isolated from others
● Doesn‘t disturb other tasks and daily business
● Therefore slower sources are no problem at all
● Import is atomic -> all or nothing
IMPORT

AUTOMATION
● Decoupled from Import
● Builds old and new objects in Memory
● Writes Diff to Database
● Single Transaction - all or nothing, once again
● But: Activity Log is decoupled for performance reasons
SYNC

AUTOMATION
● Jobs run as background services
● Configurable independently
● No defined order
● Jobs can trigger Import, Sync and Deployments
JOBS

SOME NUMBERS
HINT: DIRECTOR v1.0.0 HAS BEEN TAGGED ON MARCH 24th, 2016

SOME NUMBERS
● Active since slightly more than 2 years.
● 150k single Import Runs (currently 3-600 a day)
● 16k Sync Runs (currently about 70 every day)
● 5.7k Deployments (currently 30+ automated ones every day)
● 660k Activity Log entries
● 93k Services, 3.3k Hosts, average check interval 1min
FROM A PRODUCTIVE ENVIRONMENT, YESTERDAY

SCENARIO 1
IDO SYNC

IDO SYNC
● Running in parallel during Migration
● Pretty often: Icinga 1 is running, Icinga 2 being prepared
● Comfortable Option: Import from the IDO
● Preferrably only Hosts with attributes
● Groups and memberships
● Build Service checks from scratch to benefit from Icinga 2
TASK / CHALLENGE

IDO SYNC
● Important for IDO: always check icinga_objects.is_active
A FIRST QUERY

IDO SYNC
ADD CUSTOM VARS TO THE MIX

AUTOMATION
EVEN MORE CUSTOM VARS?

AUTOMATON
● Emty Strings
● Invalid values
CLEAN UP DIRTY DATA

CONCLUSION
NICE. WHAT ELSE SPEAKS SQL? HEY, THERE IS A CMDB!

SCENARIO 2
DATA CLEANUP

WE HAVE A CMDB
IMPORT IS RUNNING. BUT.

INSIGHT
OUR CMDB DATA IS GARBAGE

CHALLENGES
● Location
● Nürnberg
● Nuernberg
● Nuremberg
● Hostname:
● spooler
● PRINTSRV
● FILESERVER.example.com
● localhost
● dbserver.example.com
OUR CMDB DATA IS GARBAGE

DECISION
NOT WITH US, WE‘LL VALIDATE ALL THE VALUES

CONCLUSION
BEING STRICT DOESN‘T WORK. SOCIAL ENGINEERING DIDN‘T HELP EITHER.

CONCLUSION
● Some people are slower
● It always worked that way
● Others are not willing to do extra work
● Those who want to work have insufficient permissions
THIS DOESN‘T WORK

DECISION
WE‘LL FIX IT ON OUR OWN. LET‘S MAP THAT DIRTY DATA!

● We can still be strict
● New variant?
● New list entry!

CONCLUSION
KEEPING THAT MAP UP TO DATE IS NO FUN

CONCLUSION
● That‘s a job for the apprentice
● Director provides no "Access to a single list" restriction
● The apprentice shouldn‘t be granted other permissions
KEEPING THAT MAP UP TO DATE IS NO FUN

DECISION
GIVE HIM A CSV-FILE, WE‘RE USING IT AS AN IMPORT SOURCE

CONCLUSION
WHEN SOMETHING GOES WRONG WE HAVE TO TELL THE APPRENTICE

CONCLUSION
● We are not willing to care
● This shouldn‘t bother us
● It‘s time for a Health-Check
WHEN SOMETHING GOES WRONG WE HAVE TO TELL THE APPRENTICE

CONCLUSION
● Notifications?
● Directly to the apprentice!
...WE HAVE TO TELL THE APPRENTICE

INSIGHT
OUR APPRENTICE IS SUPER INTELLIGENT

INSIGHT
● He makes the most creative excuses
● „My dog ate the Notification“
● Automation can be blocked for a day or so
● Physical violence could improve his motivation
● HR-people don‘t want us to practice that
OUR APPRENTICE IS SUPER INTELLIGENT

CONCLUSION
WE NEED TO ESCALATE IN TIME

SCENARIO 3
MONITOR ALL THE THINGS.

MONITOR ALL THE THINGS
● Disclaimer: I do not consider this being useful at all
● Some people continue to claim this being an essential feature
● So let me show you that we could.
● If we would.
NOT A REAL CHALLENGE

MONITOR ALL...
● git clone
● module enable
● Configure an
Import Source

MONITOR ALL...
● ...that‘s it.

CONCLUSION
THIS RARELY MAKES ANY SENSE. ANYWAYS, WE CAN.

SCENARIO 4
CONTACT WITH THE DARK SIDE

WINDOWS MONITORING
● We were successful. Success sucks.
● Have been told to also monitor our Windows Servers
● Not in the main CMDB, they are running their own one
● Their CMDB is not available, ongoing migration project
TASK / CHALLENGE

INSIGHT
WINDOWS SERVERS ARE TO BE FOUND IN THE ACTIVE DIRECTORY

USE PROPERTY MODIFIERS
SOME HAVE BEEN BUILT FOR ACTIVE DIRECTORY – USE THEM

BITMASK MATCH
A SPECIAL PROPERTY MODIFIER

SCENARIO 5
EVERYTHING IS GOING TO BE VIRTUALIZED

VIRTUALIZE ALL THE THINGS
● All Servers are going to be virtualized
● On our metal there is running only ESX
TASK / CHALLENGE

● https://github.com/Icinga/icingaweb2-module-vsphere
● Enable the module
● Configure an Import Source
TASK / CHALLENGE

● Import works fine
● Lots of single checks are hammering our VCenter
● Checks based on SDK eat a lot of memory
● Many of them in parallel waste lots of resources
● It‘s a little bit boring and annoying.
CONCLUSION

CONCLUSION
THIS MUST BECOME MORE FANCY.

https://github.com/Thomas-Gelf/icingaweb2-module-vspheredb
It‘s done when it is done.
But as you could see, we are very very close.
TASK / CHALLENGE

INSIGHT
THAT‘S AMAZING. BUT THERE IS STILL HARDWARE. THAT‘S SOOOO 90‘s

SCENARIO 6
LET‘S MOVE EVERYTHING INTO THE CLOUD

CONCLUSION
EVERYBODY IS DOING AWS – WE MUST FOLLOW THEM

WE ARE IN THE CLOUD
● https://github.com/Icinga/icingaweb2-module-aws
● Enable the module
● Define an Import Source
IMPORT FROM AWS

● You could import all your
instances, but this rarely
makes any sense
● Focus on your applications
● Autoscaling Groups are
usually a good match

INSIGHT
THEY GOT ALL OUR DATA. WHO EXPECTED THAT?

CONCLUSION
LET‘S DO CONTAINERS. EVERYBODY DOES.

WE ARE CONTAINER
● Marathon/Mesos
https://github.com/b0e/icingaweb2-module-marathon
● Proxmox
https://github.com/nbuchwitz/icingaweb2-module-pve
● Kubernetes?
IMPORT FROM...

INSIGHT
IT‘S EASY. DON‘T BE SCARED. THINK APPLICATIONS.

SCENARIO X
BEYOND THE CLOUD

SCENARIO X
● Everybody is doing „a little bit of Cloud“
● Life in the real world out there is rough
● Time-tested configuration management systems
● Kind of standard in Enterprise environments
● Let‘s have a look at the related OSI Model
BEYOND THE CLOUD

CONFIGURATION MANAGEMENT
YOU GROW ALONG WITH YOUR TASKS

THANK YOU!

QUESTIONS?
THANK YOU!

Icinga Camp Berlin 2018 - Automated Configuration with Icinga Director

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (10)

Mehr von Icinga

Mehr von Icinga (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Icinga Camp Berlin 2018 - Automated Configuration with Icinga Director