The devops approach to monitoring, Open Source and Infrastructure as Code Style

;
The devops approach to monitoring,The devops approach to monitoring,
Open Source and IAC StyleOpen Source and IAC Style
Julien PivottoJulien Pivotto
Open World ForumOpen World Forum
October 4, 2013October 4, 2013

;
whoamiwhoami
• sysadmin @ inuitssysadmin @ inuits
• open-source defender for 7+ yearsopen-source defender for 7+ years
• devops believerdevops believer
• @roidelapluie on twitter/github@roidelapluie on twitter/github
Julien Pivotto DevOps and monitoring

;
DevOpsDevOps
• CultureCulture
• AutomationAutomation
• MeasurementMeasurement
• SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis

;
Monitoring is usuallyMonitoring is usually
an afterthoughtan afterthought
ENOTIME, ENOBUDGETENOTIME, ENOBUDGET

;
#monitoringsucks#monitoringsucks
• A movement started in 2011A movement started in 2011
• http://github.com/monitoringsuckshttp://github.com/monitoringsucks
• A lot of tools and informationA lot of tools and information

;
GoalsGoals
• Find when a service is unavailableFind when a service is unavailable
• Understand failure post-mortemUnderstand failure post-mortem

;
GoalsGoals
• Find when a service is unavailableFind when a service is unavailable
• Understand failure post-mortemUnderstand failure post-mortem
• Learn from your infrastructureLearn from your infrastructure
• AnticipateAnticipate

;
Monitor everythingMonitor everything
• ServersServers
• ServicesServices
• UsageUsage
• HardwareHardware
• SoftwareSoftware
• PeoplePeople

;
Monitor every environmentMonitor every environment
• See performance changes in devSee performance changes in dev
• Fix them before it hits productionFix them before it hits production

;
MetricMetric
• Time + name + value = metricTime + name + value = metric
• Can be anythingCan be anything

;
EventEvent
• Time + ﬁelds = metricTime + ﬁelds = metric
• Logs become usable dataLogs become usable data
• Can be transformed into metricsCan be transformed into metrics

;
Metrics + eventsMetrics + events
• Overview of your infrastructureOverview of your infrastructure
• Usage and state of the servicesUsage and state of the services
• Combine several metricsCombine several metrics
• Extract business valuesExtract business values

;
AutomationAutomation
• Infrastructure as CodeInfrastructure as Code
• Automate everythingAutomate everything
⇒ One source of truthOne source of truth

;
DeploymentDeployment
• Deﬁnitions of a service includes monitoringDeﬁnitions of a service includes monitoring
• Deployed ⇔ monitoredDeployed ⇔ monitored

;
ToolsTools
• No all-in-one toolNo all-in-one tool
• No autodiscovery toolNo autodiscovery tool
• Text-based conﬁgurationText-based conﬁguration
• ScalableScalable
⇒ The Unix philosphyThe Unix philosphy

;
IcingaIcinga
• Fork of nagiosFork of nagios
• Large and vibrant communityLarge and vibrant community
• Conﬁguration compatible with nagiosConﬁguration compatible with nagios
• User-friendly interfaceUser-friendly interface
• Use Icinga Classic!Use Icinga Classic!

;
IcingaIcinga
https://icinga.org

;
SensuSensu
• FlexibilityFlexibility
• Compatible with nagios pluginsCompatible with nagios plugins
• Connects to your source of trustConnects to your source of trust
• Relies on RabbitMQRelies on RabbitMQ

;
CollectdCollectd
• Statistics collection daemonStatistics collection daemon
• A lot of plugins available. . .A lot of plugins available. . .
• Can send data to graphiteCan send data to graphite
• Simple conﬁgurationSimple conﬁguration

;
Collectd pluginsCollectd plugins
http://www.ﬂickr.com/photos/juhansonin/3141561416/

;
Collectd pluginsCollectd plugins
AMQP Apache APC_UPS Apple_Sensors Ascent Battery BIND CarbonAMQP Apache APC_UPS Apple_Sensors Ascent Battery BIND Carbon
ConnTrack ContextSwitch CPU CPUFreq CSV cURL cURL-JSON cURL-XMLConnTrack ContextSwitch CPU CPUFreq CSV cURL cURL-JSON cURL-XML
DBI DF Disk DNS E-Mail Entropy Exec FileCount FSCache GenericJMXDBI DF Disk DNS E-Mail Entropy Exec FileCount FSCache GenericJMX
gmond HDDTemp Interface IPMI IPTables IPVS IRQ Java libvirt Loadgmond HDDTemp Interface IPMI IPTables IPVS IRQ Java libvirt Load
LogFile LPAR MadWiﬁ MBMon memcachec memcached Memory ModbusLogFile LPAR MadWiﬁ MBMon memcachec memcached Memory Modbus
Monitorus Multimeter MySQL NetApp Netlink Network NFS nginxMonitorus Multimeter MySQL NetApp Netlink Network NFS nginx
Notify_Desktop Notify_Email NTPd NUT olsrd OneWire OpenVPN OpenVZNotify_Desktop Notify_Email NTPd NUT olsrd OneWire OpenVPN OpenVZ
Oracle Perl Pinba Ping PostgreSQL PowerDNS Processes Protocols PythonOracle Perl Pinba Ping PostgreSQL PowerDNS Processes Protocols Python
Redis RouterOS RRDCacheD RRDtool Sensors Serial SNMP Swap SysLogRedis RouterOS RRDCacheD RRDtool Sensors Serial SNMP Swap SysLog
Table Tail Tape TCPConns TeamSpeak2 TED thermal TokyoTyrant UnixSockTable Tail Tape TCPConns TeamSpeak2 TED thermal TokyoTyrant UnixSock
Uptime Users UUID Varnish vmem VServer Wireless XMMSUptime Users UUID Varnish vmem VServer Wireless XMMS
Write_Graphite Write_HTTP Write_MongoDBWrite_Graphite Write_HTTP Write_MongoDB
Write_Redis Write_Riemann ZFS_ARCWrite_Redis Write_Riemann ZFS_ARC

;
LogstashLogstash
• Ship logs from any sourceShip logs from any source
• Filter themFilter them
• Index themIndex them
• Search themSearch them
• Backed with elasticsearchBacked with elasticsearch

;
LogstashLogstash

;
GraphiteGraphite
• GraphingGraphing
• Accept any metricAccept any metric
• Store data in ﬁles (whisper)Store data in ﬁles (whisper)
• A lot of helpers functionsA lot of helpers functions
• Listen on UDP and TCPListen on UDP and TCP

;
Send data to graphiteSend data to graphite
echo "stats.sshd.login 1 $(date +%s)" | nc -u graphite.example.com 2003echo "stats.sshd.login 1 $(date +%s)" | nc -u graphite.example.com 2003

;
Graphite APIGraphite API

;
StatsdStatsd
• Graphite friendGraphite friend
• Stats aggregationStats aggregation
• Simple countersSimple counters
• Flushes every XX seconds to graphiteFlushes every XX seconds to graphite
• UDPUDP

;
Feeding statsdFeeding statsd
echo "stats.sshd.login:1|c" |
nc -u statsd.example.com 8125

;
gdashgdash
https://github.com/ripienaar/gdash

;
giraﬀegiraﬀe
Alternative to gdash

;
KibanaKibana
• Kibana is a web interface for Logstash/ESKibana is a web interface for Logstash/ES
• Kibana 1 was written in PHPKibana 1 was written in PHP
• Kibana 2 was written in RubyKibana 2 was written in Ruby
• Kibana 3 is written in AngularJSKibana 3 is written in AngularJS

;
Kibana 3Kibana 3
• Everything happens in the browserEverything happens in the browser
• The browser is connected to ElasticsearchThe browser is connected to Elasticsearch
• You can save dashboards into ESYou can save dashboards into ES
• You can write/template dashboards to ﬁlesYou can write/template dashboards to ﬁles

;
Kibana queriesKibana queries
Example of a kibana query
@fields.syslog_program:"httpd" AND
@fields.http_host:"test.example.com" AND
@fields.response:"404"
• LLucene query syntax
• SSimple and eﬀective
• PPoint & click web interface

;Julien Pivotto DevOps and monitoring

;
Toolchain exampleToolchain example
• Apache ships logs to rsyslogApache ships logs to rsyslog
• Rsyslog ships logs to logstashRsyslog ships logs to logstash
• Logstash ships metrics to statsdLogstash ships metrics to statsd
• Statsd ships metrics to GraphiteStatsd ships metrics to Graphite
• Icinga query metric from graphiteIcinga query metric from graphite
• https://github.com/etsy/nagios_toolshttps://github.com/etsy/nagios_tools

;
Reusing Icinga/Nagios perfdataReusing Icinga/Nagios perfdata
• Icinga performs various checksIcinga performs various checks
• Icinga sends perfdata to graphiteIcinga sends perfdata to graphite
• Graphite stores the dataGraphite stores the data
• Gdash serves them inside dashboardsGdash serves them inside dashboards
• https://github.com/roidelapluie/icinga-to-graphitehttps://github.com/roidelapluie/icinga-to-graphite

;
SharingSharing
• Build dashboard: dashing, teamdashBuild dashboard: dashing, teamdash
• Share with developersShare with developers
• Share with managersShare with managers

;
Try them yourselfTry them yourself
https://github.com/KrisBuytaert/vagrant-graphitehttps://github.com/KrisBuytaert/vagrant-graphite

;
Thank youThank you
Any question?Any question?

;
ContactContact
Julien PivottoJulien Pivotto
julien@inuits.eujulien@inuits.eu
@roidelapluie@roidelapluie
INUITS bvbaINUITS bvba
Duboisstraat 50Duboisstraat 50
2060 Antwerp2060 Antwerp
BelgiumBelgium
+32 473 441 636+32 473 441 636
https://inuits.euhttps://inuits.eu

The devops approach to monitoring, Open Source and Infrastructure as Code Style

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie The devops approach to monitoring, Open Source and Infrastructure as Code Style

Ähnlich wie The devops approach to monitoring, Open Source and Infrastructure as Code Style (20)

Mehr von Julien Pivotto

Mehr von Julien Pivotto (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

The devops approach to monitoring, Open Source and Infrastructure as Code Style