Conference given on June 13th 2013 in Amsterdam

1. ;
Monitoring at Cloud ScaleMonitoring at Cloud Scale
Julien PivottoJulien Pivotto
Build a cloud Day AmsterdamBuild a cloud Day Amsterdam
June 13th, 2013June 13th, 2013

2. ;
Introduction
Around monitoring
Tools
Conclusion
Table of contents
1 Introduction
DevOps
monitoringsucks
monitoringsucks
2 Around monitoring
The cloud
The past
Environment
Challenges
Infrastructure as code
3 Tools
Collectd
Logstash
Statsd
Graphite
Icinga
4 Conclusion
They work together
Sharing
There are solutions
Julien Pivotto Monitoring at Cloud Scale

3. ;
Introduction
Around monitoring
Tools
Conclusion
Julien PivottoJulien Pivotto
• sysadmin @ inuitssysadmin @ inuits
• open-source defender for 7+ yearsopen-source defender for 7+ years
• devops believerdevops believer
• @roidelapluie on twitter/github@roidelapluie on twitter/github
Julien Pivotto Monitoring at Cloud Scale

4. ;
Introduction
Around monitoring
Tools
Conclusion
DevOps
monitoringsucks
monitoringsucks
What is that DevOps stuff again?
• CultureCulture
• (Lean)(Lean)
• AutomationAutomation
• MeasurementMeasurement
• SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Julien Pivotto Monitoring at Cloud Scale

5. ;
Introduction
Around monitoring
Tools
Conclusion
DevOps
monitoringsucks
monitoringsucks
#monitoringsucks
• https://github.com/monitoringsuckshttps://github.com/monitoringsucks
• a movement to find a solution to monitoringa movement to find a solution to monitoring
• the feeling that monitoring is stucked in the pastthe feeling that monitoring is stucked in the past
Julien Pivotto Monitoring at Cloud Scale

6. ;
Introduction
Around monitoring
Tools
Conclusion
DevOps
monitoringsucks
monitoringsucks
#monitoringlove
• then it turned into #monitoringlovethen it turned into #monitoringlove
• relevant tools existrelevant tools exist
• they just need to be usedthey just need to be used
• following the unix philosophyfollowing the unix philosophy
we are going to explore some of themwe are going to explore some of them
Julien Pivotto Monitoring at Cloud Scale

7. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
What is different in the cloud?
• ScaleScale
• VelocityVelocity
• More changes, more oftenMore changes, more often
Julien Pivotto Monitoring at Cloud Scale

8. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
What do you need?
• scalabilityscalability
• automationautomation
Julien Pivotto Monitoring at Cloud Scale

9. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
time for retirement
• forget all-in-one toolsforget all-in-one tools
• forget auto-discovery toolsforget auto-discovery tools
• forget non-scalable toolsforget non-scalable tools
• forget tools you can not automateforget tools you can not automate
Julien Pivotto Monitoring at Cloud Scale

10. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
ZabbixZabbix
Julien Pivotto Monitoring at Cloud Scale

11. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
CentreonCentreon
Julien Pivotto Monitoring at Cloud Scale

12. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
GroundWorkGroundWork
Julien Pivotto Monitoring at Cloud Scale

13. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
CactiCacti
Julien Pivotto Monitoring at Cloud Scale

14. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
HypericHyperic
Julien Pivotto Monitoring at Cloud Scale

15. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
BigBrotherBigBrother
Julien Pivotto Monitoring at Cloud Scale

16. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
MuninMunin
Julien Pivotto Monitoring at Cloud Scale

17. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
forget about. . .
http://www.flickr.com/photos/mourner/150844753/
ZenossZenoss
Julien Pivotto Monitoring at Cloud Scale

18. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
Your infrastructure today
http://www.flickr.com/photos/bjbrake/235217140/
Julien Pivotto Monitoring at Cloud Scale

19. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
Your infrastructure tomorrow
http://www.flickr.com/photos/bjbrake/235217140/
Julien Pivotto Monitoring at Cloud Scale

20. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
Your infrastructure in 6 months
http://www.flickr.com/photos/bjbrake/235217140/
Julien Pivotto Monitoring at Cloud Scale

21. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
Challenges
• ReproducabilityReproducability
• SpeedSpeed
• MetricsMetrics
• OrchestrationOrchestration
Julien Pivotto Monitoring at Cloud Scale

22. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
WANTED
http://www.flickr.com/photos/pagedooley/3124443099/
• Small toolsSmall tools
• Collect / MangleCollect / Mangle
• Analyse / ActAnalyse / Act
• VisualizeVisualize
Julien Pivotto Monitoring at Cloud Scale

23. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
WANTED
The UNIX philosophyThe UNIX philosophy
Julien Pivotto Monitoring at Cloud Scale

24. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
Automation
• One source of trust: puppet, chef, . . .One source of trust: puppet, chef, . . .
• Exported resourceExported resource
• Monitor in the same location you deployMonitor in the same location you deploy
• Infrastructure-as-CodeInfrastructure-as-Code
• no autodiscovery toolsno autodiscovery tools
Julien Pivotto Monitoring at Cloud Scale

25. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
Automation
If it is not automated || not monitoredIf it is not automated || not monitored
then it does not exist!then it does not exist!
Julien Pivotto Monitoring at Cloud Scale

26. ;
Introduction
Around monitoring
Tools
Conclusion
The cloud
The past
Environment
Challenges
Infrastructure as code
Example in puppet
• Create a definition for your applicationCreate a definition for your application
• In that definition, add the configuration, the vhosts. . .In that definition, add the configuration, the vhosts. . .
• Then export the monitoring (@@icinga_service)Then export the monitoring (@@icinga_service)
• In bonus you can export DB configuration, etc. . .In bonus you can export DB configuration, etc. . .
• Use only the "meta" definitionUse only the "meta" definition
• Collect the exported ressources (Nagios_service <||>)Collect the exported ressources (Nagios_service <||>)
Julien Pivotto Monitoring at Cloud Scale

27. ;
Introduction
Around monitoring
Tools
Conclusion
Collectd
Logstash
Statsd
Graphite
Icinga
Collectd
• Statistics collection daemonStatistics collection daemon
• A lot of plugins available. . .A lot of plugins available. . .
• Can send data to graphiteCan send data to graphite
• Simple configurationSimple configuration
Julien Pivotto Monitoring at Cloud Scale

28. ;
Introduction
Around monitoring
Tools
Conclusion
Collectd
Logstash
Statsd
Graphite
Icinga
Collectd plugins
http://www.flickr.com/photos/juhansonin/3141561416/
Julien Pivotto Monitoring at Cloud Scale

29. ;
Introduction
Around monitoring
Tools
Conclusion
Collectd
Logstash
Statsd
Graphite
Icinga
Collectd plugins
AMQP Apache APC_UPS Apple_Sensors Ascent Battery BIND CarbonAMQP Apache APC_UPS Apple_Sensors Ascent Battery BIND Carbon
ConnTrack ContextSwitch CPU CPUFreq CSV cURL cURL-JSON cURL-XMLConnTrack ContextSwitch CPU CPUFreq CSV cURL cURL-JSON cURL-XML
DBI DF Disk DNS E-Mail Entropy Exec FileCount FSCache GenericJMXDBI DF Disk DNS E-Mail Entropy Exec FileCount FSCache GenericJMX
gmond HDDTemp Interface IPMI IPTables IPVS IRQ Java libvirt Loadgmond HDDTemp Interface IPMI IPTables IPVS IRQ Java libvirt Load
LogFile LPAR MadWifi MBMon memcachec memcached Memory ModbusLogFile LPAR MadWifi MBMon memcachec memcached Memory Modbus
Monitorus Multimeter MySQL NetApp Netlink Network NFS nginxMonitorus Multimeter MySQL NetApp Netlink Network NFS nginx
Notify_Desktop Notify_Email NTPd NUT olsrd OneWire OpenVPN OpenVZNotify_Desktop Notify_Email NTPd NUT olsrd OneWire OpenVPN OpenVZ
Oracle Perl Pinba Ping PostgreSQL PowerDNS Processes Protocols PythonOracle Perl Pinba Ping PostgreSQL PowerDNS Processes Protocols Python
Redis RouterOS RRDCacheD RRDtool Sensors Serial SNMP Swap SysLogRedis RouterOS RRDCacheD RRDtool Sensors Serial SNMP Swap SysLog
Table Tail Tape TCPConns TeamSpeak2 TED thermal TokyoTyrant UnixSockTable Tail Tape TCPConns TeamSpeak2 TED thermal TokyoTyrant UnixSock
Uptime Users UUID Varnish vmem VServer Wireless XMMSUptime Users UUID Varnish vmem VServer Wireless XMMS
Write_Graphite Write_HTTP Write_MongoDBWrite_Graphite Write_HTTP Write_MongoDB
Write_Redis Write_Riemann ZFS_ARCWrite_Redis Write_Riemann ZFS_ARC
Julien Pivotto Monitoring at Cloud Scale

30. ;
Introduction
Around monitoring
Tools
Conclusion
Collectd
Logstash
Statsd
Graphite
Icinga
Logstash
• Ship logs from any sourceShip logs from any source
• Filter themFilter them
• Index themIndex them
• Search themSearch them
• Backed with elasticsearchBacked with elasticsearch
Julien Pivotto Monitoring at Cloud Scale

31. ;
Introduction
Around monitoring
Tools
Conclusion
Collectd
Logstash
Statsd
Graphite
Icinga
Kibana
http://kibana.org/images/screenshots/searchss.png
Julien Pivotto Monitoring at Cloud Scale

32. ;
Introduction
Around monitoring
Tools
Conclusion
Collectd
Logstash
Statsd
Graphite
Icinga
Statsd
• Stats aggregationStats aggregation
• Simple countersSimple counters
• Flushes every XX seconds to graphiteFlushes every XX seconds to graphite
• Text over UDPText over UDP
Julien Pivotto Monitoring at Cloud Scale

33. ;
Introduction
Around monitoring
Tools
Conclusion
Collectd
Logstash
Statsd
Graphite
Icinga
Statsd
echo "stats.sshd.login:1|c" | nc -u statsd.example.com 8125echo "stats.sshd.login:1|c" | nc -u statsd.example.com 8125
Julien Pivotto Monitoring at Cloud Scale

34. ;
Introduction
Around monitoring
Tools
Conclusion
Collectd
Logstash
Statsd
Graphite
Icinga
Graphite
• Graphing made simpleGraphing made simple
• A lot of helpers functionsA lot of helpers functions
• Listening on UDP and TCPListening on UDP and TCP
• Text over UDP/TCPText over UDP/TCP
Julien Pivotto Monitoring at Cloud Scale

35. ;
Introduction
Around monitoring
Tools
Conclusion
Collectd
Logstash
Statsd
Graphite
Icinga
Send data to graphite
echo "stats.sshd.login 1 $(date +%s)" | nc -u graphite.example.com 2003echo "stats.sshd.login 1 $(date +%s)" | nc -u graphite.example.com 2003
Julien Pivotto Monitoring at Cloud Scale

36. ;
Introduction
Around monitoring
Tools
Conclusion
Collectd
Logstash
Statsd
Graphite
Icinga
Graphite API
Julien Pivotto Monitoring at Cloud Scale

37. ;
Introduction
Around monitoring
Tools
Conclusion
Collectd
Logstash
Statsd
Graphite
Icinga
gdash
https://github.com/ripienaar/gdash
Julien Pivotto Monitoring at Cloud Scale

38. ;
Introduction
Around monitoring
Tools
Conclusion
Collectd
Logstash
Statsd
Graphite
Icinga
Icinga
• Fork of nagiosFork of nagios
• Large and vibrant communityLarge and vibrant community
• Configuration compatible with nagiosConfiguration compatible with nagios
• User-friendly interfaceUser-friendly interface
• Use Icinga Classic!Use Icinga Classic!
Julien Pivotto Monitoring at Cloud Scale

39. ;
Introduction
Around monitoring
Tools
Conclusion
Collectd
Logstash
Statsd
Graphite
Icinga
Icinga
https://icinga.org
Julien Pivotto Monitoring at Cloud Scale

40. ;
Introduction
Around monitoring
Tools
Conclusion
They work together
Sharing
There are solutions
Toolchain from apache to nagios
• Apache ships logs to rsyslogApache ships logs to rsyslog
• Rsyslog ships logs to logstashRsyslog ships logs to logstash
• Logstash ships metrics to statsdLogstash ships metrics to statsd
• Statsd ships metrics to GraphiteStatsd ships metrics to Graphite
• Icinga query metric from graphiteIcinga query metric from graphite
• https://github.com/etsy/nagios_toolshttps://github.com/etsy/nagios_tools
Julien Pivotto Monitoring at Cloud Scale

41. ;
Introduction
Around monitoring
Tools
Conclusion
They work together
Sharing
There are solutions
Reusing Icinga/Nagios perfdata
• Icinga performs various checksIcinga performs various checks
• Icinga sends perfdata to graphiteIcinga sends perfdata to graphite
• Graphite stores the dataGraphite stores the data
• Gdash serves them inside dashboardsGdash serves them inside dashboards
• https://github.com/roidelapluie/icinga-to-graphitehttps://github.com/roidelapluie/icinga-to-graphite
Julien Pivotto Monitoring at Cloud Scale

42. ;
Introduction
Around monitoring
Tools
Conclusion
They work together
Sharing
There are solutions
The metrics
• Everything can become a metricEverything can become a metric
• Performance metricsPerformance metrics
• Usage metricsUsage metrics
• Business-valuable metricsBusiness-valuable metrics
• People metricsPeople metrics
• Metrics are knowledgeMetrics are knowledge
Julien Pivotto Monitoring at Cloud Scale

43. ;
Introduction
Around monitoring
Tools
Conclusion
They work together
Sharing
There are solutions
Metrics that matter
http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
Julien Pivotto Monitoring at Cloud Scale

44. ;
Introduction
Around monitoring
Tools
Conclusion
They work together
Sharing
There are solutions
What have we seen?
• We have seen only open-source softwareWe have seen only open-source software
• Small, pluggable daemonsSmall, pluggable daemons
• Robust solutionsRobust solutions
• Nice & user-friendly outputNice & user-friendly output
• They play togetherThey play together
Julien Pivotto Monitoring at Cloud Scale

45. ;
Introduction
Around monitoring
Tools
Conclusion
They work together
Sharing
There are solutions
Homework
• SensuSensu
• RiemannRiemann
• dashingdashing
• ExtreMonExtreMon
• EsperEsper
• SkylineSkyline
• OculusOculus
Julien Pivotto Monitoring at Cloud Scale

46. ;
Introduction
Around monitoring
Tools
Conclusion
They work together
Sharing
There are solutions
Try them yourself
https://github.com/KrisBuytaert/vagrant-graphitehttps://github.com/KrisBuytaert/vagrant-graphite
https://github.com/KrisBuytaert/vagrant-puppet-logstashhttps://github.com/KrisBuytaert/vagrant-puppet-logstash
Julien Pivotto Monitoring at Cloud Scale

47. ;
Introduction
Around monitoring
Tools
Conclusion
They work together
Sharing
There are solutions
Contact
Julien PivottoJulien Pivotto
julien@inuits.eujulien@inuits.eu
@roidelapluie@roidelapluie
INUITS bvbaINUITS bvba
Duboisstraat 50Duboisstraat 50
2060 Antwerp2060 Antwerp
BelgiumBelgium
+32 473 441 636+32 473 441 636
https://inuits.euhttps://inuits.eu
Julien Pivotto Monitoring at Cloud Scale