Clearspring has several hundred servers across two data centers that receive 3.5 billion events per day from Java, Scala, and PHP applications. To monitor this large infrastructure, they use Nagios to check server health and Ganglia to collect metrics from applications and servers, which are then sent to Graphite for long term storage and visualization. Codahale Metrics libraries are used to instrument the applications to collect different metric types like gauges, counters, histograms and timers to understand performance and detect issues. Graphs of these metrics in Graphite have helped the small operations team at Clearspring to determine what to set alerts on to quickly address problems.
2. The problem?
• Several hundred servers
• 2 data centers (currently)
• Java, Scala, and PHP applications
• 3.5B+ inbound events per day
• And a small (but growing) team to manage it
all….
3. The Solution
Nagios
JVMs
PHP[1] MC[2] Ganglia Graphite
Linux
[1] Metric Catcher: https://github.com/clearspring/MetricCatcher
[2] Phetric: https://github.com/clearspring/Phetric
4. Codahale (Yammer) Metrics[1]
• Gauges – instantaneous measurement
• Counters – count occurrences
• Meters – measure rate over time
• Histograms – statistical distribution
• Timers – rate and distribution of duration
• Health Checks – ummm… it’s a health check
[1] http://metrics.codahale.com/getting-started.html
