Weitere Ă€hnliche Inhalte KĂŒrzlich hochgeladen (20) Metrics driven engineering (velocity 2011)1. METRICS-DRIVEN
ENGINEERING at
Kellan Elliott-McCrea, VP of Eng.
kellan@etsy.com @kellan
Tuesday, June 5, 12
18. Can we deploy that
to
100% of our visitors?
Tuesday, June 5, 12
20. Did I just break
something?
Tuesday, June 5, 12
21. Q. WHO MAKES THESE
GRAPHS?
A. Well,racksOps team manages thethe
network,
the
the servers, installed
monitoring tools, wears the pagers,
blah, blah, blah...
Tuesday, June 5, 12
25. Yes! No.
Tuesday, June 5, 12
30. Cacti (network, SNMP)
Ganglia (machines)
Graphite (application)
Splunk (log analysis, nightly
reports)
Nagios (alerting)
Tuesday, June 5, 12
31. Gan
â
cluster oriented
â
huge community contributed
recipes
â
2.0 released today (including
several Flickr and Etsy patches!)
â
gmetad makes it easy to track
custom metrics
Tuesday, June 5, 12
33. Graphite
â
super ïŹexible collection and
display
â
per metrics buckets
â
single instance
â
super easy to write and use
custom display functions
Tuesday, June 5, 12
36. web0054 [Fri Mar 04 16:27:48
2011] [error] [login] [14531658]
User login failed. Reason: wrong
password for ...
Tuesday, June 5, 12
37. web0054 [Fri Mar 04 16:27:48
2011] [error] [login] [14531658]
User login failed. Reason: wrong
password for ...
Tuesday, June 5, 12
38. web0054 [Fri Mar 04 16:27:48
2011] [error] [login] [14531658]
User login failed. Reason: wrong
password for ...
Tuesday, June 5, 12
39. web0054 [Fri Mar 04 16:27:48
2011] [info] [login] [14531658]
User login failed. Reason: wrong
password for ...
Tuesday, June 5, 12
40. web0054 [Fri Mar 04 16:27:48
2011] [info] [login] [14531658]
User login failed. Reason: wrong
password for ...
Tuesday, June 5, 12
41. web0054 [Fri Mar 04 16:27:48
2011] [info] [login] [14531658]
User login failed. Reason: wrong
password for ...
Tuesday, June 5, 12
42. Counting
and Timing
http://code.ïŹickr.com/blog/
2008/10/27/counting-timing/
Tuesday, June 5, 12
44. Logster
https://github.com/etsy/logster
Tuesday, June 5, 12
46. web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppppp!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a grue.
web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.
web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.
web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.
web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0003 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a grue.
web0066 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling
Tuesday, June 5, 12
47. Fatals Errors Warnings
Tuesday, June 5, 12
48. â
runs out of cron
â
maintains a cursor into log ïŹles
â
supports ganglia and graphite
â
custom parsers much easier to
write then gmetad
Tuesday, June 5, 12
51. LogFormat "%{X-Forwarded-For}i %
{True-Client-IP}i %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i" %
{etsy_shop_id}n %{etsy_uaid}n %V %
{etsy_ab_selections}n %
{etsy_request_uuid}n %
{etsy_api_consumer_key}n %
{etsy_api_method_name}n %
{php_memory_usage_bytes}n %
{php_time_microsec}n %D" combined
Tuesday, June 5, 12
55. âIf Engineering at Etsy has
a religion, itâs the Church
of Graphs. If it moves, we
track it.â - Erik Kastner
http://codeascraft.etsy.com/2011/02/15/measure-
anything-measure-everything/
Tuesday, June 5, 12
58. StatsD
https://github.com/
etsy/statsd/
Tuesday, June 5, 12
60. 90th pct
average
lower
StatsD::timing("gearman.time", $msec);
Tuesday, June 5, 12
61. Ad hoc
name value timestamp
Tuesday, June 5, 12
65. Trends + Events
target=drawAsInïŹnite(events.deploy.site)
Tuesday, June 5, 12
69. "Aberrant Behavior
Detection in Time Series
for Network Monitoring".
Tuesday, June 5, 12
79. Hard
<a href="http://graphite.etsycorp.com/render?
from=-1hours&width=800&height=600&title=File+or+Script+Not
+Found&yMin=0&target=webs.errorLog.notExist&target=drawAsInïŹnite
%28deploys.conïŹg.production%29&target=drawAsInïŹnite%28deploys.web.production
%29&target=drawAsInïŹnite%28deploys.search.production%29&target=drawAsInïŹnite
%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,
%23ff0000,%23006633,%23cc6600">
<img src="http://graphite.etsycorp.com/render?
from=-1hours&width=280&height=220&title=File+or+Script+Not
+Found&hideLegend=1&yMin=0&target=webs.errorLog.notExist&target=drawAsInïŹnite
%28deploys.conïŹg.production%29&target=drawAsInïŹnite%28deploys.web.production
%29&target=drawAsInïŹnite%28deploys.search.production%29&target=drawAsInïŹnite
%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,
%23ff0000,%23006633,%23cc6600">
</a>
Tuesday, June 5, 12
80. Easy!
$g = new Graphite($time);
$g->setTitle('File Not Found');
$g->addMetric('webs.errorLog.notExist', '#00cc00');
$g->showDeploys(true);
echo $g->getDashboardHTML(280, 220);
Tuesday, June 5, 12
90. codeascraft.etsy.com
github.com/etsy/statsd
github.com/etsy/logster
bitbucket.org/maplebed/ganglia-
logtailer
Tuesday, June 5, 12