Mike Brittain presented on web performance culture and tools at Etsy. He discussed how Etsy focuses on performance to improve business metrics like conversion rates and page views. Engineers use tools like logging, Logster, Graphite, StatsD, and custom dashboards to measure performance. They have processes for continuous deployment, data-driven development and prioritizing optimizations. The tools and focus on measurement help Etsy engineers improve site stability and user experience.

1. Web Performance
Culture and Tools
at Etsy
Mike Brittain
Dir. of Engineering, Infrastructure
Etsy
Boston Web Performance Meetup
Aug 24, 2011

2. Overview
Etsy and Engineering
Make Performance
Matter
Tools and Process

4. http://qacreate.etsy.com

5. Monthly
Weather Report
$38 MM in sales
1.9 MM items sold
http://etsy.me/weather-report-june-2011

6. Monthly
Weather Report
990 MM page views
http://etsy.me/weather-report-june-2011

7. Engineering
Technologies
Linux, Apache, MySQL,
PHP, Memcache
Solr, Squid, Hadoop, Amazon S3, EC2, EMR
Ganglia, Cacti, Nagios, Graphite, Splunk, and
some of our own...

8. Engineering
Teams
90 Engineers
3-6 engineers
+ product developer and/or designer

9. Engineering
Continuous
Deployment
~40 releases per day
inc. app code and config changes

10. 1. Write code
2. Code review
3. Automated tests

11. 1. Write code
2. Code review
3. Automated tests
4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod

12. 1. Write code
2. Code review
3. Automated tests
4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod
5. Monitor!
6. Monitor!
7. Monitor!

13. Engineering
Data-Driven
Development
45,000+ metrics
50+ dashboards

14. Make
Performance
Matter
Have a story

15. Business Impact
Measure
“Our own years of testing have conclusively
shown that when speed of a feature or
product improves, usage, quite simply,
goes up.”
~Google
Abundance of research from Google, Bing,
AOL, Amazon, Shopzilla, etc.
http://googleblog.blogspot.com/2009/12/this-week-in-search-121809.html

16. Business Impact
Measure
Bounce rate
Search conversion

17. Business Impact
Measure
Purchase funnel
Ad impressions, page views and
tracking discrepancies
Social, engagement

18. Operations
Site Stability
Measure perf for discrete pieces
of your infrastructure
Contention for shared resources like
database, memcache, solr, or even web
server processes

19. Morale
Happy Engineers

20. Getting to work
Where to
Start?
Focus your efforts where it makes
sense for your business

21. Getting to work
Where to
Optimize?
Tiers of service
SLAs

22. Getting to work
Focus
Beware of CTS:
Constant Tweaking Syndrome

23. Getting to work
Focus
Friends don’t let friends
tweak without graphing

24. Make
Performance
Matter
Have your story

25. Tools and
Process at
Etsy

26. Process
Etsy’s
Perf Team
Standardize patterns
Create tools and reports
Coordinate efforts

27. Process
Server-side
Performance
There is nothing worse
than a blank page or a spinning
thing-a-ma-jigger

28. Process
Server-side
Performance
95th Percentile
> 800 ms is a P2 bug

29. Process
Server-side
Performance

30. Tools
Logging
Perf-related data belongs in
your server logs
apache_note()

31. Tools
Logging
$timer_start = microtime(true);
...
$timer_diff = microtime(true) - $timer_start;
register_shutdown_function()
apache_note('php_microsec', $timer_diff);

32. Tools
Logging
LogFormat "%{True-Client-IP}i %l %u %t "%r"
%>s %b "%{Referer}i" "%{User-Agent}i"
%{etsy_user_id}n %{php_bytes}n %{php_microsec}n
%D" combined

33. Tools
Logging
LogFormat "%{True-Client-IP}i %l %u %t "%r"
%>s %b "%{Referer}i" "%{User-Agent}i"
%{etsy_user_id}n %{php_bytes}n %{php_microsec}n
%D" combined

34. Tools
Logging
LogFormat "%{True-Client-IP}i %l %u %t "%r"
%>s %b "%{Referer}i" "%{User-Agent}i"
%{etsy_user_id}n %{php_bytes}n %{php_microsec}n
%D" combined

35. Tools
Logging
web0060 66.249.71.110 - - [24/Aug/2011:04:16:52
+0000] "GET /listing/12189259/tropical-etched-
pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://
www.google.com/bot.html)" - 13399576 505780
554876

36. Tools
Analyze
grep ... access.log | awk ...
grep "/listing/" access.log |
awk '{sum=sum+$(NF-1)} END {print sum/NR}'
grep "/listing/" access.log |
awk 'BEGIN {max=0} {if ($(NF-1)>max) max=$(NF-1)}
END {print max}'

37. Tools
Analyze

38. Tools
Analyze
1. Capture perf data in logs
page gen., Boomerang, Gomez
2. Aggregate
Splunk, Logster, StatsD
3. Record in Graphite
http://graphite.wikidot.com

39. Tools
Analyze

40. Tools
Logster
1. Read new log entries every
minute
2. Parse and aggregate useful
numbers
3. Send to Graphite
http://github.com/etsy

41. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllpp
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a g
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a g
web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.
web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.
web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.
web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.
web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo

42. Fatals Errors Warnings

43. :)

44. Tools
StatsD
Collects metrics from your app
code and sticks them in Graphite
StatsD::increment("logins.success");
StatsD::timing("gearman.time", $msec);

45. Tools
StatsD
90th pct
average
lower

46. :) :)

47. Tools
Perf Dashboard

48. Tools
I/O Profiler
Lightweight, inline profiling
Start and end times wrapped
around service calls
databases, memcache, apc, etc.

49. Tools
I/O Profiler

50. Tools
I/O Profiler

51. Tools
Client-side
Testing
Gomez (API), Boomerang,
WebPagetest

52. Tools
ShowSlow
Tracks front-end best practices

53. Tools
ShowSlow
Internal instance
Automated testing
End-point for I/O Profiler
Trending on individual rules
http://showslow.com

54. Other thoughts
YSlow &
Page Speed
Mind your development process
3rd-party content and ga.js
“Use a CDN”

55. Other thoughts
Device-Specific
Design
The mobile web is very much
about designing for performance
Screen size, pixel density, connection speed,
wi-fi vs. cellular, browser cache size, local
storage, connections per host, metered
pricing, etc.

56. Make
Performance
Matter
Have your story
Use tools and process
to focus your efforts

57. We are
Hiring
http://etsy.com/careers
Software Engineering positions available in
a number of teams, including Analytics,
Operations, Web Performance, Payments,
Core Platform, Front-End, Internal Apps,
Search, Security, and more...

58. Thank
You
Mike Brittain
mike@etsy.com
@mikebrittain
CodeAsCraft.etsy.com
github.com/etsy
graphite.wikidot.com
showslow.com