1. fast and good
Alternative Approaches to Quality at Etsy
Noah Sussman
STPCon Fall 2011, Dallas TX
#STPCon

3. Our mission is to empower people to change the way
the global economy works. We see a world in which
very-very small businesses have much-much more
sway in shaping the economy, local living economies
are thriving everywhere, and people value authorship
and provenance as much as price and convenience. We
are bringing heart to commerce and making the world
more fair, more sustainable, and more fun.

7. September 2011: 489 Web deployments
110 people committed to trunk
4,519 changesets in all
9.2 changesets average per production deployment

8. September 2011: 489 Web deployments
80 people deployed to production
20 or fewer committers in each deployment
5.7 committers average per deployment

9. 10 million registered members
25 million unique visitors per month
800,000 active sellers
150 different countries
10 million listed items
$314 million in sales in 2010

10. In 2010, MTTR was 6.4 minutes
for severity one outages

11. Six change-related outages in 2010
Out of thousands of production deployments

12. “All existing revision control systems were built by
people who build installed software.”
~Paul Hammond

13. All canonical software life cycles were designed by
people who build installed software

14. “In the online world… there is one consumer of the
software: you. The users are consumers of the
service built atop the software.”
~ Theo Schlossnagle

15. Optimize for recovery
rather than focusing on
preventing failure.
Failure is inevitable.

16. Default to open access

17. No blame

18. Use your own
judgement

19. “an airport without an air traffic controller”
Chad Dickerson

20. faster deploys = smaller changesets = fewer change-related outages

21. Self-serve deployment

22. princess
bleeding edge code,
production data and services

23. Shared development environment
Single dev database
Everyone commits to trunk

24. low-ceremony process

25. A test is a design document first

26. Many small deployments, many small tests

27. Beyond developer testing
1. Test planning
2. GUI-driven regression testing
3. Performance testing
4. Security / Penetration testing
5. Nagios

28. Make it easy to write and run tests

29. Make it painless to use fixtures and mocks

30. Most bugs, most of the time, are easily nailed given even
an incomplete but suggestive characterization of their
error conditions at source-code level.
~ESR, “How Many Eyeballs Tame Complexity,” 1997

31. Debugging is twice as hard
as writing the code in the
first place. Therefore, if you
write the code as cleverly as
possible, you are, by
definition, not smart enough
to debug it.
~ Brian Kernighan

32. Test to address technical debt

33. Keep the feedback loop short

34. Graphmonitorit didn’t happen
At Etsy we
or over 40,000 real-time metrics

35. Business Metrics

36. Application Metrics

37. System Metrics

38. We hire our users

39. Tests are one of the tools we use to
detect and prevent failure

40. There will always be bugs in production

41. Unexpected failures tend to be catastrophic

42. Anomaly detection is hard

43. Just ship

44. How do you deploy large features?

45. Releasing a feature is decoupled from deploying code

46. if ($cfg["new_search"]) {
// new hotness
$resp = search_solr();
} else {
// old busted
$resp = search_grep();
}

47. $cfg = array(
'checkout' => true,
'homepage' => true,
'profiles' => true,
'new_search' => false,
);

48. Dark launches
Admin-only rollouts
A/B campaigns
etsy.com/prototypes
We are always testing

49. A culture of responsibility

50. A community of artisans

51. You are a participant observer

52. etsy.com/careers

53. Noah Sussman
noah@etsy.com
@noahsussman

54. photo credits
penguins: paulmcdee http://www.flickr.com/photos/paulmcdee/2806715569
aerial photograph of snowy road: K. Leidorf http://leidorf.blogspot.com/search/label/20D03855
death in the mirror: Richard Avedon for the New Yorker
Women at work on C-47 Douglas cargo transport http://www.flickr.com/photos/library_of_congress/2179137415/in/photostream/
Woman aircraft worker http://www.flickr.com/photos/library_of_congress/2179930812/
cool bike with wooden handlebars: Bertelli http://www.bertellibici.com/product.php?works_id=112
clover keys: Brian Cerveny http://www.flickr.com/photos/briancerveny/48889960/in/photostream/
Etsy office monitor wall and Etsy bike rack: Mashable's Cubicle Spy http://mashable.com/2010/10/13/etsy-office-pics
horse fly: Lukjonis http://www.flickr.com/photos/38628972@N05/5884111163/in/photostream/
Gandalf vs balrog: the brothers Hildebrandt
Just Ship t-shirt: Kellan http://www.etsy.com/shop/kellanem/sold
barn raising: dogpose http://www.flickr.com/photos/8371074@N05/2865798475/
model train coupling: David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122/sizes/l/in/photostream/
If I used your image and didn't credit you, it's because I couldn't figure out who you were -- email me and I will gladly credit you.
If you object to my using an image to which you own the rights, please contact me and I will remove that image from the deck.