Continuous deployment is core to Etsy, and we push public facing code over 30 times per day. Evaluating an experiment amidst this shifting landscape is a difficult task as our traditional methods of monitoring operational metrics don’t provide enough information to make product-level decisions. To this end, we have developed internal tooling for deep analytics that enables us to systematically analyze our experimental results in a continuously changing environment. This talk will focus on the analysis framework that we have built – from the raw logging data, to our elastic mapreduce-based data transformations, to the final dashboards and underlying statistics that drive decision making.

1. Continuous
Experimentation
with Continuous
Deployment
Steve Mardenfeld
Etsy
Monday, November 5, 2012

2. Monday, November 5, 2012

3. World’s Handmade Market
Monday, November 5, 2012

4. Continuous Deployment
Monday, November 5, 2012

5. Monday, November 5, 2012

6. Monday, November 5, 2012

7. Metrics
Monday, November 5, 2012

8. Graph Factory
Monday, November 5, 2012

9. Oops
Monday, November 5, 2012

10. Oops
Monday, November 5, 2012

11. Good Enough?
Monday, November 5, 2012

12. What
is
Better?
Monday, November 5, 2012

13. Sometimes it’s Obvious
Monday, November 5, 2012

14. Most Times it’s Not
Monday, November 5, 2012

15. Most Times it’s Not
Cart Page
Restyled
Monday, November 5, 2012

16. Most Times it’s Not
Etsy Mentioned
on Oprah Cart Page
Restyled
Justin Beiber Internet Goes
Sneezes Down in Brooklyn
New Cat Meme
Monday, November 5, 2012

17. A/B Testing?
Monday, November 5, 2012

18. Monday, November 5, 2012

19. Monday, November 5, 2012

20. Test
Them
Both
Monday, November 5, 2012

21. Statistics?
Monday, November 5, 2012

22. First Pass
HADOOP
Monday, November 5, 2012

23. First Pass
CASCADING
Monday, November 5, 2012

24. First Pass
R
Monday, November 5, 2012

25. First Pass
CASCADING
HADOOP R
Monday, November 5, 2012

26. Automate It
A/B Analyzer
Monday, November 5, 2012

27. A/B Analyzer
•Framework for looking at experiments
•Done for every experiment beforehand
Monday, November 5, 2012

28. A/B Analyzer
Monday, November 5, 2012

29. A/B Analyzer
Monday, November 5, 2012

30. Monday, November 5, 2012

31. 3 Types of Metrics
•Co-occurence
•visits that did X (searched)
•average number of X (searches)
•Funnels
•Special Cases*
Monday, November 5, 2012

32. Co-Occurence
•Co-occurence of all actions with the variant
•i.e. - how many people saw this variant and
then purchased
•i.e. - average number of purchases per variant
Monday, November 5, 2012

33. Funnels
Search Listing Buy
Buy
Monday, November 5, 2012

34. Funnels
Search Listing Buy
Buy
Home Search Listing BuyBuy
Monday, November 5, 2012

35. Funnels
Search Listing Buy
Buy
Home Search Listing Buy
BuyBuy
Monday, November 5, 2012

36. Special Cases
•Exiting pages
•Bounce rate
•Visit length
Monday, November 5, 2012

37. Pipeline
Logs A/B
COLLECTION
Client HDFS Events Visits MySql
PROCESSING
DB
Monday, November 5, 2012

38. Bucketing
Home Search Listing Buy
Search A/B Test
Monday, November 5, 2012

39. Bucketing
Home Search Listing Buy
Search A/B Test
Monday, November 5, 2012

40. Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012

41. Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012

42. Idea
Monday, November 5, 2012

43. Changing Search
Monday, November 5, 2012

44. Related Searches?
Monday, November 5, 2012

45. Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012

46. Be Specific
•What does an improvement look like?
•More clicks?
•Less searches?
•Shorter visits?
•Longer visits?
Monday, November 5, 2012

47. Bet Driven Development
Monday, November 5, 2012

48. Is this even worth it?
Monday, November 5, 2012

49. Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012

50. Code
It
Up
Monday, November 5, 2012

51. Code
Monday, November 5, 2012

52. Turn it On
Monday, November 5, 2012

53. Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012

54. Realtime?
Monday, November 5, 2012

55. Wait
Monday, November 5, 2012

56. Process
•Idea
•Hypothesis
•Code
•Wait
•Analysis
Monday, November 5, 2012

57. Making Sense of Results
Monday, November 5, 2012

58. Back To Related Searches
Monday, November 5, 2012

59. Back To Related Searches
Monday, November 5, 2012

60. Monday, November 5, 2012

61. Monday, November 5, 2012

62. Final Thoughts
•Bet driven development - be explicit about your
changes
•Easy experimental setup and quick analysis
makes for more experiments
•An experiment is successful if you learned
something, not if you launched
Monday, November 5, 2012

63. Monday, November 5, 2012

64. http://www.slideshare.net/powerlexis/retro-slides-from-ibm
Monday, November 5, 2012