Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

A/B Testing at Pinterest: Building a Culture of Experimentation

14.576 Aufrufe

Veröffentlicht am

Presenter: Andrea Burbank, Pinterest

A successful experimentation program consists of much more than mere randomization and measurement. How do you help stakeholders understand the right things to measure, avoid common pitfalls, and learn to rely on A/B tests as the best way to measure a new system or feature? In this talk, Andrea will explain how building a culture of experimentation and the right tools to support it is just as important as the statistics behind the comparisons themselves - and potentially much trickier to get right.

Veröffentlicht in: Technologie
  • Do This Simple 2-Minute Ritual To Loss 1 Pound Of Belly Fat Every 72 Hours ●●● https://tinyurl.com/y6qaaou7
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Doctor's 2-Minute Ritual For Shocking Daily Belly Fat Loss! Watch This Video ★★★ http://ishbv.com/bkfitness3/pdf
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Doctor's 2-Minute Ritual For Shocking Daily Belly Fat Loss! Watch This Video  https://tinyurl.com/bkfitness4u
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

A/B Testing at Pinterest: Building a Culture of Experimentation

  1. 1. @arburbank Building a culture of experimentation scaling data science at Pinterest @arburbank Andrea Burbank
  2. 2. @arburbank σ, μ
  3. 3. @arburbank Organizational maturity model use source control write unit tests track bugs write a spec build often
  4. 4. @arburbank Experimentation maturity model
  5. 5. @arburbank Experimentation maturity model get started get big get better get out get tools
  6. 6. @arburbank Stage 1: get started get started
  7. 7. @arburbank problem: people making bad decisions get started
  8. 8. @arburbank Run experiments entire population control enable d
  9. 9. @arburbank Cultural maturity model get started entire population control enabled data data insight
  10. 10. @arburbank Stage 2: get big get started get big
  11. 11. @arburbank problem: underutilization get started get big
  12. 12. @arburbank http://altmba.com/wp-content/uploads/2015/06/fieldofdreamscorn.jpg
  13. 13. @arburbank http://altmba.com/wp-content/uploads/2015/06/fieldofdreamscorn.jpg
  14. 14. @arburbank if you build it, they won’t come marketing
  15. 15. @arburbank if you build it, they won’t come evangelism
  16. 16. @arburbank if you build it, they won’t come salesmanship
  17. 17. @arburbank Cultural maturity model evangelize educate explain get big
  18. 18. @arburbank Stage 3: get better get started get big get better
  19. 19. @arburbank problem: guidance get started get big get better needed
  20. 20. @arburbank you are the human in the loop ensure successrun test YOU
  21. 21. @arburbank you are the human in the loop ensure successrun test YOU ensure successrun test ensure success run test ensure success run test
  22. 22. @arburbank
  23. 23. @arburbank It is not your career goal to be the experiments person
  24. 24. @arburbank (you should have higher ambitions)
  25. 25. @arburbank Cultural maturity model how can I help? get better
  26. 26. @arburbank Stage 4: get out get started get big get better get out
  27. 27. @arburbank problem: scale yourself get started get big get better get out
  28. 28. @arburbank Write down the process What mistakes do you see in experiments? What questions do you answer repeatedly? How will learning this help others?
  29. 29. @arburbank
  30. 30. @arburbank
  31. 31. @arburbank
  32. 32. @arburbank “if you let engineers run experiments, they will screw them up in every way possible.”
  33. 33. @arburbank “if you let untrained engineers run experiments, they will screw them up in every way possible.”
  34. 34. @arburbank@arburbank
  35. 35. @arburbank For every important mistake, explain why it’s wrong and how to avoid it.
  36. 36. @arburbank launch landing in-flight
  37. 37. @arburbank launch @arburbank
  38. 38. @arburbank in-flight @arburbank
  39. 39. @arburbank landing @arburbank
  40. 40. @arburbank Make a list, check it twice    landing in-flight launch
  41. 41. @arburbank Make a list, check it twice e+r+
  42. 42. @arburbank Make a list, check it twice @experiments-help
  43. 43. @arburbank @experiments-help names matter: “help,” not “on-call”
  44. 44. @arburbank @experiments-help engineer partners: move fast, own the process
  45. 45. @arburbank @experiments-help the right people: thoughtful, well-respected
  46. 46. @arburbank
  47. 47. @arburbank Implement a process 1. Checklists for experiments 2. @experiments-help mention in code review 3. e+ as part of code review 4. Mailing list: experiments-help@ 5. Experiment document template 6. Rotation of experiment helpers
  48. 48. @arburbank Implement a process 1. Checklists for experiments 2. @experiments-help mention in code review 3. e+ as part of code review 4. Mailing list: experiments-help@ 5. Experiment document template 6. Rotation of experiment helpers
  49. 49. @arburbank Train your successors So you want to be an experiment helper? • Step 1: read the documentation • Step 2: take the experiment quiz • Step 3: review all experiments for a week
  50. 50. @arburbank
  51. 51. @arburbank
  52. 52. @arburbank 50 trained experiment helpers
  53. 53. @arburbank Cultural maturity model how would you answer that? get out
  54. 54. @arburbank Stage 5: get tools get started get big get better get out get tools
  55. 55. @arburbank problem: simple mistakes get started get big get better get out get tools
  56. 56. @arburbank get tools launch
  57. 57. @arburbank simplify experiment API
  58. 58. @arburbank remove untriggered experiments
  59. 59. @arburbank create helper functions
  60. 60. @arburbank get tools in flight
  61. 61. @arburbank add a control group automatically when a new variant is introduced
  62. 62. @arburbank expand experiment groups at the same rate
  63. 63. @arburbank get tools landing
  64. 64. @arburbank detect errors
  65. 65. @arburbank Automation: analysis chi-squared test on group sizes
  66. 66. @arburbank Automation: analysis test that groups grew at the same rate
  67. 67. @arburbank Automation: analysis verify similar distributions of users
  68. 68. @arburbank Automation: analysis hide results that are likely to be wrong
  69. 69. @arburbank simplify analysis
  70. 70. @arburbank Automation: analysis automatically track important metrics (and compute statistical significance)
  71. 71. @arburbank Automation: analysis segment important populations
  72. 72. @arburbank Automation: analysis measure novelty vs. long-term effects
  73. 73. @arburbank Cultural maturity model just use humans for the hard part: thinking get tools
  74. 74. @arburbank Experimentation maturity model get started get big get better get out get tools
  75. 75. @arburbank Stage 6: the future ??
  76. 76. @arburbank data science: changing minds, one at a time andrea@pinterest.com

×