Más contenido relacionado

Similar a Conversion Hotel 2018 Keynote: Chad Sanderson(20)


Más de Webanalisten .nl(20)


Conversion Hotel 2018 Keynote: Chad Sanderson

  1. Next Speaker: Chad Sanderson (USA) Experiments taken to a broader level Next Speaker:
  2. Hello! I am Chad Sanderson Product Manager, amateur data scientist, dog father, and conversion optimization practitioner. I like getting smart about testing. 5 @chadjsanderson
  3. Beyond Traditional A/B Testing
  4. What is a Digital Randomized Controlled Trial?
  5. What is the value of Digital RCT’s? Automated Randomization Treatment Assignment, or the method of distributing users into bucketed groups Experimental Statistics Methods to calculate aggregate metrics and conduct ‘tests’ using measured variance. Frequentist techniques offer strong error quantification Boundaries & Safeguards Quickly determine whether or not new features are negatively impacting performance or user metrics
  6. Well…not quite SRM Representation Seasonality Cookie Churn
  7. “Experimentation [CRO] is a framework for quantifying the risk of a decision by inferring causality based on data.”
  8. Areas of Business Risk • Hiring a new team member • Executive leadership performance • Purchase of new tools • Launching loyalty programs • Changing a creative direction • Beginning a social media campaign • Evaluating a vendor • Writing documentation guidelines • Making a sales pitch • Responding to a customer
  9. ‘Strict Randomized Controlled Trials’ All hail our robot overlords Time Series Analysis Pre-Post? No problem Factorials / MVT’s Finding the right levers to pull ‘Loose Randomized Controlled Trials’ Getting our hands dirty
  10. Small Sample Testing A convenience sample of 789 adults (248 women and 541 men) was recruited for the study during hospital admission
  11. Remember what matters Representation! Sample size DOES NOT affect the accuracy of your data so long as what you collect is representative of your underlying population.
  12. Garbage in, Garbage out -18%-- Product Teams who pre-registered test designs experienced a 18% reduction in submission volume but the % of winning experiments remained fixed.
  13. Segmented Regression • Randomization isn’t always possible • Regression can be used to identify correlations in data • ‘Time before and after a change point’ can be used as an independent variable • Historical data can be used • Control is not needed!
  14. Factorial Experiments • In factorial experiments, we typically are working with very small sample sizes, where each sample is potentially expensive • What we care about is not the BEST performing combinations of treatments, but whether there are sizeable main effects or interaction effects that can inform us which ‘levers to pull.’ • Continual optimization can be performed by ‘hill climbing’ without the need of large sample sizes, hypotheses, or even p values.
  15. Getting more $$$ for your $$$ The social media team was small (1 person) and wanted to understand how to optimize advertising spend. Because each experiment was high cost, a factorial experiment was the perfect tool to understand important factors: 1. Advertisement location a. Newsfeed b. Sidebar 2. Audience Age a. Under 25 b. Over 45
  16. Holy interaction plots, Batman! Even with relatively few samples, we were able to determine that the group Age made a dramatic difference in ad performance, while there also appeared to be a weak interaction between Age and Advertisement Location. All age groups had better performance when ads were targeted to the Newsfeed, but young people seemed to respond especially poorly for the right hand panel ads.
  17. Rules of the Road 1. Understand your limits. Take it slow! 2. Take the data for what it is. Accept imperfection. Data doesn’t always tell the full story, and that’s okay. 3. Experiment for ROI. A cheap test that could help avert a major disaster? Great! An expensive test that won’t tell you much of anything? Not great. 4. And uh…don’t do anything that would get you arrested.
  18. Good luck!