Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
•
2 likes•3,461 views
Dan Siroker's presentation at Business of Software 2013 about the universal startup algorithm and how to apply lessons learned from the 2008 Obama campaign and 5,000 Optimizely customers who have run over 300,000 experiments.
Recommended
More Related Content
Similar to Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Similar to Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business? (20)
Recently uploaded
Recently uploaded (20)
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
- 1. A/B Testing If data can help win elections, what can it help you do for your business? Dan Siroker Co-Founder & CEO
- 10. Millions $7 Team: 4 Customers: 644 Annual Run Rate: $0.1m $6 $5 $4 $3 $2 $1 Dan Pete Eric $0 2010 Q2 Q1 Elliot Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
- 11. Millions $7 Team: 10 (+150%) Customers: 1,508 (+134%) Annual Run Rate: $1.2m(+982%) $6 $5 $4 $3 $2 Julio Ryan M $1 Ricky Meng Xiao Dan Mat Jeff P Pete Eric $0 2010 Q2 Q1 Elliot Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
- 12. Millions $7 Team: 42 (+320%) Customers: 3,460 (+129%) Annual Run Rate: $7.6m (+533%) $6 $5 Steve Tyle Jeff Z Arthu Amina Greg Sandy $4 Brendan Sophie Dogan Sonesh Shanann Tommy $3 Dan G Cara Byron Mik Alan Eddie Jodie $2 Dae-Ho Cyrus Julio Ryan M $1 Lauren Ashley Ricky Meng Xiao Dan Mat Cristian Travis Jeff P Pete Eric $0 2010 Q2 Q1 Elliot Katy Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Ryan L Ian Jonathan Q4 Joanna Jon
- 15. Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 2010 Q2 Q1 Q3 Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Q3 Q4 2013 Q2 Q1 Q3 Q4
- 16. Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 Steve Team: 42 Tyle Jeff Z Arthu Amina Greg Sophie Dogan Cara Alan Dan G Shanann Byron Eddie Sandy Dae-Ho Cyrus Sonesh Brendan Cristian Mik Ricky Dan Pete Eric 2010 Q2 Q1 Elliot Q3 Jeff P Lauren Ashley Tommy Ryan M Jodie Mat Meng Xiao Julio Katy Travis Jonathan Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Ryan L Q3 Q4 2013 Q2 Joanna Q1 Ian Jon Q3 Q4
- 17. Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 Steve Steve Team: 130 (+210%) Tyle Marius Adam Bo Evan Junan Ash Grace Anshu Lucas Benson Bryce Darwish Jeff Z Arthu David Jaron Richar Tammie Khattaab Laura Asia Amanda Amina Pamela Brian Greg Nikhil Martjin Sophie Amelia Dogan Gary Marco Natarajan Ben Kasra Katie Andrea Daniel SG Dae-Ho Omari Andy Eric Andy Mustafa Ranjit Chris Waylon Pedro Bradley Neha Lauren Adrian Chrix Cristian Mik Ricky Robin Dan Pete Eric 2010 Q2 Q1 Elliot Q3 Jeff P Ashley Tommy Ryan M Mat Julio Katy Travis Lisa Jamie Jonathan Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Nick Miyo Jodie Meng Xiao Ryan L Q3 Becca Oren Cyrus Brendan Megan Byron Eddie Sonesh Hemant Shanann Brett Sandy Ryan Michael James Etienne Alan Lacy Dan G Luke Cara Westly Aaron Adam Q4 2013 Q2 Helen Joanna Q1 Jen Ian Jon Nick Fabian Saum Courtney Dan Yonat Kent Ural Eduar Toby Oliver Grant Q3 Cate Christop Julia Q4 Danielle Tai Jonathan Chris Chris Rama
- 24. How?
- 31. Lesson #1: Define quantifiable success metrics.
- 32. visitors per day Visit sign up rate $ per recipient Email $
- 33. visitors per day Visit sign up rate $ per recipient Email $
- 34. Media Button
- 36. Splash page experiment Variations: Button: 1. Sign Up Media: 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
- 39. Button: “Join Us Now”
- 40. Button: “Sign Up Now”
- 41. Splash page experiment Variations: Button: 1. Sign Up Media: 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
- 43. Media: “Family”
- 44. Media: “Change”
- 48. Splash page experiment Variations: Button: 1. Sign Up Media: 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
- 52. Splash page experiment Email Volunteers Subscriptions 712,000 Amount Raised Original: 7,120,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
- 53. Splash page experiment Email Volunteers Subscriptions 712,000 Amount Raised Original: 7,120,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
- 54. Splash page experiment Email Volunteers Subscriptions 712,000 Amount Raised Original: 7,120,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
- 55. Lesson #1: Define quantifiable success metrics.
- 56. Lesson #2: Less is more. Reduce choices.
- 57. vs. +15% conversion rate +10% $ per visitor
- 61. Lesson #2: Less is more. Reduce choices.
- 62. Lesson #3: Words matter. Focus on your call to action.
- 64. Donation button experiment Variations Never Signed Up Signed Up, Never Donated Previously Donated 0.0% 0.0% 0.0% +2.3% +27.8% +16.3% -27.8% N/A N/A +15.2% -24.6% +11.9% +8.5% +2.9% +18.4%
- 67. Lesson #3: Words matter. Focus on your call to action.
- 68. Lesson #4: Seek the global maximum.
- 70. vs.
- 76. vs. +49% conversion rate from 240 A/B tests
- 77. Lesson #4: Seek the global maximum.
- 86. Lessons Learned 1. Define quantifiable success metrics. 2. Less is more. Reduce choices. 3. Words matter. Focus on your call to action. 4. Seek the global maximum. 5. Start today.
- 89. A/B Testing If data can help win elections, what can it help you do for your business? Dan Siroker Co-Founder & CEO