SlideShare ist ein Scribd-Unternehmen logo
1 von 71
Downloaden Sie, um offline zu lesen
Friday, January 31, 14
Friday, January 31, 14
Best Practices and
Lessons Learned from
100,000 A/B Tests
Martijn Geerlings

Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
9:1
Friday, January 31, 14
Lessons Learned

Friday, January 31, 14
Lesson #1:
Define success .

Friday, January 31, 14
Lesson #1:
Define quantifiable success metrics.

Friday, January 31, 14
Friday, January 31, 14
visitors
per day

Friday, January 31, 14

Visit
visitors
per day

Friday, January 31, 14

Visit

sign up
rate

Email
visitors
per day

Friday, January 31, 14

Visit

sign up
rate

Email

$ per recipient

$
visitors
per day

Friday, January 31, 14

Visit

sign up
rate

Email

$ per recipient

$
Media
Button

Friday, January 31, 14
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. Join Us Now

3. Change Image

4. Sign Up Now

4. Barack’s Video
5. Springfield Video
6. Sam’s Video

Friday, January 31, 14
Button: “Sign Up”
Friday, January 31, 14
Button: “Learn More”
Friday, January 31, 14
Button: “Join Us Now”
Friday, January 31, 14
Button: “Sign Up Now”
Friday, January 31, 14
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. Join Us Now

3. Change Image

4. Sign Up Now

4. Barack’s Video
5. Springfield Video
6. Sam’s Video

Friday, January 31, 14
Media: “Get Involved”
Friday, January 31, 14
Media: “Family”
Friday, January 31, 14
Media: “Change”
Friday, January 31, 14
Media: “Barack’s Video”
Friday, January 31, 14
Media: “Springfield Video”
Friday, January 31, 14
Media: “Sam’s Video”
Friday, January 31, 14
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. Join Us Now

3. Change Image

4. Sign Up Now

4. Barack’s Video
5. Springfield Video
6. Sam’s Video

Friday, January 31, 14
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. Join Us Now

3. Change Image

4. Sign Up Now

4. Barack’s Video
5. Springfield Video
6. Sam’s Video

Friday, January 31, 14
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. Join Us Now

3. Change Image

4. Sign Up Now

4. Barack’s Video
5. Springfield Video
6. Sam’s Video

Friday, January 31, 14
Splash page experiment

Friday, January 31, 14
Splash page experiment

Friday, January 31, 14
Splash page experiment

Friday, January 31, 14
Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:

7,120,000

+40.6%

+2,880,000

+288,000 +$57,000,000

New:

10,000,000

1,000,000 $200,000,000

Friday, January 31, 14

712,000

$143,000,000
Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:

7,120,000

+40.6%

+2,880,000

+288,000 +$57,000,000

New:

10,000,000

1,000,000 $200,000,000

Friday, January 31, 14

712,000

$143,000,000
Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:

7,120,000

+40.6%

+2,880,000

+288,000 +$57,000,000

New:

10,000,000

1,000,000 $200,000,000

Friday, January 31, 14

712,000

$143,000,000
Lesson #1:
Define quantifiable success metrics.

Friday, January 31, 14
Lesson #2:
Less is more. Reduce choices.

Friday, January 31, 14
+43.8%
Purchases

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.

+14.8% conversion rate

Friday, January 31, 14
Lesson #2:
Less is more. Reduce choices.

Friday, January 31, 14
Lesson #3:
Words matter. Focus on your call to action.

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.
+14.6% clicks

Friday, January 31, 14
Donation button experiment
Variations

Friday, January 31, 14
Donation button experiment
Previously
Donated

0.0%

0.0%

+2.3%

+27.8%

+16.3%

-27.8%

N/A

N/A

+15.2%

-24.6%

+11.9%

+8.5%
Friday, January 31, 14

Signed Up,
Never Donated

0.0%

Variations

Never
Signed Up

+2.9%

+18.4%
Lesson #3:
Words matter. Focus on your call to action.

Friday, January 31, 14
Lesson #4:
Fail fast.

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.

-10% conversions
Friday, January 31, 14
Lesson #4:
Fail fast.

Friday, January 31, 14
Lesson #5:
Seek the Global Maximum.

Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
+0.5% Retention
+1.4% Engagement

Friday, January 31, 14
Lesson #5:
Seek the Global Maximum.

Friday, January 31, 14
Lessons Learned
1. Define quantifiable success metrics.
2. Less is more. Reduce choices.
3. Words matter. Focus on your call to action.
4. Fail fast.
5. Seek the Global Maximum.

Friday, January 31, 14
Thank You

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.
+8% engagement
Friday, January 31, 14
2009!

Friday, January 31, 14

2010!

2011!

2012!

2013!

Weitere ähnliche Inhalte

Kürzlich hochgeladen

Brand experience Peoria City Soccer Presentation.pdf
Brand experience Peoria City Soccer Presentation.pdfBrand experience Peoria City Soccer Presentation.pdf
Brand experience Peoria City Soccer Presentation.pdf
tbatkhuu1
 

Kürzlich hochgeladen (20)

Enjoy Night⚡Call Girls Dlf City Phase 4 Gurgaon >༒8448380779 Escort Service
Enjoy Night⚡Call Girls Dlf City Phase 4 Gurgaon >༒8448380779 Escort ServiceEnjoy Night⚡Call Girls Dlf City Phase 4 Gurgaon >༒8448380779 Escort Service
Enjoy Night⚡Call Girls Dlf City Phase 4 Gurgaon >༒8448380779 Escort Service
 
Digital Strategy Master Class - Andrew Rupert
Digital Strategy Master Class - Andrew RupertDigital Strategy Master Class - Andrew Rupert
Digital Strategy Master Class - Andrew Rupert
 
Brighton SEO April 2024 - The Good, the Bad & the Ugly of SEO Success
Brighton SEO April 2024 - The Good, the Bad & the Ugly of SEO SuccessBrighton SEO April 2024 - The Good, the Bad & the Ugly of SEO Success
Brighton SEO April 2024 - The Good, the Bad & the Ugly of SEO Success
 
Generative AI Master Class - Generative AI, Unleash Creative Opportunity - Pe...
Generative AI Master Class - Generative AI, Unleash Creative Opportunity - Pe...Generative AI Master Class - Generative AI, Unleash Creative Opportunity - Pe...
Generative AI Master Class - Generative AI, Unleash Creative Opportunity - Pe...
 
The+State+of+Careers+In+Retention+Marketing-2.pdf
The+State+of+Careers+In+Retention+Marketing-2.pdfThe+State+of+Careers+In+Retention+Marketing-2.pdf
The+State+of+Careers+In+Retention+Marketing-2.pdf
 
Google 3rd-Party Cookie Deprecation [Update] + 5 Best Strategies
Google 3rd-Party Cookie Deprecation [Update] + 5 Best StrategiesGoogle 3rd-Party Cookie Deprecation [Update] + 5 Best Strategies
Google 3rd-Party Cookie Deprecation [Update] + 5 Best Strategies
 
Situation Analysis | Management Company.
Situation Analysis | Management Company.Situation Analysis | Management Company.
Situation Analysis | Management Company.
 
BDSM⚡Call Girls in Sector 150 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 150 Noida Escorts >༒8448380779 Escort ServiceBDSM⚡Call Girls in Sector 150 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 150 Noida Escorts >༒8448380779 Escort Service
 
Unlocking the Mystery of the Voynich Manuscript
Unlocking the Mystery of the Voynich ManuscriptUnlocking the Mystery of the Voynich Manuscript
Unlocking the Mystery of the Voynich Manuscript
 
No Cookies No Problem - Steve Krull, Be Found Online
No Cookies No Problem - Steve Krull, Be Found OnlineNo Cookies No Problem - Steve Krull, Be Found Online
No Cookies No Problem - Steve Krull, Be Found Online
 
Brand experience Peoria City Soccer Presentation.pdf
Brand experience Peoria City Soccer Presentation.pdfBrand experience Peoria City Soccer Presentation.pdf
Brand experience Peoria City Soccer Presentation.pdf
 
Five Essential Tools for International SEO - Natalia Witczyk - SearchNorwich 15
Five Essential Tools for International SEO - Natalia Witczyk - SearchNorwich 15Five Essential Tools for International SEO - Natalia Witczyk - SearchNorwich 15
Five Essential Tools for International SEO - Natalia Witczyk - SearchNorwich 15
 
SEO Master Class - Steve Wiideman, Wiideman Consulting Group
SEO Master Class - Steve Wiideman, Wiideman Consulting GroupSEO Master Class - Steve Wiideman, Wiideman Consulting Group
SEO Master Class - Steve Wiideman, Wiideman Consulting Group
 
Labour Day Celebrating Workers and Their Contributions.pptx
Labour Day Celebrating Workers and Their Contributions.pptxLabour Day Celebrating Workers and Their Contributions.pptx
Labour Day Celebrating Workers and Their Contributions.pptx
 
Developing Marketing Strategies and Plans kotler
Developing Marketing Strategies and Plans kotlerDeveloping Marketing Strategies and Plans kotler
Developing Marketing Strategies and Plans kotler
 
Social media, ppt. Features, characteristics
Social media, ppt. Features, characteristicsSocial media, ppt. Features, characteristics
Social media, ppt. Features, characteristics
 
BDSM⚡Call Girls in Sector 144 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 144 Noida Escorts >༒8448380779 Escort ServiceBDSM⚡Call Girls in Sector 144 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 144 Noida Escorts >༒8448380779 Escort Service
 
The Science of Landing Page Messaging.pdf
The Science of Landing Page Messaging.pdfThe Science of Landing Page Messaging.pdf
The Science of Landing Page Messaging.pdf
 
Kraft Mac and Cheese campaign presentation
Kraft Mac and Cheese campaign presentationKraft Mac and Cheese campaign presentation
Kraft Mac and Cheese campaign presentation
 
BDSM⚡Call Girls in Sector 128 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 128 Noida Escorts >༒8448380779 Escort ServiceBDSM⚡Call Girls in Sector 128 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 128 Noida Escorts >༒8448380779 Escort Service
 

Empfohlen

Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
 

Empfohlen (20)

PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy Presentation
 

Search Marketing Thursday - Januari 2014 - Martijn Geerlings