SlideShare ist ein Scribd-Unternehmen logo
1 von 45
Downloaden Sie, um offline zu lesen
Patrick McKenzie Opticon 2014: Advanced A/B Testing
Conversion Optimization:
The World Beyond
Headlines and Button Color
Patrick McKenzie
Founder, Kalzumeus Software
Low hanging fruit tastes great,
but the rest is worth the work, too.
Test Idea #1
Quotas / Tier Levels / etc
Patrick McKenzie Opticon 2014: Advanced A/B Testing
@user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")
What Did We Learn From That Test?
• ~60% increase in sales going from 15 -> 8 cards
• Synergistic effects can occasionally actually happen.
• You don’t know what you don’t know until you test
for it.
Key Points With Testing Quotas
• Grandfather in existing users.
• Set quota levels intelligently.
• Be ready to tolerate small uptick in support load.
• Your product is still too cheap.
Test Idea #2
Payment Methods / Checkout Flow
Threw This Together In An Hour (Oops)
Rethought Design & UX
What We Learned From This Test
• A single A/B test can help your bottom line more
than a product launch.
• Dealing with synergistic effects in testing can
occasionally be maddening.
• The data is never wrong, but sometimes the data
isn’t telling you what you think it is.
Key Points With Testing Checkout Flow
• Make absolutely sure you don’t break this!
• Have UX-focused team (product / marketing / etc)
go over every step of checkout funnel with zero
tolerance for bad UX.
• Checkout flows go “stale” at many successful
companies. Schedule a deep dive on them twice a
year, at least.
• Upsells and cross selling are classics for a reason!
Test Idea #3
Offline Business Processes
Patrick McKenzie Opticon 2014: Advanced A/B Testing
Testing degree of sales which
maximizes conversion of leads
What We Learned From This Test
• Important to communicate with sales/CS.
• Lead to resource-intensive but hugely successful
lead-scoring efforts.
• Justified very expensive interventions like e.g.
onboarding help for everybody.
Key Points With Testing Offline Processes
• Your software/workflow has to close the loop back
to the testing software. This can be tricky.
• Virtually no CRM or A/B testing software accurately
models enterprise decisionmaking process. You
might be forced to optimize for proxy conversions.
• Volumes required for statistical significance are
really punishing, so sometimes we “proxy and
pray.”
Test Idea #4
Product On-boarding / FRE
What Engineers Will Ship If You Let Them
Key Goals of Onboarding Process
• Get user to a concrete success as soon as possible.
• For more complex sales, lay groundwork for future
success by e.g. inviting in team or getting
permission to follow-up.
• For B2C, especially in social, you live and die on
virality during the onboarding experience.
In-app Tour Introducing Core Use
Cheat Code for B2B SaaS Sales
What We Learned From This Test
• Users who start onboarding tour 25% to 100%+
more likely to purchase software
• Users sometimes have visceral reaction to elements
of the onboarding experience.
• Engineers prefer in-app onboarding but it is
expensive as heck to build and does not always
justify itself versus e.g. a Youtube video.
Key Points With Testing Offline Processes
• Your software/workflow has to close the loop back
to the testing software. This can be tricky.
• Virtually no CRM or A/B testing software accurately
models enterprise decisionmaking process. You
might be forced to optimize for proxy conversions.
• Volumes required for statistical significance are
really punishing, so sometimes we “proxy and
pray.”
Test Idea #5
Core Product Interactions
Too Complicated
80% of Users Use 20% of Features
Try Radically Simplifying
Improved Presentation and UX Of Core
Workflow
What We Learned From This Test
• Just hiding the advanced features increased task
success by 10%, which directly improved sales
• Further jump of 15% when we did the redesign
– Absolutely no increase in sales…
– … but we saw in the funnel that this was solely due to
less people hitting the trial limits …
– … which we were able to profitably exploit with the
quota limit test described earlier
Key Points With Testing Core Interactions
• Test the entire funnel. Sometimes task success
doesn’t (unfortunately) help bottom line metrics.
• A good understanding of where people spend their
time/energy is vital – test the 90% interaction, not
the 2%.
ROI on ~$500 A Month Testing Budget
0.00%
0.50%
1.00%
1.50%
2.00%
2.50%
3.00%
3.50%
February,2010
March,2010
April,2010
May,2010
June,2010
July,2010
August,2010
September,2010
October,2010
November,2010
December,2010
January,2011
February,2011
March,2011
April,2011
May,2011
June,2011
July,2011
August,2011
September,2011
October,2011
November,2011
December,2011
January,2012
February,2012
March,2012
April,2012
May,2012
June,2012
July,2012
August,2012
September,2012
October,2012
November,2012
December,2012
January,2013
February,2013
March,2013
April,2013
May,2013
June,2013
July,2013
August,2013
September,2013
October,2013
November,2013
December,2013
January,2014
February,2014
Let’s Talk Testing Challenges
A/B Testing Across System
Boundaries
Not A Hard Problem
Our Marketing Site
A/B
Gets A Wee Bit Trickier With 2 Sites
Our Marketing Site
A/B
The Web Application
Total Freaking Madness
Our Marketing Site
A/B
The Web Application
iPhone app
Offline processes
Android app
Partner site
Content minisite
Offsite checkout
Japanese subsidiary
Sanity Restored
Consumer Applications
Identity & A/B Manager
Lessons Learned With This
• Think about strategy for transitioning people between
systems – easier to build into design earlier than later
• Cookies with unique identities are a stopgap, not a
solution. You need to understand the sales process.
• This is a good opportunity to standardize on an A/B
testing tool and workflow… or you’ll hate your life
• This is a “high class problem” which can be deferred
until $50M+ in revenue, but when you solve it, it is
going to cost serious time/money.
Increasing A/B Testing Adoption In
Engineering and Marketing Teams
Single Most Important Lesson:
One Line of Code
@user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")
Other Lessons Learned
• Organization should learn rather than guess better.
– Best investment ever: organization-wide wiki.
– Next best: weekly “What we tried this week” email.
• Engineers love A/B testing when you describe it like
a puzzle or a video game, with bragging rights.
• Marketers love A/B testing when it’s positioned as a
way to increase their status in the organization
rather than a replacement for their talents.
Things To Never Be Without
• CS should be able to force-override a user’s
participation in an A/B test (and, relatedly, “see
what users is seeing” is great if you can get it).
• A/B tests deeper than surface level need unit/etc
tests. Make sure your framework supports it.
• Test switcher UI in development/staging is a huge
win.
• URL query parameter to flip tests a huge win.
Thanks for Listening!
• I love talking about this and you cannot possibly
waste my time: @patio11 or
patrick@kalzumeus.com
• Free email mini-course on CRO in software
companies: http://bit.ly/patrick-optimizely
• Liked this presentation? I deliver something like
this twice a month, free, to my email list.
http://training.kalzumeus.com to sign up

Weitere ähnliche Inhalte

Was ist angesagt?

Building B2B Minimum Viable Products (MVP)
Building B2B Minimum Viable Products (MVP)Building B2B Minimum Viable Products (MVP)
Building B2B Minimum Viable Products (MVP)Totango
 
Build A Minimum Viable Product PowerPoint Presentation Slides
Build A Minimum Viable Product PowerPoint Presentation Slides Build A Minimum Viable Product PowerPoint Presentation Slides
Build A Minimum Viable Product PowerPoint Presentation Slides SlideTeam
 
Customer development and Agile development
Customer development and Agile developmentCustomer development and Agile development
Customer development and Agile developmentdchurchv
 
Test for Success: A Guide to A/B Testing on Emails & Landing Pages
Test for Success: A Guide to A/B Testing on Emails & Landing PagesTest for Success: A Guide to A/B Testing on Emails & Landing Pages
Test for Success: A Guide to A/B Testing on Emails & Landing PagesOptimizely
 
Optimizely Workshop: Mobile Walkthrough
Optimizely Workshop: Mobile Walkthrough Optimizely Workshop: Mobile Walkthrough
Optimizely Workshop: Mobile Walkthrough Optimizely
 
Optimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - AtlassianOptimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - AtlassianOptimizely
 
Workshop 6: Build Your Organization's Optimization Culture
Workshop 6: Build Your Organization's Optimization CultureWorkshop 6: Build Your Organization's Optimization Culture
Workshop 6: Build Your Organization's Optimization CultureOptimizely
 
Optimizing Your B2B Demand Generation Machine
Optimizing Your B2B Demand Generation MachineOptimizing Your B2B Demand Generation Machine
Optimizing Your B2B Demand Generation MachineOptimizely
 
Empowering You to Empower Them: Product-Led Onboarding: How to Turn New Users...
Empowering You to Empower Them: Product-Led Onboarding: How to Turn New Users...Empowering You to Empower Them: Product-Led Onboarding: How to Turn New Users...
Empowering You to Empower Them: Product-Led Onboarding: How to Turn New Users...Aggregage
 
How to make $200 a day with videos
How to make $200 a day with videosHow to make $200 a day with videos
How to make $200 a day with videosrajprasad63
 
Losing is the New Winning
Losing is the New WinningLosing is the New Winning
Losing is the New WinningOptimizely
 
Optimizely Workshop: Take Action on Results with Statistics
Optimizely Workshop: Take Action on Results with StatisticsOptimizely Workshop: Take Action on Results with Statistics
Optimizely Workshop: Take Action on Results with StatisticsOptimizely
 
[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven
[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven
[Elite Camp 2016] Peep Laja - Fresh Out Of the OvenCXL
 
How to create better A/B tests based on user research
How to create better A/B tests based on user researchHow to create better A/B tests based on user research
How to create better A/B tests based on user researchAGConsult
 
Eric Ries, Author/Speaker/Consultant, The Lean Startup
Eric Ries, Author/Speaker/Consultant, The Lean StartupEric Ries, Author/Speaker/Consultant, The Lean Startup
Eric Ries, Author/Speaker/Consultant, The Lean Startup500 Startups
 
Website Redesigns: Why they Fail and How to Ensure Success
Website Redesigns: Why they Fail and How to Ensure SuccessWebsite Redesigns: Why they Fail and How to Ensure Success
Website Redesigns: Why they Fail and How to Ensure SuccessOptimizely
 
Conversion optimization for startups
Conversion optimization for startupsConversion optimization for startups
Conversion optimization for startupsExperiments
 
How to be a (Kick Ass) PM w/ LoopNet's former VP of Product
How to be a (Kick Ass) PM w/ LoopNet's former VP of ProductHow to be a (Kick Ass) PM w/ LoopNet's former VP of Product
How to be a (Kick Ass) PM w/ LoopNet's former VP of ProductProduct School
 
Successful Prototyping
Successful PrototypingSuccessful Prototyping
Successful Prototypingtawi123
 

Was ist angesagt? (20)

Building B2B Minimum Viable Products (MVP)
Building B2B Minimum Viable Products (MVP)Building B2B Minimum Viable Products (MVP)
Building B2B Minimum Viable Products (MVP)
 
Build A Minimum Viable Product PowerPoint Presentation Slides
Build A Minimum Viable Product PowerPoint Presentation Slides Build A Minimum Viable Product PowerPoint Presentation Slides
Build A Minimum Viable Product PowerPoint Presentation Slides
 
Customer development and Agile development
Customer development and Agile developmentCustomer development and Agile development
Customer development and Agile development
 
Test for Success: A Guide to A/B Testing on Emails & Landing Pages
Test for Success: A Guide to A/B Testing on Emails & Landing PagesTest for Success: A Guide to A/B Testing on Emails & Landing Pages
Test for Success: A Guide to A/B Testing on Emails & Landing Pages
 
Optimizely Workshop: Mobile Walkthrough
Optimizely Workshop: Mobile Walkthrough Optimizely Workshop: Mobile Walkthrough
Optimizely Workshop: Mobile Walkthrough
 
Optimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - AtlassianOptimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - Atlassian
 
Workshop 6: Build Your Organization's Optimization Culture
Workshop 6: Build Your Organization's Optimization CultureWorkshop 6: Build Your Organization's Optimization Culture
Workshop 6: Build Your Organization's Optimization Culture
 
Optimizing Your B2B Demand Generation Machine
Optimizing Your B2B Demand Generation MachineOptimizing Your B2B Demand Generation Machine
Optimizing Your B2B Demand Generation Machine
 
Empowering You to Empower Them: Product-Led Onboarding: How to Turn New Users...
Empowering You to Empower Them: Product-Led Onboarding: How to Turn New Users...Empowering You to Empower Them: Product-Led Onboarding: How to Turn New Users...
Empowering You to Empower Them: Product-Led Onboarding: How to Turn New Users...
 
How to make $200 a day with videos
How to make $200 a day with videosHow to make $200 a day with videos
How to make $200 a day with videos
 
Losing is the New Winning
Losing is the New WinningLosing is the New Winning
Losing is the New Winning
 
Optimizely Workshop: Take Action on Results with Statistics
Optimizely Workshop: Take Action on Results with StatisticsOptimizely Workshop: Take Action on Results with Statistics
Optimizely Workshop: Take Action on Results with Statistics
 
[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven
[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven
[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven
 
007 model
007 model007 model
007 model
 
How to create better A/B tests based on user research
How to create better A/B tests based on user researchHow to create better A/B tests based on user research
How to create better A/B tests based on user research
 
Eric Ries, Author/Speaker/Consultant, The Lean Startup
Eric Ries, Author/Speaker/Consultant, The Lean StartupEric Ries, Author/Speaker/Consultant, The Lean Startup
Eric Ries, Author/Speaker/Consultant, The Lean Startup
 
Website Redesigns: Why they Fail and How to Ensure Success
Website Redesigns: Why they Fail and How to Ensure SuccessWebsite Redesigns: Why they Fail and How to Ensure Success
Website Redesigns: Why they Fail and How to Ensure Success
 
Conversion optimization for startups
Conversion optimization for startupsConversion optimization for startups
Conversion optimization for startups
 
How to be a (Kick Ass) PM w/ LoopNet's former VP of Product
How to be a (Kick Ass) PM w/ LoopNet's former VP of ProductHow to be a (Kick Ass) PM w/ LoopNet's former VP of Product
How to be a (Kick Ass) PM w/ LoopNet's former VP of Product
 
Successful Prototyping
Successful PrototypingSuccessful Prototyping
Successful Prototyping
 

Andere mochten auch

Conversion Optimization in Practice: BaconBiz 2013
Conversion Optimization in Practice: BaconBiz 2013Conversion Optimization in Practice: BaconBiz 2013
Conversion Optimization in Practice: BaconBiz 2013Patrick McKenzie
 
Software For Underserved Markets
Software For Underserved MarketsSoftware For Underserved Markets
Software For Underserved MarketsPatrick McKenzie
 
Productizing Twilio Applications
Productizing Twilio ApplicationsProductizing Twilio Applications
Productizing Twilio ApplicationsPatrick McKenzie
 
Selling Your Twilio-powered Apps to Businesses
Selling Your Twilio-powered Apps to BusinessesSelling Your Twilio-powered Apps to Businesses
Selling Your Twilio-powered Apps to BusinessesPatrick McKenzie
 
Data-Driven Software Design
Data-Driven Software DesignData-Driven Software Design
Data-Driven Software DesignPatrick McKenzie
 
A/B Testing Framework Design
A/B Testing Framework DesignA/B Testing Framework Design
A/B Testing Framework DesignPatrick McKenzie
 
Why Great Marketers Must Be Great Skeptics
Why Great Marketers Must Be Great SkepticsWhy Great Marketers Must Be Great Skeptics
Why Great Marketers Must Be Great SkepticsRand Fishkin
 

Andere mochten auch (7)

Conversion Optimization in Practice: BaconBiz 2013
Conversion Optimization in Practice: BaconBiz 2013Conversion Optimization in Practice: BaconBiz 2013
Conversion Optimization in Practice: BaconBiz 2013
 
Software For Underserved Markets
Software For Underserved MarketsSoftware For Underserved Markets
Software For Underserved Markets
 
Productizing Twilio Applications
Productizing Twilio ApplicationsProductizing Twilio Applications
Productizing Twilio Applications
 
Selling Your Twilio-powered Apps to Businesses
Selling Your Twilio-powered Apps to BusinessesSelling Your Twilio-powered Apps to Businesses
Selling Your Twilio-powered Apps to Businesses
 
Data-Driven Software Design
Data-Driven Software DesignData-Driven Software Design
Data-Driven Software Design
 
A/B Testing Framework Design
A/B Testing Framework DesignA/B Testing Framework Design
A/B Testing Framework Design
 
Why Great Marketers Must Be Great Skeptics
Why Great Marketers Must Be Great SkepticsWhy Great Marketers Must Be Great Skeptics
Why Great Marketers Must Be Great Skeptics
 

Ähnlich wie Patrick McKenzie Opticon 2014: Advanced A/B Testing

Conversion Optimization: The World Beyond Headlines & Button Color
Conversion Optimization: The World Beyond Headlines & Button ColorConversion Optimization: The World Beyond Headlines & Button Color
Conversion Optimization: The World Beyond Headlines & Button ColorOptimizely
 
How to Effectively Experiment in PM by LendingTree Sr PM
How to Effectively Experiment in PM by LendingTree Sr PMHow to Effectively Experiment in PM by LendingTree Sr PM
How to Effectively Experiment in PM by LendingTree Sr PMProduct School
 
Presentations - Zarget CRO meetup 2017
Presentations - Zarget CRO meetup 2017Presentations - Zarget CRO meetup 2017
Presentations - Zarget CRO meetup 2017ZargetHQ
 
The Lean Enterprise
The Lean EnterpriseThe Lean Enterprise
The Lean EnterpriseRyan Dorrell
 
Top 15 reasons to choose qa testing as career
Top 15 reasons to choose qa testing as career Top 15 reasons to choose qa testing as career
Top 15 reasons to choose qa testing as career JanBask Training
 
The Leaders Guide to Getting Started with Automated Testing
The Leaders Guide to Getting Started with Automated TestingThe Leaders Guide to Getting Started with Automated Testing
The Leaders Guide to Getting Started with Automated TestingJames Briers
 
5 Lessons Learned in Product Management by Twitch Senior PM
5 Lessons Learned in Product Management by Twitch Senior PM5 Lessons Learned in Product Management by Twitch Senior PM
5 Lessons Learned in Product Management by Twitch Senior PMProduct School
 
Why We Test - Rethinking Your Approach
Why We Test - Rethinking Your ApproachWhy We Test - Rethinking Your Approach
Why We Test - Rethinking Your Approachaudreybloemer
 
Ericriesleanstartuppresentationforweb2
Ericriesleanstartuppresentationforweb2Ericriesleanstartuppresentationforweb2
Ericriesleanstartuppresentationforweb2Edmund FOng
 
Testing Intelligence
Testing IntelligenceTesting Intelligence
Testing IntelligenceLalit Bhamare
 
ITM business barriers.pptx
ITM  business barriers.pptxITM  business barriers.pptx
ITM business barriers.pptxrajalakshmi5921
 
An Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit GrowthAn Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit GrowthOptimizely
 
Anton Muzhailo - Practical Test Process Improvement using ISTQB
Anton Muzhailo - Practical Test Process Improvement using ISTQBAnton Muzhailo - Practical Test Process Improvement using ISTQB
Anton Muzhailo - Practical Test Process Improvement using ISTQBIevgenii Katsan
 
Supercharge Your Testing Program
Supercharge Your Testing ProgramSupercharge Your Testing Program
Supercharge Your Testing ProgramOptimizely
 
The Lean Startup
The Lean StartupThe Lean Startup
The Lean StartupItai Damti
 
Optimizing Dev Portals with Analytics and Feedback
Optimizing Dev Portals with Analytics and FeedbackOptimizing Dev Portals with Analytics and Feedback
Optimizing Dev Portals with Analytics and FeedbackPronovix
 
10 Best Practices for Magento Maintenance and Support
10 Best Practices for Magento Maintenance and Support10 Best Practices for Magento Maintenance and Support
10 Best Practices for Magento Maintenance and SupportAPPSeCONNECT
 

Ähnlich wie Patrick McKenzie Opticon 2014: Advanced A/B Testing (20)

Conversion Optimization: The World Beyond Headlines & Button Color
Conversion Optimization: The World Beyond Headlines & Button ColorConversion Optimization: The World Beyond Headlines & Button Color
Conversion Optimization: The World Beyond Headlines & Button Color
 
How to Effectively Experiment in PM by LendingTree Sr PM
How to Effectively Experiment in PM by LendingTree Sr PMHow to Effectively Experiment in PM by LendingTree Sr PM
How to Effectively Experiment in PM by LendingTree Sr PM
 
Presentations - Zarget CRO meetup 2017
Presentations - Zarget CRO meetup 2017Presentations - Zarget CRO meetup 2017
Presentations - Zarget CRO meetup 2017
 
The Lean Enterprise
The Lean EnterpriseThe Lean Enterprise
The Lean Enterprise
 
Top 15 reasons to choose qa testing as career
Top 15 reasons to choose qa testing as career Top 15 reasons to choose qa testing as career
Top 15 reasons to choose qa testing as career
 
The Leaders Guide to Getting Started with Automated Testing
The Leaders Guide to Getting Started with Automated TestingThe Leaders Guide to Getting Started with Automated Testing
The Leaders Guide to Getting Started with Automated Testing
 
5 Lessons Learned in Product Management by Twitch Senior PM
5 Lessons Learned in Product Management by Twitch Senior PM5 Lessons Learned in Product Management by Twitch Senior PM
5 Lessons Learned in Product Management by Twitch Senior PM
 
Why We Test - Rethinking Your Approach
Why We Test - Rethinking Your ApproachWhy We Test - Rethinking Your Approach
Why We Test - Rethinking Your Approach
 
Ericriesleanstartuppresentationforweb2
Ericriesleanstartuppresentationforweb2Ericriesleanstartuppresentationforweb2
Ericriesleanstartuppresentationforweb2
 
Testing Intelligence
Testing IntelligenceTesting Intelligence
Testing Intelligence
 
ITM business barriers.pptx
ITM  business barriers.pptxITM  business barriers.pptx
ITM business barriers.pptx
 
The Finishing Line
The Finishing LineThe Finishing Line
The Finishing Line
 
An Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit GrowthAn Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit Growth
 
Anton Muzhailo - Practical Test Process Improvement using ISTQB
Anton Muzhailo - Practical Test Process Improvement using ISTQBAnton Muzhailo - Practical Test Process Improvement using ISTQB
Anton Muzhailo - Practical Test Process Improvement using ISTQB
 
Supercharge Your Testing Program
Supercharge Your Testing ProgramSupercharge Your Testing Program
Supercharge Your Testing Program
 
The Lean Startup
The Lean StartupThe Lean Startup
The Lean Startup
 
Optimizing Dev Portals with Analytics and Feedback
Optimizing Dev Portals with Analytics and FeedbackOptimizing Dev Portals with Analytics and Feedback
Optimizing Dev Portals with Analytics and Feedback
 
PQF Overview
PQF OverviewPQF Overview
PQF Overview
 
CRO Cheat Sheet
CRO Cheat SheetCRO Cheat Sheet
CRO Cheat Sheet
 
10 Best Practices for Magento Maintenance and Support
10 Best Practices for Magento Maintenance and Support10 Best Practices for Magento Maintenance and Support
10 Best Practices for Magento Maintenance and Support
 

Kürzlich hochgeladen

מיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלי
מיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלימיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלי
מיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגליBen Shushan & Co. C.P.A (Isr.)
 
BCE24 | Virtual Brand Ambassadors: Making Brands Personal - John Meulemans
BCE24 | Virtual Brand Ambassadors: Making Brands Personal - John MeulemansBCE24 | Virtual Brand Ambassadors: Making Brands Personal - John Meulemans
BCE24 | Virtual Brand Ambassadors: Making Brands Personal - John MeulemansBBPMedia1
 
Six Sigma Improvement Process: Transforming Processes, Elevating Performance
Six Sigma Improvement Process: Transforming Processes, Elevating PerformanceSix Sigma Improvement Process: Transforming Processes, Elevating Performance
Six Sigma Improvement Process: Transforming Processes, Elevating PerformanceOperational Excellence Consulting
 
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 Building Your Personal Brand on LinkedIn - Expert Planet-  2024 Building Your Personal Brand on LinkedIn - Expert Planet-  2024
Building Your Personal Brand on LinkedIn - Expert Planet- 2024Stephan Koning
 
Entrepreneurship & organisations: influences and organizations
Entrepreneurship & organisations: influences and organizationsEntrepreneurship & organisations: influences and organizations
Entrepreneurship & organisations: influences and organizationsP&CO
 
Data skills for Agile Teams- Killing story points
Data skills for Agile Teams- Killing story pointsData skills for Agile Teams- Killing story points
Data skills for Agile Teams- Killing story pointsyasinnathani
 
Chicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdf
Chicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdfChicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdf
Chicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdfSourav Sikder
 
Upgrade Your Banking Experience with Advanced Core Banking Applications
Upgrade Your Banking Experience with Advanced Core Banking ApplicationsUpgrade Your Banking Experience with Advanced Core Banking Applications
Upgrade Your Banking Experience with Advanced Core Banking ApplicationsIntellect Design Arena Ltd
 
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003believeminhh
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access
 
NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023Steve Rader
 
Human Resource Management Chapter 9: Performance Management and Appraisal
Human Resource Management Chapter 9: Performance Management and AppraisalHuman Resource Management Chapter 9: Performance Management and Appraisal
Human Resource Management Chapter 9: Performance Management and Appraisalgomezdominic3
 
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessQ2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessAPCO
 
Transform Your Kitchen Essential Tips for Renovations in Launceston
Transform Your Kitchen Essential Tips for Renovations in LauncestonTransform Your Kitchen Essential Tips for Renovations in Launceston
Transform Your Kitchen Essential Tips for Renovations in Launcestondjhbuildersau
 
HELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptx
HELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptxHELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptx
HELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptxHelene Heckrotte
 
Live-Streaming in the Music Industry Webinar
Live-Streaming in the Music Industry WebinarLive-Streaming in the Music Industry Webinar
Live-Streaming in the Music Industry WebinarNathanielSchmuck
 
TalentView Webinar: Empowering the Modern Workforce_ Redefininig Success from...
TalentView Webinar: Empowering the Modern Workforce_ Redefininig Success from...TalentView Webinar: Empowering the Modern Workforce_ Redefininig Success from...
TalentView Webinar: Empowering the Modern Workforce_ Redefininig Success from...TalentView
 
Ethical stalking by Mark Williams. UpliftLive 2024
Ethical stalking by Mark Williams. UpliftLive 2024Ethical stalking by Mark Williams. UpliftLive 2024
Ethical stalking by Mark Williams. UpliftLive 2024Winbusinessin
 

Kürzlich hochgeladen (20)

Deira Dubai Call Girls +971552825767 Call Girls In Downtown Dubai
Deira Dubai Call Girls +971552825767 Call Girls In Downtown DubaiDeira Dubai Call Girls +971552825767 Call Girls In Downtown Dubai
Deira Dubai Call Girls +971552825767 Call Girls In Downtown Dubai
 
מיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלי
מיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלימיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלי
מיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלי
 
BCE24 | Virtual Brand Ambassadors: Making Brands Personal - John Meulemans
BCE24 | Virtual Brand Ambassadors: Making Brands Personal - John MeulemansBCE24 | Virtual Brand Ambassadors: Making Brands Personal - John Meulemans
BCE24 | Virtual Brand Ambassadors: Making Brands Personal - John Meulemans
 
Six Sigma Improvement Process: Transforming Processes, Elevating Performance
Six Sigma Improvement Process: Transforming Processes, Elevating PerformanceSix Sigma Improvement Process: Transforming Processes, Elevating Performance
Six Sigma Improvement Process: Transforming Processes, Elevating Performance
 
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 Building Your Personal Brand on LinkedIn - Expert Planet-  2024 Building Your Personal Brand on LinkedIn - Expert Planet-  2024
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 
Entrepreneurship & organisations: influences and organizations
Entrepreneurship & organisations: influences and organizationsEntrepreneurship & organisations: influences and organizations
Entrepreneurship & organisations: influences and organizations
 
Data skills for Agile Teams- Killing story points
Data skills for Agile Teams- Killing story pointsData skills for Agile Teams- Killing story points
Data skills for Agile Teams- Killing story points
 
Chicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdf
Chicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdfChicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdf
Chicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdf
 
Upgrade Your Banking Experience with Advanced Core Banking Applications
Upgrade Your Banking Experience with Advanced Core Banking ApplicationsUpgrade Your Banking Experience with Advanced Core Banking Applications
Upgrade Your Banking Experience with Advanced Core Banking Applications
 
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024
 
NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023
 
Human Resource Management Chapter 9: Performance Management and Appraisal
Human Resource Management Chapter 9: Performance Management and AppraisalHuman Resource Management Chapter 9: Performance Management and Appraisal
Human Resource Management Chapter 9: Performance Management and Appraisal
 
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessQ2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
 
Transform Your Kitchen Essential Tips for Renovations in Launceston
Transform Your Kitchen Essential Tips for Renovations in LauncestonTransform Your Kitchen Essential Tips for Renovations in Launceston
Transform Your Kitchen Essential Tips for Renovations in Launceston
 
HELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptx
HELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptxHELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptx
HELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptx
 
Sixth Sense Media Portfolio (Eng Ver) PDF
Sixth Sense Media Portfolio (Eng Ver) PDFSixth Sense Media Portfolio (Eng Ver) PDF
Sixth Sense Media Portfolio (Eng Ver) PDF
 
Live-Streaming in the Music Industry Webinar
Live-Streaming in the Music Industry WebinarLive-Streaming in the Music Industry Webinar
Live-Streaming in the Music Industry Webinar
 
TalentView Webinar: Empowering the Modern Workforce_ Redefininig Success from...
TalentView Webinar: Empowering the Modern Workforce_ Redefininig Success from...TalentView Webinar: Empowering the Modern Workforce_ Redefininig Success from...
TalentView Webinar: Empowering the Modern Workforce_ Redefininig Success from...
 
Ethical stalking by Mark Williams. UpliftLive 2024
Ethical stalking by Mark Williams. UpliftLive 2024Ethical stalking by Mark Williams. UpliftLive 2024
Ethical stalking by Mark Williams. UpliftLive 2024
 

Patrick McKenzie Opticon 2014: Advanced A/B Testing

  • 2. Conversion Optimization: The World Beyond Headlines and Button Color Patrick McKenzie Founder, Kalzumeus Software
  • 3. Low hanging fruit tastes great, but the rest is worth the work, too.
  • 4. Test Idea #1 Quotas / Tier Levels / etc
  • 6. @user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")
  • 7. What Did We Learn From That Test? • ~60% increase in sales going from 15 -> 8 cards • Synergistic effects can occasionally actually happen. • You don’t know what you don’t know until you test for it.
  • 8. Key Points With Testing Quotas • Grandfather in existing users. • Set quota levels intelligently. • Be ready to tolerate small uptick in support load. • Your product is still too cheap.
  • 9. Test Idea #2 Payment Methods / Checkout Flow
  • 10. Threw This Together In An Hour (Oops)
  • 12. What We Learned From This Test • A single A/B test can help your bottom line more than a product launch. • Dealing with synergistic effects in testing can occasionally be maddening. • The data is never wrong, but sometimes the data isn’t telling you what you think it is.
  • 13. Key Points With Testing Checkout Flow • Make absolutely sure you don’t break this! • Have UX-focused team (product / marketing / etc) go over every step of checkout funnel with zero tolerance for bad UX. • Checkout flows go “stale” at many successful companies. Schedule a deep dive on them twice a year, at least. • Upsells and cross selling are classics for a reason!
  • 14. Test Idea #3 Offline Business Processes
  • 16. Testing degree of sales which maximizes conversion of leads
  • 17. What We Learned From This Test • Important to communicate with sales/CS. • Lead to resource-intensive but hugely successful lead-scoring efforts. • Justified very expensive interventions like e.g. onboarding help for everybody.
  • 18. Key Points With Testing Offline Processes • Your software/workflow has to close the loop back to the testing software. This can be tricky. • Virtually no CRM or A/B testing software accurately models enterprise decisionmaking process. You might be forced to optimize for proxy conversions. • Volumes required for statistical significance are really punishing, so sometimes we “proxy and pray.”
  • 19. Test Idea #4 Product On-boarding / FRE
  • 20. What Engineers Will Ship If You Let Them
  • 21. Key Goals of Onboarding Process • Get user to a concrete success as soon as possible. • For more complex sales, lay groundwork for future success by e.g. inviting in team or getting permission to follow-up. • For B2C, especially in social, you live and die on virality during the onboarding experience.
  • 23. Cheat Code for B2B SaaS Sales
  • 24. What We Learned From This Test • Users who start onboarding tour 25% to 100%+ more likely to purchase software • Users sometimes have visceral reaction to elements of the onboarding experience. • Engineers prefer in-app onboarding but it is expensive as heck to build and does not always justify itself versus e.g. a Youtube video.
  • 25. Key Points With Testing Offline Processes • Your software/workflow has to close the loop back to the testing software. This can be tricky. • Virtually no CRM or A/B testing software accurately models enterprise decisionmaking process. You might be forced to optimize for proxy conversions. • Volumes required for statistical significance are really punishing, so sometimes we “proxy and pray.”
  • 26. Test Idea #5 Core Product Interactions
  • 28. 80% of Users Use 20% of Features
  • 30. Improved Presentation and UX Of Core Workflow
  • 31. What We Learned From This Test • Just hiding the advanced features increased task success by 10%, which directly improved sales • Further jump of 15% when we did the redesign – Absolutely no increase in sales… – … but we saw in the funnel that this was solely due to less people hitting the trial limits … – … which we were able to profitably exploit with the quota limit test described earlier
  • 32. Key Points With Testing Core Interactions • Test the entire funnel. Sometimes task success doesn’t (unfortunately) help bottom line metrics. • A good understanding of where people spend their time/energy is vital – test the 90% interaction, not the 2%.
  • 33. ROI on ~$500 A Month Testing Budget 0.00% 0.50% 1.00% 1.50% 2.00% 2.50% 3.00% 3.50% February,2010 March,2010 April,2010 May,2010 June,2010 July,2010 August,2010 September,2010 October,2010 November,2010 December,2010 January,2011 February,2011 March,2011 April,2011 May,2011 June,2011 July,2011 August,2011 September,2011 October,2011 November,2011 December,2011 January,2012 February,2012 March,2012 April,2012 May,2012 June,2012 July,2012 August,2012 September,2012 October,2012 November,2012 December,2012 January,2013 February,2013 March,2013 April,2013 May,2013 June,2013 July,2013 August,2013 September,2013 October,2013 November,2013 December,2013 January,2014 February,2014
  • 34. Let’s Talk Testing Challenges
  • 35. A/B Testing Across System Boundaries
  • 36. Not A Hard Problem Our Marketing Site A/B
  • 37. Gets A Wee Bit Trickier With 2 Sites Our Marketing Site A/B The Web Application
  • 38. Total Freaking Madness Our Marketing Site A/B The Web Application iPhone app Offline processes Android app Partner site Content minisite Offsite checkout Japanese subsidiary
  • 40. Lessons Learned With This • Think about strategy for transitioning people between systems – easier to build into design earlier than later • Cookies with unique identities are a stopgap, not a solution. You need to understand the sales process. • This is a good opportunity to standardize on an A/B testing tool and workflow… or you’ll hate your life • This is a “high class problem” which can be deferred until $50M+ in revenue, but when you solve it, it is going to cost serious time/money.
  • 41. Increasing A/B Testing Adoption In Engineering and Marketing Teams
  • 42. Single Most Important Lesson: One Line of Code @user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")
  • 43. Other Lessons Learned • Organization should learn rather than guess better. – Best investment ever: organization-wide wiki. – Next best: weekly “What we tried this week” email. • Engineers love A/B testing when you describe it like a puzzle or a video game, with bragging rights. • Marketers love A/B testing when it’s positioned as a way to increase their status in the organization rather than a replacement for their talents.
  • 44. Things To Never Be Without • CS should be able to force-override a user’s participation in an A/B test (and, relatedly, “see what users is seeing” is great if you can get it). • A/B tests deeper than surface level need unit/etc tests. Make sure your framework supports it. • Test switcher UI in development/staging is a huge win. • URL query parameter to flip tests a huge win.
  • 45. Thanks for Listening! • I love talking about this and you cannot possibly waste my time: @patio11 or patrick@kalzumeus.com • Free email mini-course on CRO in software companies: http://bit.ly/patrick-optimizely • Liked this presentation? I deliver something like this twice a month, free, to my email list. http://training.kalzumeus.com to sign up