SlideShare ist ein Scribd-Unternehmen logo
1 von 71
Downloaden Sie, um offline zu lesen
Myths,
Lies and
Illusions
of Split
Testing
@OptimiseOrDie
@OptimiseOrDie
• UX, Analytics, Split Testing and Growth Rate Optimisation
• Started doing testing & CRO 2004
• Split tested over 45M visitors in 19 languages
• 67+ mistakes I MADE with AB testing
• Like riding a bike…
• Optimise your optimisation? Get in touch!
Myths, Lies and Illusions of Optimisation
1. Optimisation and Testing are the same
2. Optimisation and Testing are easy
3. Statistical significance tells you when to stop
4. High traffic always makes tests shorter
5. It’s all about Conversion Rates
6. You don’t need to test the test
7. Your tests will give you the quoted lift
8. High traffic pages give the best tests
9. It’s about getting all your tests to win
10. Optimisation is only about websites
@OptimiseOrDie
11. Segmentation will tell you what happened
12. You can’t run multiple simultaneous tests
13. Testing is great for settling arguments
14. You can spot trends early on in a test
15. More test volumes = better results
16. Your tests tell you truths that last forever
17. You can test even on low traffic sites
18. Other people’s tests are ‘Best Practice’
19. Doesn’t involve changing the way you work
20. Testing makes you a Data Scientist
#fail
@OptimiseOrDie
@OptimiseOrDie
26.6M
@OptimiseOrDie28.4M
Oppan Gangnam Style!
@OptimiseOrDie6.9M
You naughty boy…
@OptimiseOrDie1. Testing & Optimisation are the same thing
Testin
g
Optimise
What is CRO/Optimisation?
• “Using Analytics data and
Customer feedback to improve the
performance of your website.”
• “Finding out why visitors aren’t
converting and then resolving
these issues.”
• “Running loads of crappy split tests
randomly until the heat death of
the universe”
@OptimiseOrDie
What is my definition?
“A structured, systematic and continuous
application of techniques that are used to
discover, quantify and prioritise issues.
These can be turned into hypotheses to drive
experiments and opportunity in the following
business outcomes:”
• Increased revenue or profitability
• Increasing LTV, loyalty, NPS/Sat scores, Kristoffer Potential
• Removing cost from the business or contact center
• Higher productivity or labour flexibility
• Delighting customers
• Reduced development effort
@OptimiseOrDie
What Optimisation is NOT!
• A way to change things you (or others) hate
• A methodology for running split tests
• A guarantee of increased conversion
• A methodology for looking at analytics data
• A rescue for silo-bound or non-agile design or
development processes
• A way to trick people into buying
• A bolt-on – it IS the process
@OptimiseOrDie
Optimisation is a:
• Way of joining the worlds of Customer insight, UX,
Analytics, Split testing and Business Strategy
• Overarching Design and development process which
prioritises work around opportunity
• Strategic, not tactical, response to wasted development
effort or product change
• Way to create a deep and meaningful connection
between the team, customers, business and the outcomes
of making product changes
• The killer app to remove ego, opinion, assumptions,
cherished notions or ‘we just do it that way’ from decision
making.
• More powerful method than UX research or analytics
alone, in guiding the directionality of product change
@OptimiseOrDie
Optimisation includes these:
• Qualitative research
• Analytics, Quant analysis and
insight
• UX inspection and discovery
• Competitive Intelligence
• Priority based opportunity
• VOC, Surveys and Customer
Satisfaction (NPS)
• Call tracking & Call centre
optimisation
• AB testing
• Multivariate testing
• Photography optimisation
• EEG / ECG / Galvanic response
@OptimiseOrDie
• Web performance tuning
• Forms analytics
• Eye tracking
• Market research
• Big & Unstructured data analysis
• PPC optimisation
• Session replay analysis
• Customer journey mapping
• Ethnographic (diary) study
research
• Cross device, platform and
channel insight
• Email optimisation
2. Optimisation & Testing are
EASY
Surely you just add Javascript?
• This work is harder than anything I’ve ever done!
• You all have expensive & limited resources for testing - like Airport take-off slots
• You MUST make use of these efficiently
• You HAVE to balance resource cost with opportunity by prioritising carefully
@OptimiseOrDie
• You’re doing the equivalent of drug trials!
• For large OR small companies, the instrumentation, analytics, tools
setup, plan design, test methodology and analysis cannot be ‘done
later on’. It’s not complex but it is vital.
• The best companies have a structural shell (process, methodology,
management) around their activities
• If you’re not optimising the process continuously (Kaizen), you won’t
increase your velocity of iterations.
• Optimise the Optimisation
3. Statistical significance tells you when to
stop
@OptimiseOrDie
The 95% Stopping Problem
@OptimiseOrDie
• Many people use 95, 99% ‘confidence’ to stop
• This value is unreliable and moves around
• Nearly all my tests reach significance before they are
actually ready
• You can hit 95% early in a test (18 minutes!)
• If you stop, it could be a false result
• Read this Nature article : bit.ly/1dwk0if
• Optimizely have changed their stats engine
• This 95% thingy – must be LAST on your stop list
• Let me explain
The 95% Stopping Problem
@OptimiseOrDie
Scenario 1 Scenario 2 Scenario 3 Scenario 4
After 200
observations
Insignificant Insignificant Significant! Significant!
After 500
observations
Insignificant Significant! Insignificant Significant!
End of
experiment
Insignificant Significant! Insignificant Significant!
“You should know that stopping a test once it’s significant is deadly sin
number 1 in A/B testing land. 77% of A/A tests (testing the same thing
as A and B) will reach significance at a certain point.”
Ton Wesseling, Online Dialogue
The 95% Stopping Problem
@OptimiseOrDie
“Statistical Significance does not equal Validity”
http://bit.ly/1wMfmY2
“Why every Internet Marketer should be a Statistician”
http://bit.ly/1wMfs1G
“Understanding the Cycles in your site”
http://mklnd.com/1pGSOUP
4. High Traffic always means short tests
@OptimiseOrDie
Business & Purchase Cycles
@OptimiseOrDie
• Customers change
• Your traffic mix changes
• Markets, competitors
• Be aware of all the waves
• Always test whole cycles
• Don’t exclude slower buyers
• When you stop, let test
subjects still complete!
Start Test Finish Avg Cycle
• TWO BUSINESS CYCLES minimum (week/mo)
• 1 PURCHASE CYCLE minimum
• 250 CONVERSIONS minimum per creative (e.g. checkouts)
• 350 & MORE! if response is very similar
• FULL WEEKS/CYCLES never part of one
• KNOW what marketing, competitors and cycles are doing
• RUN a test length calculator - bit.ly/XqCxuu
• SET your test run time , RUN IT, STOP IT, ANALYSE IT
• ONLY RUN LONGER if you need more data
• DON’T RUN LONGER just because the test isn’t giving the result you want!
@OptimiseOrDie
How Long? Simple Rules to follow
5. It’s ALL about Conversion Rate
@OptimiseOrDie
It’s all about the business
@OptimiseOrDie
• You’re optimising a business here, not a page or site
• Tricking, pushing or persuading people at a superficial level to
take an action is not a viable strategy
• Your optimisation strategy is a series of steps, not a tool
• Testing is about learning, not converting.
• Tests that fail to tell you anything (regardless of outcome) are
a failure themselves
• If you don’t shift the business goals, your optimisation and
testing budget will be threatened
6. You don’t need to test the test – just
go
@OptimiseOrDie
Browser testing www.crossbrowsertesting.com
www.browserstack.com
www.spoon.net
www.saucelabs.com
www.multibrowserviewer.com
Mobile devices www.appthwack.com
www.deviceanywhere.com
www.opendevicelab.com
Read this article bit.ly/1wBccsJ
7. The test result gives the promised lift
@OptimiseOrDie
The result is a range
@OptimiseOrDie
• Version A is 3% conversion
• Version B is 4% conversion
• Yay! That’s a 25% lift
• Let’s tell everyone
• When it goes live, you get 5.7%
• That’s because it was A RANGE
• 3% +/- 0.5
• 4% +/- 0.4
• Actual result was 3.5% for A
• Actual result was 3.7% for B
AB Testing Visualisation Tool
@OptimiseOrDie
abtestguide.com/calc/
8. Testing is best on high traffic pages
@OptimiseOrDie
Think like the CEO of a
department store!
If you can’t refurbish the entire
store, which floors or
departments will you invest in
optimising?
Wherever there is:
• Footfall
• Low return
• Opportunity
Ecommerce
Bounce
Search or
Category
Product
Page
Add to
basket
View
basket
Checkout
Complete
@OptimiseOrDie
8. Testing is best on high traffic pages
@OptimiseOrDie
8. It’s all about prioritisation!
9. It’s all about WINNING test results
@OptimiseOrDie
Failing is good
@OptimiseOrDie
• Tests that are ‘about the same’ are a failure
• They’re also very hard to call
• That means you have to be BOLD not conservative
• A test that comes out negative is NOT a failure
• If a ‘negative’ test teaches you something, it’s a success!
• If you hit 40/50/60% failed tests, that’s fine
• If you aren’t failing regularly, you’re not BOLD enough
• Success is about the number of tests you finish each month,
and what you learn from them
We believe that doing [A]
for People [B] will make
outcome [C] happen.
We’ll know this when we
observe data [D] and
obtain feedback [E].
(reverse)
@OptimiseOrDie
10. Optimisation is only for Websites
@OptimiseOrDie
Optimisation is just for
websites
@OptimiseOrDie
• Service Design (Airbnb)
• Onboarding flows with emails
• Email templates
• Apps – testing, debugging, tracking
• Phone tracking and call centre optimisation
• Social, Display, TV, Video and other advertising
• Print adverts, Direct Mail
• In-store promotions
• Product manuals, guides, interfaces
• EVERYTHING has elasticity – just find it
• Even Multi-variate call centre scripts
11. Segmentation will explain things
@OptimiseOrDie
Segmentation explains stuff
@OptimiseOrDie
• Beware of small sample sizes
• A = 350 conversions
• B = 300 conversions
• A Conversions for Safari = 20
• B Conversions for Safari = 25
• Only needs 2 people to change that from 25% lift to 14%
12. You can’t run concurrent split tests
@OptimiseOrDie
Oh yes you can, with GA!
@OptimiseOrDie
• If you push events or variables into GA, you can report on
behaviour for A or B (or any variations).
• If you do it that way, you can easily run multiple tests on
different page targets simultaneously.
• You grab AAA, AAB, ABA, ABB and analyse.
• Test subjects get a recipe of tests, so one caveat
• If you pick things that clash or jar the experience
• If you have $5 delivery but then $2 in another test
• Apart from that, it’s fine to run these
• They tell you about the right experience recipe across several
pages being optimised in concert
13. Testing is great for settling arguments
@OptimiseOrDie
14. You can spot trends early in a test
@OptimiseOrDie
You can spot trends early
@OptimiseOrDie
• Tests are volatile in the early stages
• Watch but shut your mouth and wait
• Keep an eye on any odd behaviour
• Be patient!
15. More tests = Better results
@OptimiseOrDie
More tests = Better results
@OptimiseOrDie
• Increasing volume without optimising velocity
• Not optimising the process means it scales badly
• If you put garbage in, you’ll get garbage out
• Ramping up before you have it tuned is crazy
• Get the team, process, project management, methodology,
toolkits, selling and PR nailed first
• THEN optimise and scale
• I know that one UK retailer is doing 140 per month!
• That’s LOW compared to some companies I work with
• Big companies doing 2 a month? Meh.
16. Each test is a truth FOREVER
@OptimiseOrDie
Tests are a Truth Forever
@OptimiseOrDie
• Traffic changes
• Prices change
• Product mix changes
• Advertising evolves and changes
• Markets are different
• Customers have changed
• Competitors or regulatory landscape moves
• Things happen outside of your control
• You need to revisit tests or ITERATE
• Always be trying to beat the new winner, not basking in the
glory of a test you ran 9 months ago
• The lift may have vanished
• Schrodinger’s AB test
17. Testing is fine on Low Traffic sites
@OptimiseOrDie
Testing is fine on low traffic
sites
@OptimiseOrDie
• Yes, if you estimate your minimum testing unit
• This is the time to AB test a sample of say 250 conversions, with
the rules I set earlier.
• A payday loan company? 2 months minimum!
• Run the calculations
• Check the test length
• If it takes like, 8 million years, what can you do?
• Read this or download my AB testing decks:
bit.ly/1umy5Y6
18. Other people’s tests are Best Practice
@OptimiseOrDie
“STOP copying your competitors
They may not know what the
f*** they are doing either”
Peep Laja, ConversionXL
Tests you see online?
@OptimiseOrDie
• Your customers are not the same
• Your site is not the same
• Your advertising and traffic is not the same
• Your UX is not the same
• How the f*** do you think it guarantees a result?
• Use them to inform or suggest ideas
• They’re like the picture on meal packets
• Serving Suggestion Only
19. Testing makes you a Data Scientist
@OptimiseOrDie
20. Testing and Innovation is a plugin
@OptimiseOrDie
WE’RE ALL WINGING
IT
Guessaholics Anonymous
2004 Headspace
What I thought
I knew in 2004
Reality
2014 Headspace
What I
KNOW
I know
Me, on a
good day
Rumsfeldian
Space
@OptimiseOrDie
#1 Culture & Team
#2 Toolkit & Analytics investment
#3 UX, CX, Service Design, Insight
#4 Persuasive Copywriting
#5 Experimentation (testing) tools
The 5 Legged Optimisation
Barstool
@OptimiseOrDie
Marketing
?
Variation
Heritability
of good
ideas
Selection
based on
death
Business Future Testing?
Congratulations!
Today you’re the lucky
winner of our random
awards programme.
You get all these extra
features for free, on us.
Enjoy.
Mr D. Vader
Thank You!
Mail : sullivac@gmail.com
Deck :
Linkedin : linkd.in/pvrg14
READ STUFF
READ STUFF
READ STUFF
#5 : FIND STUFF
@OptimiseOrDie
@danbarker Analytics
@fastbloke Analytics
@timlb Analytics
@jamesgurd Analytics
@therustybear Analytics
@carmenmardiros Analytics
@davechaffey Analytics
@priteshpatel9 Analytics
@cutroni Analytics
@avinash Analytics
@Aschottmuller Analytics, CRO
@cartmetrix Analytics,
CRO
@Kissmetrics CRO / UX
@Unbounce CRO / UX
@Morys CRO / Neuro
@UXFeeds UX / Neuro
@Psyblog Neuro
@Gfiorelli1 SEO / Analytics
@PeepLaja CRO
@TheGrok CRO
@UIE UX
@LukeW UX / Forms
@cjforms UX / Forms
@axbom UX
@iatv UX
@Chudders Photo UX
@JeffreyGroks Innovation
@StephanieRieger Innovation
@BrianSolis Innovation
@DrEscotet Neuro
@TheBrainLady Neuro
@RogerDooley Neuro
@Cugelman Neuro
@Smashingmag Dev / UX
@uxmag UX
@Webtrends UX /
CRO
#5 : LEARN STUFF
@OptimiseOrDie
Baymard.com
Lukew.com
Smashingmagazine.com
ConversionXL.com
Medium.com
Whichtestwon.com
Unbounce.com
Measuringusability.com
RogerDooley.com
Kissmetrics.com
Uxmatters.com
Smartinsights.com
Econsultancy.com
Cutroni.com
www.GetMentalNotes.com
#12 : The Best Companies…
• Invest continually in analytics instrumentation, tools, people
• Use an Agile, iterative, cross-silo, one team project culture
• Prefer collaborative tools to having lots of meetings
• Prioritise development based on numbers and insight
• Practice real continuous product improvement, not SLEDD*
• Are fixing bugs, cruft, bad stuff as well as optimising
• Source photos and content that support persuasion and utility
• Have cross channel, cross device design, testing and QA
• Segment their data for valuable insights, every test or change
• Continually reduce cycle (iteration) time in their process
• Blend ‘long’ design, continuous improvement AND split tests
• Make optimisation the engine of change, not the slave of ego
* Single Large Expensive Doomed Developments
THE FUTURE OF TESTING :
www.conductrics.com

Weitere ähnliche Inhalte

Was ist angesagt?

Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015Craig Sullivan
 
Web Analytics Wednesday - Session Replay Tools are Vital
Web Analytics Wednesday - Session Replay Tools are VitalWeb Analytics Wednesday - Session Replay Tools are Vital
Web Analytics Wednesday - Session Replay Tools are VitalCraig Sullivan
 
Product design is Poo - And how to fix it!
Product design is Poo - And how to fix it!Product design is Poo - And how to fix it!
Product design is Poo - And how to fix it!Craig Sullivan
 
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...Craig Sullivan
 
Why do my AB tests suck? measurecamp
Why do my AB tests suck?   measurecampWhy do my AB tests suck?   measurecamp
Why do my AB tests suck? measurecampCraig Sullivan
 
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeBrighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeCraig Sullivan
 
Mobile presentation - Sydney Online Retailer - 26 Sep 2011
Mobile presentation - Sydney Online Retailer - 26 Sep 2011Mobile presentation - Sydney Online Retailer - 26 Sep 2011
Mobile presentation - Sydney Online Retailer - 26 Sep 2011Craig Sullivan
 
Cross Device Optimisation - Google Analytics Shortcuts
Cross Device Optimisation - Google Analytics ShortcutsCross Device Optimisation - Google Analytics Shortcuts
Cross Device Optimisation - Google Analytics ShortcutsCraig Sullivan
 
20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion Conference20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion ConferenceCraig Sullivan
 
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to meDigital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to meCraig Sullivan
 
Product Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to dieProduct Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to dieCraig Sullivan
 
Onboard like a juggernaut - Elite camp 2015
Onboard like a juggernaut - Elite camp 2015Onboard like a juggernaut - Elite camp 2015
Onboard like a juggernaut - Elite camp 2015Conversionista
 
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014Marketing Festival
 
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014Craig Sullivan
 
eMetrics London - The AB Testing Hype Cycle
eMetrics London - The AB Testing Hype CycleeMetrics London - The AB Testing Hype Cycle
eMetrics London - The AB Testing Hype CycleCraig Sullivan
 
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CROUXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CROCraig Sullivan
 
Natural born conversion killers - Conversion Jam
Natural born conversion killers - Conversion JamNatural born conversion killers - Conversion Jam
Natural born conversion killers - Conversion JamCraig Sullivan
 
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...PRWD
 
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not makingSAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not makingZack Notes
 
Talks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet ScaleTalks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet Scalecourseratalks
 

Was ist angesagt? (20)

Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
 
Web Analytics Wednesday - Session Replay Tools are Vital
Web Analytics Wednesday - Session Replay Tools are VitalWeb Analytics Wednesday - Session Replay Tools are Vital
Web Analytics Wednesday - Session Replay Tools are Vital
 
Product design is Poo - And how to fix it!
Product design is Poo - And how to fix it!Product design is Poo - And how to fix it!
Product design is Poo - And how to fix it!
 
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
 
Why do my AB tests suck? measurecamp
Why do my AB tests suck?   measurecampWhy do my AB tests suck?   measurecamp
Why do my AB tests suck? measurecamp
 
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeBrighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
 
Mobile presentation - Sydney Online Retailer - 26 Sep 2011
Mobile presentation - Sydney Online Retailer - 26 Sep 2011Mobile presentation - Sydney Online Retailer - 26 Sep 2011
Mobile presentation - Sydney Online Retailer - 26 Sep 2011
 
Cross Device Optimisation - Google Analytics Shortcuts
Cross Device Optimisation - Google Analytics ShortcutsCross Device Optimisation - Google Analytics Shortcuts
Cross Device Optimisation - Google Analytics Shortcuts
 
20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion Conference20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion Conference
 
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to meDigital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
 
Product Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to dieProduct Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to die
 
Onboard like a juggernaut - Elite camp 2015
Onboard like a juggernaut - Elite camp 2015Onboard like a juggernaut - Elite camp 2015
Onboard like a juggernaut - Elite camp 2015
 
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
 
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
 
eMetrics London - The AB Testing Hype Cycle
eMetrics London - The AB Testing Hype CycleeMetrics London - The AB Testing Hype Cycle
eMetrics London - The AB Testing Hype Cycle
 
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CROUXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
 
Natural born conversion killers - Conversion Jam
Natural born conversion killers - Conversion JamNatural born conversion killers - Conversion Jam
Natural born conversion killers - Conversion Jam
 
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
 
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not makingSAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
 
Talks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet ScaleTalks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet Scale
 

Andere mochten auch

UX and copywriting - Blend Conference
UX and copywriting - Blend ConferenceUX and copywriting - Blend Conference
UX and copywriting - Blend Conferencepaulin.dementhon
 
Human-Centered Copywriting: How Your Words Can Make or Break Your User Experi...
Human-Centered Copywriting: How Your Words Can Make or Break Your User Experi...Human-Centered Copywriting: How Your Words Can Make or Break Your User Experi...
Human-Centered Copywriting: How Your Words Can Make or Break Your User Experi...UserTesting
 
UX/CASE STUDY-STYLE COPYWRITING: Product Buying Guides
UX/CASE STUDY-STYLE COPYWRITING: Product Buying GuidesUX/CASE STUDY-STYLE COPYWRITING: Product Buying Guides
UX/CASE STUDY-STYLE COPYWRITING: Product Buying GuidesAdam Stanley
 
USECON RoX 2015: Slip into your customers' shoes - Mobile Ethnography
USECON RoX 2015: Slip into your customers' shoes - Mobile EthnographyUSECON RoX 2015: Slip into your customers' shoes - Mobile Ethnography
USECON RoX 2015: Slip into your customers' shoes - Mobile EthnographyUSECON
 
UX for Apps - GDG at BUET 9 Nov 2013 Masrur Hannan and Shadman Rahman
UX for Apps - GDG at BUET 9 Nov 2013 Masrur Hannan and Shadman RahmanUX for Apps - GDG at BUET 9 Nov 2013 Masrur Hannan and Shadman Rahman
UX for Apps - GDG at BUET 9 Nov 2013 Masrur Hannan and Shadman RahmanMasrur Hannan
 
How to Create Attention Grabbing Headlines
How to Create Attention Grabbing HeadlinesHow to Create Attention Grabbing Headlines
How to Create Attention Grabbing HeadlinesGwyneth Orford
 
October 2015 headlines bar graph outline
October 2015 headlines bar graph outlineOctober 2015 headlines bar graph outline
October 2015 headlines bar graph outlineIsaac Hart
 
Webinar: How to Write Headlines for the Web
Webinar: How to Write Headlines for the WebWebinar: How to Write Headlines for the Web
Webinar: How to Write Headlines for the WebEric Athas
 
Nathalie Nahai - 5 psychological principles of persuasive design
Nathalie Nahai - 5 psychological principles of persuasive designNathalie Nahai - 5 psychological principles of persuasive design
Nathalie Nahai - 5 psychological principles of persuasive designNathalie Nahai
 
The psychology of successful ecommerce
The psychology of successful ecommerceThe psychology of successful ecommerce
The psychology of successful ecommerceNathalie Nahai
 
Breaking: How Native Advertising Makes Headlines
Breaking: How Native Advertising Makes HeadlinesBreaking: How Native Advertising Makes Headlines
Breaking: How Native Advertising Makes HeadlinesHawkeye Thompson
 
Beyond SEO: copywriting for professionals
Beyond SEO: copywriting for professionalsBeyond SEO: copywriting for professionals
Beyond SEO: copywriting for professionalsJoost de Valk
 
Why User Experience Matters | By UX Professionals from Centerline Digital
Why User Experience Matters | By UX Professionals from Centerline DigitalWhy User Experience Matters | By UX Professionals from Centerline Digital
Why User Experience Matters | By UX Professionals from Centerline DigitalCenterline Digital
 
Copywriting Strategies for Better UX
Copywriting Strategies for Better UXCopywriting Strategies for Better UX
Copywriting Strategies for Better UXTracey John
 
The secret psychology behind social behaviour
The secret psychology behind social behaviourThe secret psychology behind social behaviour
The secret psychology behind social behaviourNathalie Nahai
 

Andere mochten auch (15)

UX and copywriting - Blend Conference
UX and copywriting - Blend ConferenceUX and copywriting - Blend Conference
UX and copywriting - Blend Conference
 
Human-Centered Copywriting: How Your Words Can Make or Break Your User Experi...
Human-Centered Copywriting: How Your Words Can Make or Break Your User Experi...Human-Centered Copywriting: How Your Words Can Make or Break Your User Experi...
Human-Centered Copywriting: How Your Words Can Make or Break Your User Experi...
 
UX/CASE STUDY-STYLE COPYWRITING: Product Buying Guides
UX/CASE STUDY-STYLE COPYWRITING: Product Buying GuidesUX/CASE STUDY-STYLE COPYWRITING: Product Buying Guides
UX/CASE STUDY-STYLE COPYWRITING: Product Buying Guides
 
USECON RoX 2015: Slip into your customers' shoes - Mobile Ethnography
USECON RoX 2015: Slip into your customers' shoes - Mobile EthnographyUSECON RoX 2015: Slip into your customers' shoes - Mobile Ethnography
USECON RoX 2015: Slip into your customers' shoes - Mobile Ethnography
 
UX for Apps - GDG at BUET 9 Nov 2013 Masrur Hannan and Shadman Rahman
UX for Apps - GDG at BUET 9 Nov 2013 Masrur Hannan and Shadman RahmanUX for Apps - GDG at BUET 9 Nov 2013 Masrur Hannan and Shadman Rahman
UX for Apps - GDG at BUET 9 Nov 2013 Masrur Hannan and Shadman Rahman
 
How to Create Attention Grabbing Headlines
How to Create Attention Grabbing HeadlinesHow to Create Attention Grabbing Headlines
How to Create Attention Grabbing Headlines
 
October 2015 headlines bar graph outline
October 2015 headlines bar graph outlineOctober 2015 headlines bar graph outline
October 2015 headlines bar graph outline
 
Webinar: How to Write Headlines for the Web
Webinar: How to Write Headlines for the WebWebinar: How to Write Headlines for the Web
Webinar: How to Write Headlines for the Web
 
Nathalie Nahai - 5 psychological principles of persuasive design
Nathalie Nahai - 5 psychological principles of persuasive designNathalie Nahai - 5 psychological principles of persuasive design
Nathalie Nahai - 5 psychological principles of persuasive design
 
The psychology of successful ecommerce
The psychology of successful ecommerceThe psychology of successful ecommerce
The psychology of successful ecommerce
 
Breaking: How Native Advertising Makes Headlines
Breaking: How Native Advertising Makes HeadlinesBreaking: How Native Advertising Makes Headlines
Breaking: How Native Advertising Makes Headlines
 
Beyond SEO: copywriting for professionals
Beyond SEO: copywriting for professionalsBeyond SEO: copywriting for professionals
Beyond SEO: copywriting for professionals
 
Why User Experience Matters | By UX Professionals from Centerline Digital
Why User Experience Matters | By UX Professionals from Centerline DigitalWhy User Experience Matters | By UX Professionals from Centerline Digital
Why User Experience Matters | By UX Professionals from Centerline Digital
 
Copywriting Strategies for Better UX
Copywriting Strategies for Better UXCopywriting Strategies for Better UX
Copywriting Strategies for Better UX
 
The secret psychology behind social behaviour
The secret psychology behind social behaviourThe secret psychology behind social behaviour
The secret psychology behind social behaviour
 

Ähnlich wie Myths, Lies and Illusions of AB and Split Testing

AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...Northern User Experience
 
John lewis lego craig sullivan 1335
John lewis lego craig sullivan  1335John lewis lego craig sullivan  1335
John lewis lego craig sullivan 1335Charlie Lines
 
Losing is the New Winning
Losing is the New WinningLosing is the New Winning
Losing is the New WinningOptimizely
 
Fail and Win: Why a Failed Test Isn’t a Bad Thing
Fail and Win: Why a Failed Test Isn’t a Bad ThingFail and Win: Why a Failed Test Isn’t a Bad Thing
Fail and Win: Why a Failed Test Isn’t a Bad ThingOptimizely
 
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9Craig Sullivan
 
Patrick McKenzie Opticon 2014: Advanced A/B Testing
Patrick McKenzie Opticon 2014: Advanced A/B TestingPatrick McKenzie Opticon 2014: Advanced A/B Testing
Patrick McKenzie Opticon 2014: Advanced A/B TestingPatrick McKenzie
 
Conversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynoteConversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynoteWebanalisten .nl
 
Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel ...
Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel ...Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel ...
Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel ...Webanalisten .nl
 
Google Analytics Powerups and Smartcuts
Google Analytics Powerups and Smartcuts Google Analytics Powerups and Smartcuts
Google Analytics Powerups and Smartcuts Charles Meaden
 
Master Conversion Optimization
Master Conversion OptimizationMaster Conversion Optimization
Master Conversion OptimizationAustin Walker
 
Construction Future Wales Performance Management (Benchmarking) 2016
Construction Future Wales Performance Management (Benchmarking) 2016Construction Future Wales Performance Management (Benchmarking) 2016
Construction Future Wales Performance Management (Benchmarking) 2016Rae Davies
 
Conversion Rate Optimisation: The Science Behind Turning Visitors Into Customers
Conversion Rate Optimisation: The Science Behind Turning Visitors Into CustomersConversion Rate Optimisation: The Science Behind Turning Visitors Into Customers
Conversion Rate Optimisation: The Science Behind Turning Visitors Into CustomersWeb Marketing ROI
 
How to Turn Your Optimization Team into a Revenue Doubling Machine
How to Turn Your Optimization Team into a Revenue Doubling MachineHow to Turn Your Optimization Team into a Revenue Doubling Machine
How to Turn Your Optimization Team into a Revenue Doubling MachineOptimizely
 
Getting Started with Experimentation
Getting Started with ExperimentationGetting Started with Experimentation
Getting Started with ExperimentationOptimizely
 
How to Calculate the Financial Impact of OEE
How to Calculate the Financial Impact of OEEHow to Calculate the Financial Impact of OEE
How to Calculate the Financial Impact of OEESafetyChain Software
 
Anton Muzhailo - Practical Test Process Improvement using ISTQB
Anton Muzhailo - Practical Test Process Improvement using ISTQBAnton Muzhailo - Practical Test Process Improvement using ISTQB
Anton Muzhailo - Practical Test Process Improvement using ISTQBIevgenii Katsan
 
Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experienc...
Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experienc...Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experienc...
Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experienc...Optimizely
 

Ähnlich wie Myths, Lies and Illusions of AB and Split Testing (20)

AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
 
John lewis lego craig sullivan 1335
John lewis lego craig sullivan  1335John lewis lego craig sullivan  1335
John lewis lego craig sullivan 1335
 
Losing is the New Winning
Losing is the New WinningLosing is the New Winning
Losing is the New Winning
 
Fail and Win: Why a Failed Test Isn’t a Bad Thing
Fail and Win: Why a Failed Test Isn’t a Bad ThingFail and Win: Why a Failed Test Isn’t a Bad Thing
Fail and Win: Why a Failed Test Isn’t a Bad Thing
 
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
 
Patrick McKenzie Opticon 2014: Advanced A/B Testing
Patrick McKenzie Opticon 2014: Advanced A/B TestingPatrick McKenzie Opticon 2014: Advanced A/B Testing
Patrick McKenzie Opticon 2014: Advanced A/B Testing
 
Conversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynoteConversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynote
 
The Finishing Line
The Finishing LineThe Finishing Line
The Finishing Line
 
Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel ...
Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel ...Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel ...
Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel ...
 
Google Analytics Powerups and Smartcuts
Google Analytics Powerups and Smartcuts Google Analytics Powerups and Smartcuts
Google Analytics Powerups and Smartcuts
 
Master Conversion Optimization
Master Conversion OptimizationMaster Conversion Optimization
Master Conversion Optimization
 
Construction Future Wales Performance Management (Benchmarking) 2016
Construction Future Wales Performance Management (Benchmarking) 2016Construction Future Wales Performance Management (Benchmarking) 2016
Construction Future Wales Performance Management (Benchmarking) 2016
 
Conversion Rate Optimisation: The Science Behind Turning Visitors Into Customers
Conversion Rate Optimisation: The Science Behind Turning Visitors Into CustomersConversion Rate Optimisation: The Science Behind Turning Visitors Into Customers
Conversion Rate Optimisation: The Science Behind Turning Visitors Into Customers
 
Andys guideto pm
Andys guideto pmAndys guideto pm
Andys guideto pm
 
Lean startup
Lean startup Lean startup
Lean startup
 
How to Turn Your Optimization Team into a Revenue Doubling Machine
How to Turn Your Optimization Team into a Revenue Doubling MachineHow to Turn Your Optimization Team into a Revenue Doubling Machine
How to Turn Your Optimization Team into a Revenue Doubling Machine
 
Getting Started with Experimentation
Getting Started with ExperimentationGetting Started with Experimentation
Getting Started with Experimentation
 
How to Calculate the Financial Impact of OEE
How to Calculate the Financial Impact of OEEHow to Calculate the Financial Impact of OEE
How to Calculate the Financial Impact of OEE
 
Anton Muzhailo - Practical Test Process Improvement using ISTQB
Anton Muzhailo - Practical Test Process Improvement using ISTQBAnton Muzhailo - Practical Test Process Improvement using ISTQB
Anton Muzhailo - Practical Test Process Improvement using ISTQB
 
Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experienc...
Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experienc...Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experienc...
Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experienc...
 

Mehr von Craig Sullivan

Why does my Mobile Conversion rate suck? 19 Sep 2013 @ Conversion Thursday #...
Why does my Mobile Conversion rate suck?  19 Sep 2013 @ Conversion Thursday #...Why does my Mobile Conversion rate suck?  19 Sep 2013 @ Conversion Thursday #...
Why does my Mobile Conversion rate suck? 19 Sep 2013 @ Conversion Thursday #...Craig Sullivan
 
Toolkits and tips of the conversion pros v 1.6
Toolkits and tips of the conversion pros v 1.6Toolkits and tips of the conversion pros v 1.6
Toolkits and tips of the conversion pros v 1.6Craig Sullivan
 
Elite Camp 2013 - Estonia
Elite Camp 2013 - EstoniaElite Camp 2013 - Estonia
Elite Camp 2013 - EstoniaCraig Sullivan
 
Conversionista : Conversion manager course - Stockholm 20 march 2013
Conversionista : Conversion manager course  - Stockholm 20 march 2013Conversionista : Conversion manager course  - Stockholm 20 march 2013
Conversionista : Conversion manager course - Stockholm 20 march 2013Craig Sullivan
 
3 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
3 Optimisation Decks : WAW Copenhagen - 27 Feb 20133 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
3 Optimisation Decks : WAW Copenhagen - 27 Feb 2013Craig Sullivan
 
Measure camp tools of the cro rabble
Measure camp   tools of the cro rabbleMeasure camp   tools of the cro rabble
Measure camp tools of the cro rabbleCraig Sullivan
 
5 cro tools that i can't live without
5 cro tools that i can't live without5 cro tools that i can't live without
5 cro tools that i can't live withoutCraig Sullivan
 
eMetrics Stockholm - What the F*** is wrong with my conversion?
eMetrics Stockholm - What the F*** is wrong with my conversion?eMetrics Stockholm - What the F*** is wrong with my conversion?
eMetrics Stockholm - What the F*** is wrong with my conversion?Craig Sullivan
 

Mehr von Craig Sullivan (8)

Why does my Mobile Conversion rate suck? 19 Sep 2013 @ Conversion Thursday #...
Why does my Mobile Conversion rate suck?  19 Sep 2013 @ Conversion Thursday #...Why does my Mobile Conversion rate suck?  19 Sep 2013 @ Conversion Thursday #...
Why does my Mobile Conversion rate suck? 19 Sep 2013 @ Conversion Thursday #...
 
Toolkits and tips of the conversion pros v 1.6
Toolkits and tips of the conversion pros v 1.6Toolkits and tips of the conversion pros v 1.6
Toolkits and tips of the conversion pros v 1.6
 
Elite Camp 2013 - Estonia
Elite Camp 2013 - EstoniaElite Camp 2013 - Estonia
Elite Camp 2013 - Estonia
 
Conversionista : Conversion manager course - Stockholm 20 march 2013
Conversionista : Conversion manager course  - Stockholm 20 march 2013Conversionista : Conversion manager course  - Stockholm 20 march 2013
Conversionista : Conversion manager course - Stockholm 20 march 2013
 
3 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
3 Optimisation Decks : WAW Copenhagen - 27 Feb 20133 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
3 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
 
Measure camp tools of the cro rabble
Measure camp   tools of the cro rabbleMeasure camp   tools of the cro rabble
Measure camp tools of the cro rabble
 
5 cro tools that i can't live without
5 cro tools that i can't live without5 cro tools that i can't live without
5 cro tools that i can't live without
 
eMetrics Stockholm - What the F*** is wrong with my conversion?
eMetrics Stockholm - What the F*** is wrong with my conversion?eMetrics Stockholm - What the F*** is wrong with my conversion?
eMetrics Stockholm - What the F*** is wrong with my conversion?
 

Kürzlich hochgeladen

办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...vmzoxnx5
 
IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119
IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119
IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119APNIC
 
Tari Eason Warriors Come Out To Play T Shirts
Tari Eason Warriors Come Out To Play T ShirtsTari Eason Warriors Come Out To Play T Shirts
Tari Eason Warriors Come Out To Play T Shirtsrahman018755
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119APNIC
 
Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...
Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...
Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...Andreas Sfakianakis
 
Summary ID-IGF 2016 National Dialogue - English (tata kelola internet / int...
Summary  ID-IGF 2016 National Dialogue  - English (tata kelola internet / int...Summary  ID-IGF 2016 National Dialogue  - English (tata kelola internet / int...
Summary ID-IGF 2016 National Dialogue - English (tata kelola internet / int...ICT Watch - Indonesia
 
Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119
Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119
Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119APNIC
 
draft-harrison-sidrops-manifest-number-01, presented at IETF 119
draft-harrison-sidrops-manifest-number-01, presented at IETF 119draft-harrison-sidrops-manifest-number-01, presented at IETF 119
draft-harrison-sidrops-manifest-number-01, presented at IETF 119APNIC
 
2024_hackersuli_mobil_ios_android ______
2024_hackersuli_mobil_ios_android ______2024_hackersuli_mobil_ios_android ______
2024_hackersuli_mobil_ios_android ______hackersuli
 
Basic Security.pptx is a awsome PPT on your mobiel
Basic Security.pptx is a awsome PPT on your mobielBasic Security.pptx is a awsome PPT on your mobiel
Basic Security.pptx is a awsome PPT on your mobielpratamakiki860
 
Is DNS ready for IPv6, presented by Geoff Huston at IETF 119
Is DNS ready for IPv6, presented by Geoff Huston at IETF 119Is DNS ready for IPv6, presented by Geoff Huston at IETF 119
Is DNS ready for IPv6, presented by Geoff Huston at IETF 119APNIC
 
Summary IGF 2013 Bali - English (tata kelola internet / internet governance)
Summary  IGF 2013 Bali - English (tata kelola internet / internet governance)Summary  IGF 2013 Bali - English (tata kelola internet / internet governance)
Summary IGF 2013 Bali - English (tata kelola internet / internet governance)ICT Watch - Indonesia
 
Power of Social Media for E-commerce.pdf
Power of Social Media for E-commerce.pdfPower of Social Media for E-commerce.pdf
Power of Social Media for E-commerce.pdfrajats19920
 

Kürzlich hochgeladen (13)

办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
 
IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119
IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119
IPv6 Operational Issues (with DNS), presented by Geoff Huston at IETF 119
 
Tari Eason Warriors Come Out To Play T Shirts
Tari Eason Warriors Come Out To Play T ShirtsTari Eason Warriors Come Out To Play T Shirts
Tari Eason Warriors Come Out To Play T Shirts
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119
 
Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...
Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...
Cyber Shield Up - They Shall Not Pass - Andreas Sfakianakis - Lecture at CSD ...
 
Summary ID-IGF 2016 National Dialogue - English (tata kelola internet / int...
Summary  ID-IGF 2016 National Dialogue  - English (tata kelola internet / int...Summary  ID-IGF 2016 National Dialogue  - English (tata kelola internet / int...
Summary ID-IGF 2016 National Dialogue - English (tata kelola internet / int...
 
Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119
Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119
Making an RFC in Today's IETF, presented by Geoff Huston at IETF 119
 
draft-harrison-sidrops-manifest-number-01, presented at IETF 119
draft-harrison-sidrops-manifest-number-01, presented at IETF 119draft-harrison-sidrops-manifest-number-01, presented at IETF 119
draft-harrison-sidrops-manifest-number-01, presented at IETF 119
 
2024_hackersuli_mobil_ios_android ______
2024_hackersuli_mobil_ios_android ______2024_hackersuli_mobil_ios_android ______
2024_hackersuli_mobil_ios_android ______
 
Basic Security.pptx is a awsome PPT on your mobiel
Basic Security.pptx is a awsome PPT on your mobielBasic Security.pptx is a awsome PPT on your mobiel
Basic Security.pptx is a awsome PPT on your mobiel
 
Is DNS ready for IPv6, presented by Geoff Huston at IETF 119
Is DNS ready for IPv6, presented by Geoff Huston at IETF 119Is DNS ready for IPv6, presented by Geoff Huston at IETF 119
Is DNS ready for IPv6, presented by Geoff Huston at IETF 119
 
Summary IGF 2013 Bali - English (tata kelola internet / internet governance)
Summary  IGF 2013 Bali - English (tata kelola internet / internet governance)Summary  IGF 2013 Bali - English (tata kelola internet / internet governance)
Summary IGF 2013 Bali - English (tata kelola internet / internet governance)
 
Power of Social Media for E-commerce.pdf
Power of Social Media for E-commerce.pdfPower of Social Media for E-commerce.pdf
Power of Social Media for E-commerce.pdf
 

Myths, Lies and Illusions of AB and Split Testing

  • 2. @OptimiseOrDie • UX, Analytics, Split Testing and Growth Rate Optimisation • Started doing testing & CRO 2004 • Split tested over 45M visitors in 19 languages • 67+ mistakes I MADE with AB testing • Like riding a bike… • Optimise your optimisation? Get in touch!
  • 3.
  • 4. Myths, Lies and Illusions of Optimisation 1. Optimisation and Testing are the same 2. Optimisation and Testing are easy 3. Statistical significance tells you when to stop 4. High traffic always makes tests shorter 5. It’s all about Conversion Rates 6. You don’t need to test the test 7. Your tests will give you the quoted lift 8. High traffic pages give the best tests 9. It’s about getting all your tests to win 10. Optimisation is only about websites @OptimiseOrDie 11. Segmentation will tell you what happened 12. You can’t run multiple simultaneous tests 13. Testing is great for settling arguments 14. You can spot trends early on in a test 15. More test volumes = better results 16. Your tests tell you truths that last forever 17. You can test even on low traffic sites 18. Other people’s tests are ‘Best Practice’ 19. Doesn’t involve changing the way you work 20. Testing makes you a Data Scientist
  • 10. @OptimiseOrDie1. Testing & Optimisation are the same thing Testin g Optimise
  • 11. What is CRO/Optimisation? • “Using Analytics data and Customer feedback to improve the performance of your website.” • “Finding out why visitors aren’t converting and then resolving these issues.” • “Running loads of crappy split tests randomly until the heat death of the universe” @OptimiseOrDie
  • 12. What is my definition? “A structured, systematic and continuous application of techniques that are used to discover, quantify and prioritise issues. These can be turned into hypotheses to drive experiments and opportunity in the following business outcomes:” • Increased revenue or profitability • Increasing LTV, loyalty, NPS/Sat scores, Kristoffer Potential • Removing cost from the business or contact center • Higher productivity or labour flexibility • Delighting customers • Reduced development effort @OptimiseOrDie
  • 13. What Optimisation is NOT! • A way to change things you (or others) hate • A methodology for running split tests • A guarantee of increased conversion • A methodology for looking at analytics data • A rescue for silo-bound or non-agile design or development processes • A way to trick people into buying • A bolt-on – it IS the process @OptimiseOrDie
  • 14. Optimisation is a: • Way of joining the worlds of Customer insight, UX, Analytics, Split testing and Business Strategy • Overarching Design and development process which prioritises work around opportunity • Strategic, not tactical, response to wasted development effort or product change • Way to create a deep and meaningful connection between the team, customers, business and the outcomes of making product changes • The killer app to remove ego, opinion, assumptions, cherished notions or ‘we just do it that way’ from decision making. • More powerful method than UX research or analytics alone, in guiding the directionality of product change @OptimiseOrDie
  • 15. Optimisation includes these: • Qualitative research • Analytics, Quant analysis and insight • UX inspection and discovery • Competitive Intelligence • Priority based opportunity • VOC, Surveys and Customer Satisfaction (NPS) • Call tracking & Call centre optimisation • AB testing • Multivariate testing • Photography optimisation • EEG / ECG / Galvanic response @OptimiseOrDie • Web performance tuning • Forms analytics • Eye tracking • Market research • Big & Unstructured data analysis • PPC optimisation • Session replay analysis • Customer journey mapping • Ethnographic (diary) study research • Cross device, platform and channel insight • Email optimisation
  • 16. 2. Optimisation & Testing are EASY
  • 17. Surely you just add Javascript? • This work is harder than anything I’ve ever done! • You all have expensive & limited resources for testing - like Airport take-off slots • You MUST make use of these efficiently • You HAVE to balance resource cost with opportunity by prioritising carefully @OptimiseOrDie • You’re doing the equivalent of drug trials! • For large OR small companies, the instrumentation, analytics, tools setup, plan design, test methodology and analysis cannot be ‘done later on’. It’s not complex but it is vital. • The best companies have a structural shell (process, methodology, management) around their activities • If you’re not optimising the process continuously (Kaizen), you won’t increase your velocity of iterations. • Optimise the Optimisation
  • 18. 3. Statistical significance tells you when to stop @OptimiseOrDie
  • 19. The 95% Stopping Problem @OptimiseOrDie • Many people use 95, 99% ‘confidence’ to stop • This value is unreliable and moves around • Nearly all my tests reach significance before they are actually ready • You can hit 95% early in a test (18 minutes!) • If you stop, it could be a false result • Read this Nature article : bit.ly/1dwk0if • Optimizely have changed their stats engine • This 95% thingy – must be LAST on your stop list • Let me explain
  • 20. The 95% Stopping Problem @OptimiseOrDie Scenario 1 Scenario 2 Scenario 3 Scenario 4 After 200 observations Insignificant Insignificant Significant! Significant! After 500 observations Insignificant Significant! Insignificant Significant! End of experiment Insignificant Significant! Insignificant Significant! “You should know that stopping a test once it’s significant is deadly sin number 1 in A/B testing land. 77% of A/A tests (testing the same thing as A and B) will reach significance at a certain point.” Ton Wesseling, Online Dialogue
  • 21. The 95% Stopping Problem @OptimiseOrDie “Statistical Significance does not equal Validity” http://bit.ly/1wMfmY2 “Why every Internet Marketer should be a Statistician” http://bit.ly/1wMfs1G “Understanding the Cycles in your site” http://mklnd.com/1pGSOUP
  • 22. 4. High Traffic always means short tests @OptimiseOrDie
  • 23. Business & Purchase Cycles @OptimiseOrDie • Customers change • Your traffic mix changes • Markets, competitors • Be aware of all the waves • Always test whole cycles • Don’t exclude slower buyers • When you stop, let test subjects still complete! Start Test Finish Avg Cycle
  • 24. • TWO BUSINESS CYCLES minimum (week/mo) • 1 PURCHASE CYCLE minimum • 250 CONVERSIONS minimum per creative (e.g. checkouts) • 350 & MORE! if response is very similar • FULL WEEKS/CYCLES never part of one • KNOW what marketing, competitors and cycles are doing • RUN a test length calculator - bit.ly/XqCxuu • SET your test run time , RUN IT, STOP IT, ANALYSE IT • ONLY RUN LONGER if you need more data • DON’T RUN LONGER just because the test isn’t giving the result you want! @OptimiseOrDie How Long? Simple Rules to follow
  • 25. 5. It’s ALL about Conversion Rate @OptimiseOrDie
  • 26. It’s all about the business @OptimiseOrDie • You’re optimising a business here, not a page or site • Tricking, pushing or persuading people at a superficial level to take an action is not a viable strategy • Your optimisation strategy is a series of steps, not a tool • Testing is about learning, not converting. • Tests that fail to tell you anything (regardless of outcome) are a failure themselves • If you don’t shift the business goals, your optimisation and testing budget will be threatened
  • 27. 6. You don’t need to test the test – just go @OptimiseOrDie Browser testing www.crossbrowsertesting.com www.browserstack.com www.spoon.net www.saucelabs.com www.multibrowserviewer.com Mobile devices www.appthwack.com www.deviceanywhere.com www.opendevicelab.com Read this article bit.ly/1wBccsJ
  • 28. 7. The test result gives the promised lift @OptimiseOrDie
  • 29. The result is a range @OptimiseOrDie • Version A is 3% conversion • Version B is 4% conversion • Yay! That’s a 25% lift • Let’s tell everyone • When it goes live, you get 5.7% • That’s because it was A RANGE • 3% +/- 0.5 • 4% +/- 0.4 • Actual result was 3.5% for A • Actual result was 3.7% for B
  • 30. AB Testing Visualisation Tool @OptimiseOrDie abtestguide.com/calc/
  • 31. 8. Testing is best on high traffic pages @OptimiseOrDie Think like the CEO of a department store! If you can’t refurbish the entire store, which floors or departments will you invest in optimising? Wherever there is: • Footfall • Low return • Opportunity
  • 33. @OptimiseOrDie 8. It’s all about prioritisation!
  • 34. 9. It’s all about WINNING test results @OptimiseOrDie
  • 35. Failing is good @OptimiseOrDie • Tests that are ‘about the same’ are a failure • They’re also very hard to call • That means you have to be BOLD not conservative • A test that comes out negative is NOT a failure • If a ‘negative’ test teaches you something, it’s a success! • If you hit 40/50/60% failed tests, that’s fine • If you aren’t failing regularly, you’re not BOLD enough • Success is about the number of tests you finish each month, and what you learn from them
  • 36. We believe that doing [A] for People [B] will make outcome [C] happen. We’ll know this when we observe data [D] and obtain feedback [E]. (reverse) @OptimiseOrDie
  • 37. 10. Optimisation is only for Websites @OptimiseOrDie
  • 38. Optimisation is just for websites @OptimiseOrDie • Service Design (Airbnb) • Onboarding flows with emails • Email templates • Apps – testing, debugging, tracking • Phone tracking and call centre optimisation • Social, Display, TV, Video and other advertising • Print adverts, Direct Mail • In-store promotions • Product manuals, guides, interfaces • EVERYTHING has elasticity – just find it • Even Multi-variate call centre scripts
  • 39. 11. Segmentation will explain things @OptimiseOrDie
  • 40. Segmentation explains stuff @OptimiseOrDie • Beware of small sample sizes • A = 350 conversions • B = 300 conversions • A Conversions for Safari = 20 • B Conversions for Safari = 25 • Only needs 2 people to change that from 25% lift to 14%
  • 41. 12. You can’t run concurrent split tests @OptimiseOrDie
  • 42. Oh yes you can, with GA! @OptimiseOrDie • If you push events or variables into GA, you can report on behaviour for A or B (or any variations). • If you do it that way, you can easily run multiple tests on different page targets simultaneously. • You grab AAA, AAB, ABA, ABB and analyse. • Test subjects get a recipe of tests, so one caveat • If you pick things that clash or jar the experience • If you have $5 delivery but then $2 in another test • Apart from that, it’s fine to run these • They tell you about the right experience recipe across several pages being optimised in concert
  • 43. 13. Testing is great for settling arguments @OptimiseOrDie
  • 44. 14. You can spot trends early in a test @OptimiseOrDie
  • 45. You can spot trends early @OptimiseOrDie • Tests are volatile in the early stages • Watch but shut your mouth and wait • Keep an eye on any odd behaviour • Be patient!
  • 46. 15. More tests = Better results @OptimiseOrDie
  • 47. More tests = Better results @OptimiseOrDie • Increasing volume without optimising velocity • Not optimising the process means it scales badly • If you put garbage in, you’ll get garbage out • Ramping up before you have it tuned is crazy • Get the team, process, project management, methodology, toolkits, selling and PR nailed first • THEN optimise and scale • I know that one UK retailer is doing 140 per month! • That’s LOW compared to some companies I work with • Big companies doing 2 a month? Meh.
  • 48. 16. Each test is a truth FOREVER @OptimiseOrDie
  • 49. Tests are a Truth Forever @OptimiseOrDie • Traffic changes • Prices change • Product mix changes • Advertising evolves and changes • Markets are different • Customers have changed • Competitors or regulatory landscape moves • Things happen outside of your control • You need to revisit tests or ITERATE • Always be trying to beat the new winner, not basking in the glory of a test you ran 9 months ago • The lift may have vanished • Schrodinger’s AB test
  • 50. 17. Testing is fine on Low Traffic sites @OptimiseOrDie
  • 51. Testing is fine on low traffic sites @OptimiseOrDie • Yes, if you estimate your minimum testing unit • This is the time to AB test a sample of say 250 conversions, with the rules I set earlier. • A payday loan company? 2 months minimum! • Run the calculations • Check the test length • If it takes like, 8 million years, what can you do? • Read this or download my AB testing decks: bit.ly/1umy5Y6
  • 52. 18. Other people’s tests are Best Practice @OptimiseOrDie “STOP copying your competitors They may not know what the f*** they are doing either” Peep Laja, ConversionXL
  • 53. Tests you see online? @OptimiseOrDie • Your customers are not the same • Your site is not the same • Your advertising and traffic is not the same • Your UX is not the same • How the f*** do you think it guarantees a result? • Use them to inform or suggest ideas • They’re like the picture on meal packets • Serving Suggestion Only
  • 54. 19. Testing makes you a Data Scientist @OptimiseOrDie
  • 55. 20. Testing and Innovation is a plugin @OptimiseOrDie
  • 58. 2004 Headspace What I thought I knew in 2004 Reality
  • 59. 2014 Headspace What I KNOW I know Me, on a good day
  • 61. @OptimiseOrDie #1 Culture & Team #2 Toolkit & Analytics investment #3 UX, CX, Service Design, Insight #4 Persuasive Copywriting #5 Experimentation (testing) tools The 5 Legged Optimisation Barstool
  • 63. Business Future Testing? Congratulations! Today you’re the lucky winner of our random awards programme. You get all these extra features for free, on us. Enjoy. Mr D. Vader
  • 64. Thank You! Mail : sullivac@gmail.com Deck : Linkedin : linkd.in/pvrg14
  • 68. #5 : FIND STUFF @OptimiseOrDie @danbarker Analytics @fastbloke Analytics @timlb Analytics @jamesgurd Analytics @therustybear Analytics @carmenmardiros Analytics @davechaffey Analytics @priteshpatel9 Analytics @cutroni Analytics @avinash Analytics @Aschottmuller Analytics, CRO @cartmetrix Analytics, CRO @Kissmetrics CRO / UX @Unbounce CRO / UX @Morys CRO / Neuro @UXFeeds UX / Neuro @Psyblog Neuro @Gfiorelli1 SEO / Analytics @PeepLaja CRO @TheGrok CRO @UIE UX @LukeW UX / Forms @cjforms UX / Forms @axbom UX @iatv UX @Chudders Photo UX @JeffreyGroks Innovation @StephanieRieger Innovation @BrianSolis Innovation @DrEscotet Neuro @TheBrainLady Neuro @RogerDooley Neuro @Cugelman Neuro @Smashingmag Dev / UX @uxmag UX @Webtrends UX / CRO
  • 69. #5 : LEARN STUFF @OptimiseOrDie Baymard.com Lukew.com Smashingmagazine.com ConversionXL.com Medium.com Whichtestwon.com Unbounce.com Measuringusability.com RogerDooley.com Kissmetrics.com Uxmatters.com Smartinsights.com Econsultancy.com Cutroni.com www.GetMentalNotes.com
  • 70. #12 : The Best Companies… • Invest continually in analytics instrumentation, tools, people • Use an Agile, iterative, cross-silo, one team project culture • Prefer collaborative tools to having lots of meetings • Prioritise development based on numbers and insight • Practice real continuous product improvement, not SLEDD* • Are fixing bugs, cruft, bad stuff as well as optimising • Source photos and content that support persuasion and utility • Have cross channel, cross device design, testing and QA • Segment their data for valuable insights, every test or change • Continually reduce cycle (iteration) time in their process • Blend ‘long’ design, continuous improvement AND split tests • Make optimisation the engine of change, not the slave of ego * Single Large Expensive Doomed Developments
  • 71. THE FUTURE OF TESTING : www.conductrics.com

Hinweis der Redaktion

  1. This is the title of my talk today. Wonderful picture, isn’t it? It’s from an IBM advert from 1951 and is a great piece of work, especially the copywriting. The whole message here is “Buying an IBM computer gets you the same power as 150 extra engineers”. And not a feature in sight – the trick is they’re not selling the computer, they’re selling what the computer will do for your business and your life. And what am I talking about today? Well the fact that most split tests being run these days are just bullshit – the slide rules don’t add up for a lot of companies. Many C level execs I’ve spoken to complain about the variability of return or success on this kind of testing. There’s a reason for this -
  2. And here’s a boring slide about me – and where I’ve been driving over 400M of additional revenue in the last few years. For the sharp eyed amongst you, you’ll see that Lean UX hasn’t been around since 2008. Many startups and teams were doing this stuff before it got a new name, even if the approach was slightly different. For the last 4 years, I’ve been optimising sites using a blend of these techniques.
  3. And here are some of the clients I’ve been working for. Dull bit is now officially over.
  4. And don’t worry – if it’s not working for you – and looks like this, it’s OK – you’re just doing it the wrong way. Although I admire AB testing companies - all of them - for championing the right to test and making it easy for anyone to implement - there's a problem. Democratisation of testing brings with it a large chunk of stupidity too. When YouTube first appeared, did anyone think "Oh boy, there's only ever going to be high quality content to see on here. Seriously. No”
  5. And this crappy AB testing is basically the equivalent of funny cat videos
  6. People taking videos of themselves playing video games
  7. And like, wow, there are 6.9 million Gangnam Style videos. Just incredible. But hidden in those big numbers, YouTube will always have a tiny percentage of really great stuff, very little good stuff and a long tail of absolute bollocks. And the same is true of split testing - there's some really well run stuff, getting very good results and there's a lot of air guitar going on.
  8. So – the first myth. Some people think that testing and optimisation are the same thing. AB testing is just one of the techniques that I’ll use to optimise a business and there are many more. Let’s talk about definitions.
  9. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  10. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  11. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  12. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  13. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  14. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  15. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  16. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  17. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  18. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  19. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  20. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  21. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  22. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  23. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  24. It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  25. I once explained to my daughter – you know, when adults like look really in control and making decisions and appearing not to suffer from indecision? Don’t believe it for a minute – we’re just better at winging it cause we’re older. And this is the huge hole that’s gnawing at the hear of many digital operations. The inability to understand what you can and can’t be confident about – but nobody wants to admit they’re guessing a lot of the time.
  26. There is one answer to this trap I call taking a visit to Guessaholics Anonymous - to surrender to the higher power of testing and innovation by using consumer psychological insight and data to guide your hand. To recognise you’re powerless at deciding what’s best or second guessing what will win. It's actually liberating to not be sitting in a meeting room, arguing about the wording of a bloody button for 4 fucking hours, ever again.
  27. And this was the state of my head in 2004. The inability to understand what you can and can’t be confident about – but nobody wants to admit they’re fucking guessing a lot of the time. And it took me a long time to figure out I didn’t know anything really – it was all assumptions and cherished notions. It was pretty crushing to test my way to this realisation but MUCH I’m happier now.
  28. Now I think I know this much - but I might know a wee bit more than I think I do – but I’m erring on the side of caution. That’s because I'm always questioning everything I do through the lens of that consumer insight and testing. Without customers and data driven insights, you can’t shape revenue and delight. They’ll give you the very psychological insights you need to apply levers to influence them, if you only ask questions. Everything else is just a fucking guess. Even with tests, if the only inputs you’ve got are ego and opinion, they’re going to be lousy guesses and you’re wasting your experiments.
  29. And now a bit about something I call Rumsfeldian Space – exploring the unknowns. This is vital if you want to make your testing bold enough to get great results.
  30. You need to inhabit the contextual and emotional landscape of the consumer to really shape product or service experience. The only way to do this is have teams and cultures that create a direct and meaningful connection between teams and the customer, in the impact that every change has on the outcome. Every atom of every piece of copy, design, error message, email, website, support, help content, absolutely bloody everything you do - has to be framed within knowledge and empathy with the consumer fears, worries, barriers, pain but also the real problems we solve by designing products not as features but as life enhancing. And this is the best marketing of all, like the IBM ad. Business Model Optimisation requires a watchmakers eye – a complete understanding of the watch from macro to micro - the flow of delight and money that can be shaped inside every customer experience, website, and interaction - at a component and a service design level. Most people have 1 or 2 legs at most. The best companies I've worked with are doing all of these.
  31. Darwin did NOT say 'survival of the fittest' – that was actually another guy called Herbert Spencer. What Darwin actually pushed was that the key ingredients were heritability of traits, variation and selection based on survival. If only your marketing programme was quite as ruthless eh? And if you want variation and innovation, the survival of good ideas in favour of bad and knowledge that you pass on – you need a culture of adaptability, improvement and change. Agile is about a shared mind-set across managers, leaders and everyone in the team. There’s a Harvard survey about how the *most* productive teams communicate. Not in meetings but all the time - deskside, IM, phone, skype, GitHub, agile tools, apps - these are the telegraph wires of the collaborative, participative and mission oriented teams. My key insight of the last 10 years in building and leading teams is that agile, open, flat, cross-silo, participative, flexible and collaborative environments produce customer connected products of high quality. Autoglass NPS higher than Apple.
  32. I hope you enjoyed it as much as I did writing it. All my details are here and slides will be uploaded shortly. Thank you for your time today.
  33. So – what’s driving this change then? Well there have been great books on selling and persuading people – all the way back to ‘Scientific Advertising’ in 1923. And my favourite here is the Cialdini work – simply because it’s a great help for people to find practical uses for these techniques. I’ve also included some analytics and testing books here – primarily because they help so MUCH in augmenting our customer insight, testing and measurement efforts. There are lots of books with really cool examples, great stories and absolutely no fucking useful information you can use on your website – if you’ve read some of these, you’ll know exactly what I mean. These are the tomes I got most practical use from and I’d recommend you buy the whole lot – worth every penny.
  34. So – what’s driving this change then? Well there have been great books on selling and persuading people – all the way back to ‘Scientific Advertising’ in 1923. And my favourite here is the Cialdini work – simply because it’s a great help for people to find practical uses for these techniques. I’ve also included some analytics and testing books here – primarily because they help so MUCH in augmenting our customer insight, testing and measurement efforts. There are lots of books with really cool examples, great stories and absolutely no fucking useful information you can use on your website – if you’ve read some of these, you’ll know exactly what I mean. These are the tomes I got most practical use from and I’d recommend you buy the whole lot – worth every penny.
  35. So – what’s driving this change then? Well there have been great books on selling and persuading people – all the way back to ‘Scientific Advertising’ in 1923. And my favourite here is the Cialdini work – simply because it’s a great help for people to find practical uses for these techniques. I’ve also included some analytics and testing books here – primarily because they help so MUCH in augmenting our customer insight, testing and measurement efforts. There are lots of books with really cool examples, great stories and absolutely no fucking useful information you can use on your website – if you’ve read some of these, you’ll know exactly what I mean. These are the tomes I got most practical use from and I’d recommend you buy the whole lot – worth every penny.
  36. These are all people on twitter who cover hybrid stuff – where usability, psychology, analytics and persuasive writing collide. If you follow this lot, you’ll be much smarter within a month, guaranteed.
  37. And here are the most useful resources I regularly use or share with people. They have the best and most practical advice – cool insights but with practical applications.
  38. In my opinion, these are the attributes of companies doing great things with optimisation and continuous improvement.
  39. This is the future of testing. A machine learning system that will test out variants and tell you what’s driving response to all your experiments. Know if that offer works because of someone’s age or past spending patterns – let the tool explain to you where the value is and let it exploit these patterns as an intelligent agent under your control.