SlideShare ist ein Scribd-Unternehmen logo
1 von 26
Downloaden Sie, um offline zu lesen
Olivia Simon
WSJ
Building Habits That Matter: How
WSJ Models Subscriber Behavior
to Improve Retention and
Engagement
Tune in on Twitter: #testandlearn19
Building Habits That Matter
at WSJ
DOW JONES OPTIMIZATION
What Does Experimentation at
WSJ Look Like?
DOW JONES OPTIMIZATION
Where Do We Test?
Acquisition:
Homepage placements
Article roadblocks
Shop
Checkout
Engagement:
Onboarding
Newsletter Center
Platforms Center
Customer Center
DOW JONES OPTIMIZATION
Where Do We Test?
Product:
Home page
Article pages
Videos / Video Center
Retention:
Online Cancel
Winback shop
Winback emails
Credit Card Decline
WSJ & BARRON’S iOS TESTING - 2016
DOW JONES OPTIMIZATION
How Do We Quantify the Value
of Product Testing?
DOW JONES OPTIMIZATION
Acquisition Tests Make Us $$$$
100 People
Shop Test
10% Uplift!
10 More
orders!
10 orders x $2 per order = $20!
DOW JONES OPTIMIZATION
But Do Product Tests Make Us $$$?
100 People
Article Test:
Email Shares
10% Uplift!
10 More email
shares
10 more email shares? So what?
DOW JONES OPTIMIZATION
Determining A Fair Metric For Product Testing
Did they ride it again? Did they order it again?
Test question: What does WSJ have that
keeps members around longer?
DOW JONES OPTIMIZATION
Turning Product Actions Into Value
The Habit Project:
A model to determine what onsite habits
make members stay longer.
DOW JONES OPTIMIZATION
Habit in a Nutshell
Over 50 product actions
● Content types (section,
column)
● Authors
● Video/Audio
● Sharing
● Product benefits (app,
newsletter, WS+)
Resulting 1-year
retention rate
+
Likelihood
Yes! :)
No! >:(
In a member’s first
100 days, did they
do the action?
DOW JONES OPTIMIZATION
First 100 Days = Habit Magic
Opportunity
Window
First 100 Days
DOW JONES OPTIMIZATION
Correlation vs Causation?
What if people who
share via email are
just more engaged
members than
those who watch
video content?”
1. Model focuses on non-habituated
readers (ie, first 100 days).
2. Trends held true regardless of
tenure and subscription package
selected.
DOW JONES OPTIMIZATION
Our Framework for Mapping Habits
Habit’s Impact on 1 Year Retention Rate
CurrentAdoptionRateoftheHabit
LOW HIGH
HIGH
LOW Unlikely and Not
Sticky!
Highly Likely and
Very Sticky!
Very Sticky, but
Unlikely
Very Likely, but
not Sticky
DOW JONES OPTIMIZATION
LOW HIGH
2
122 11
10
4 8HIGH
LOW
1
Our Framework for Mapping Habits
Habit’s Impact on 1 Year Retention Rate
CurrentAdoptionRateoftheHabit
DOW JONES OPTIMIZATION
A Peek At The Results
WSJ App Newsletters Our Content!
WSJ & BARRON’S iOS TESTING - 2016
DOW JONES OPTIMIZATION
Actioning Sticky Habits via Testing
DOW JONES OPTIMIZATION
Testing Our Onboarding Flow
Mobile Apps
Card 1
Newsletter
Card 2
WSJ+
Card 4
Social
Card 5
PI Card
Card 3
Control
DOW JONES OPTIMIZATION
Testing Our Onboarding Flow
Variation
App Card 1
Newsletter Card 2
Puzzle Card 4 Watchlist Card 5
Video Card 6
Podcast Card 7 WSJ+ Card 8 Social Card 9
PI Card 3
Books and Sports
Newsletters Added
+51%
increase in average
number of
engagement actions
being taken
DOW JONES OPTIMIZATION
Get Them to The App!
Variation
+4.3% app download rate
Control
DOW JONES OPTIMIZATION
Get Them to The App! Part 2
Variation
+7194% app download rate
Control
DOW JONES OPTIMIZATION
Get Them to The App! Part 3
Variation
+13% app download rate
Control
WSJ & BARRON’S iOS TESTING - 2016
DOW JONES OPTIMIZATION
Evaluating Success
DOW JONES OPTIMIZATION
How Much We’ve Changed Engagement With Sticky Actions
Habit Change
Play Podcast on WSJ.com +371%
Read Content A +118%
Play Puzzle +54%
Read Author A +51%
Read Author B +51%
Subscribe to New Newsletter +44%
Post Comment +14%
Read Content B +10%
Visit On Weekend +7%
Login App +6%
Add to Watchlist -2%
Read Content C -7%
Read Content D -16%
DOW JONES OPTIMIZATION
Building Habits That Matter
at WSJ
Next Session
How to Leverage Design to Learn & Deliver
Results Quickly
Angel Steger
Director of Product Design, Dropbox
Tune in on Twitter: #testandlearn19

Weitere ähnliche Inhalte

Was ist angesagt?

The Content Marketing Manifesto
The Content Marketing ManifestoThe Content Marketing Manifesto
The Content Marketing ManifestoRand Fishkin
 
Slide tài liệu - Tổng quan về SEO (update 2017)
Slide tài liệu - Tổng quan về SEO (update 2017)Slide tài liệu - Tổng quan về SEO (update 2017)
Slide tài liệu - Tổng quan về SEO (update 2017)Dung Hoang Seothetop
 
Level Up Your Content Strategy – 5 Steps To SEO Success.pdf
Level Up Your Content Strategy – 5 Steps To SEO Success.pdfLevel Up Your Content Strategy – 5 Steps To SEO Success.pdf
Level Up Your Content Strategy – 5 Steps To SEO Success.pdfSearch Engine Journal
 
LinkedIn Basics 101
LinkedIn Basics 101LinkedIn Basics 101
LinkedIn Basics 101grummanaa1c
 
YouTube for Your Business
YouTube for Your BusinessYouTube for Your Business
YouTube for Your BusinessDigital Vidya
 
Yutang TikTok 2021.pptx.pdf
Yutang TikTok 2021.pptx.pdfYutang TikTok 2021.pptx.pdf
Yutang TikTok 2021.pptx.pdfHoangPhan379709
 
Ejemplo de Social Listening
Ejemplo de Social ListeningEjemplo de Social Listening
Ejemplo de Social Listeningcarlos8705
 
Giới thiệu giải pháp Zalo Shop & Zalo Ads
Giới thiệu giải pháp Zalo Shop & Zalo AdsGiới thiệu giải pháp Zalo Shop & Zalo Ads
Giới thiệu giải pháp Zalo Shop & Zalo AdsDuy Nguyen
 
Nghiên cứu từ khóa Chìa khóa để SEO thành công
Nghiên cứu từ khóa Chìa khóa để SEO thành côngNghiên cứu từ khóa Chìa khóa để SEO thành công
Nghiên cứu từ khóa Chìa khóa để SEO thành côngDung Hoang Seothetop
 
Manual google analytics
Manual google analyticsManual google analytics
Manual google analyticsMafer Durán
 
20+ Digital marketing tools to Grow your business in 2022
20+ Digital marketing tools to Grow your business in 202220+ Digital marketing tools to Grow your business in 2022
20+ Digital marketing tools to Grow your business in 2022Simplilearn
 
Infosnackers Guide to the Galaxy: Embracing Short Form, Ephemeral & Realtime...
Infosnackers Guide to the Galaxy:  Embracing Short Form, Ephemeral & Realtime...Infosnackers Guide to the Galaxy:  Embracing Short Form, Ephemeral & Realtime...
Infosnackers Guide to the Galaxy: Embracing Short Form, Ephemeral & Realtime...Nick Kellet
 
Growth Hacking Basics
Growth Hacking BasicsGrowth Hacking Basics
Growth Hacking BasicsMorgan Brown
 

Was ist angesagt? (20)

The Content Marketing Manifesto
The Content Marketing ManifestoThe Content Marketing Manifesto
The Content Marketing Manifesto
 
Slide tài liệu - Tổng quan về SEO (update 2017)
Slide tài liệu - Tổng quan về SEO (update 2017)Slide tài liệu - Tổng quan về SEO (update 2017)
Slide tài liệu - Tổng quan về SEO (update 2017)
 
Tik Tok analysis
Tik Tok analysisTik Tok analysis
Tik Tok analysis
 
Level Up Your Content Strategy – 5 Steps To SEO Success.pdf
Level Up Your Content Strategy – 5 Steps To SEO Success.pdfLevel Up Your Content Strategy – 5 Steps To SEO Success.pdf
Level Up Your Content Strategy – 5 Steps To SEO Success.pdf
 
LinkedIn Basics 101
LinkedIn Basics 101LinkedIn Basics 101
LinkedIn Basics 101
 
YouTube for Your Business
YouTube for Your BusinessYouTube for Your Business
YouTube for Your Business
 
Yutang TikTok 2021.pptx.pdf
Yutang TikTok 2021.pptx.pdfYutang TikTok 2021.pptx.pdf
Yutang TikTok 2021.pptx.pdf
 
GOOGLE ALGORITHMS.pdf
GOOGLE ALGORITHMS.pdfGOOGLE ALGORITHMS.pdf
GOOGLE ALGORITHMS.pdf
 
Ejemplo de Social Listening
Ejemplo de Social ListeningEjemplo de Social Listening
Ejemplo de Social Listening
 
Giới thiệu giải pháp Zalo Shop & Zalo Ads
Giới thiệu giải pháp Zalo Shop & Zalo AdsGiới thiệu giải pháp Zalo Shop & Zalo Ads
Giới thiệu giải pháp Zalo Shop & Zalo Ads
 
Nghiên cứu từ khóa Chìa khóa để SEO thành công
Nghiên cứu từ khóa Chìa khóa để SEO thành côngNghiên cứu từ khóa Chìa khóa để SEO thành công
Nghiên cứu từ khóa Chìa khóa để SEO thành công
 
Manual google analytics
Manual google analyticsManual google analytics
Manual google analytics
 
OMEKA
OMEKAOMEKA
OMEKA
 
20+ Digital marketing tools to Grow your business in 2022
20+ Digital marketing tools to Grow your business in 202220+ Digital marketing tools to Grow your business in 2022
20+ Digital marketing tools to Grow your business in 2022
 
SEO SEM SMO Plan
SEO SEM SMO PlanSEO SEM SMO Plan
SEO SEM SMO Plan
 
e-Productos
e-Productose-Productos
e-Productos
 
Infosnackers Guide to the Galaxy: Embracing Short Form, Ephemeral & Realtime...
Infosnackers Guide to the Galaxy:  Embracing Short Form, Ephemeral & Realtime...Infosnackers Guide to the Galaxy:  Embracing Short Form, Ephemeral & Realtime...
Infosnackers Guide to the Galaxy: Embracing Short Form, Ephemeral & Realtime...
 
Linkedin 101 ppt
Linkedin 101 pptLinkedin 101 ppt
Linkedin 101 ppt
 
Growth Hacking Basics
Growth Hacking BasicsGrowth Hacking Basics
Growth Hacking Basics
 
YouTube Presentation
YouTube Presentation YouTube Presentation
YouTube Presentation
 

Ähnlich wie Test & Learn: Building Habits That Matter - How WSJ Models Subscriber Behavior to Improve Retention & Engagement

Searching higher up the funnel
Searching higher up the funnelSearching higher up the funnel
Searching higher up the funnelJono Alderson
 
eStudio34 presents Search Love London 2015 | Searching higher up the funnel b...
eStudio34 presents Search Love London 2015 | Searching higher up the funnel b...eStudio34 presents Search Love London 2015 | Searching higher up the funnel b...
eStudio34 presents Search Love London 2015 | Searching higher up the funnel b...William Renedo
 
Writing News for Mobile - Tony Gonzalez - Murfreesboro, TN - Sept. 30-Oct. 1,...
Writing News for Mobile - Tony Gonzalez - Murfreesboro, TN - Sept. 30-Oct. 1,...Writing News for Mobile - Tony Gonzalez - Murfreesboro, TN - Sept. 30-Oct. 1,...
Writing News for Mobile - Tony Gonzalez - Murfreesboro, TN - Sept. 30-Oct. 1,...News Leaders Association's NewsTrain
 
Social Media & Digital Marketing: What You Need to Know to Stay Ahead of You...
Social Media & Digital Marketing:  What You Need to Know to Stay Ahead of You...Social Media & Digital Marketing:  What You Need to Know to Stay Ahead of You...
Social Media & Digital Marketing: What You Need to Know to Stay Ahead of You...Erica Campbell Byrum
 
Measuring customer experience with social media.jan15
Measuring customer experience with social media.jan15Measuring customer experience with social media.jan15
Measuring customer experience with social media.jan15Michael Wolfe
 
Making Social Media Make Money: Ten Strategies
Making Social Media Make Money: Ten StrategiesMaking Social Media Make Money: Ten Strategies
Making Social Media Make Money: Ten StrategiesAffiliate Summit
 
BrightonSEO - Jon May and how to Send Better Emails
BrightonSEO  - Jon May and how to Send Better EmailsBrightonSEO  - Jon May and how to Send Better Emails
BrightonSEO - Jon May and how to Send Better EmailsJon May 💌 Email Expert
 
The Wall Street Journal - Optimizing Membership
The Wall Street Journal - Optimizing MembershipThe Wall Street Journal - Optimizing Membership
The Wall Street Journal - Optimizing MembershipOptimizely
 
Zappos Preso 2009-02-03 Shop.Org
Zappos Preso 2009-02-03 Shop.OrgZappos Preso 2009-02-03 Shop.Org
Zappos Preso 2009-02-03 Shop.OrgAlfred Lin
 
Zappos - National Society of Leadership and Success - 1-27-09
Zappos - National Society of Leadership and Success - 1-27-09Zappos - National Society of Leadership and Success - 1-27-09
Zappos - National Society of Leadership and Success - 1-27-09zappos
 
Inbound Marketing: Your Competitive Advantage
Inbound Marketing: Your Competitive AdvantageInbound Marketing: Your Competitive Advantage
Inbound Marketing: Your Competitive AdvantageHubSpot
 
Netbase AMA Sentiment Analysis Presentation
Netbase AMA Sentiment Analysis PresentationNetbase AMA Sentiment Analysis Presentation
Netbase AMA Sentiment Analysis PresentationNetBase
 
AAN Annual Meeting - Promotions & Interactive Content Explosion
AAN Annual Meeting - Promotions & Interactive Content ExplosionAAN Annual Meeting - Promotions & Interactive Content Explosion
AAN Annual Meeting - Promotions & Interactive Content ExplosionUpland Second Street
 
Key inflection points for influencing customer behaviour
Key inflection points for influencing customer behaviourKey inflection points for influencing customer behaviour
Key inflection points for influencing customer behaviourMichael O'Sullivan
 
Zappos Preso 2009-01-27 Health 3.0
Zappos Preso 2009-01-27 Health 3.0Zappos Preso 2009-01-27 Health 3.0
Zappos Preso 2009-01-27 Health 3.0Alfred Lin
 
WEBCON Resident Experience Cycle Toni Blake
WEBCON Resident Experience Cycle Toni BlakeWEBCON Resident Experience Cycle Toni Blake
WEBCON Resident Experience Cycle Toni BlakeToni Blake
 
Keynote Ton Wesseling at Superweek 2020: How an analyst can add value!
Keynote Ton Wesseling at Superweek 2020: How an analyst can add value!Keynote Ton Wesseling at Superweek 2020: How an analyst can add value!
Keynote Ton Wesseling at Superweek 2020: How an analyst can add value!Ton Wesseling
 
Brands the cost of being human sxsw 2012
Brands the cost of being human sxsw 2012Brands the cost of being human sxsw 2012
Brands the cost of being human sxsw 2012Jennifer van der Meer
 
Workshop: An Apartment Manager’s Crash Course on Social Media Marketing by Er...
Workshop: An Apartment Manager’s Crash Course on Social Media Marketing by Er...Workshop: An Apartment Manager’s Crash Course on Social Media Marketing by Er...
Workshop: An Apartment Manager’s Crash Course on Social Media Marketing by Er...multifamily-social-media
 

Ähnlich wie Test & Learn: Building Habits That Matter - How WSJ Models Subscriber Behavior to Improve Retention & Engagement (20)

Searching higher up the funnel
Searching higher up the funnelSearching higher up the funnel
Searching higher up the funnel
 
10 SEO Tips for Communications & PR
10 SEO Tips for Communications & PR10 SEO Tips for Communications & PR
10 SEO Tips for Communications & PR
 
eStudio34 presents Search Love London 2015 | Searching higher up the funnel b...
eStudio34 presents Search Love London 2015 | Searching higher up the funnel b...eStudio34 presents Search Love London 2015 | Searching higher up the funnel b...
eStudio34 presents Search Love London 2015 | Searching higher up the funnel b...
 
Writing News for Mobile - Tony Gonzalez - Murfreesboro, TN - Sept. 30-Oct. 1,...
Writing News for Mobile - Tony Gonzalez - Murfreesboro, TN - Sept. 30-Oct. 1,...Writing News for Mobile - Tony Gonzalez - Murfreesboro, TN - Sept. 30-Oct. 1,...
Writing News for Mobile - Tony Gonzalez - Murfreesboro, TN - Sept. 30-Oct. 1,...
 
Social Media & Digital Marketing: What You Need to Know to Stay Ahead of You...
Social Media & Digital Marketing:  What You Need to Know to Stay Ahead of You...Social Media & Digital Marketing:  What You Need to Know to Stay Ahead of You...
Social Media & Digital Marketing: What You Need to Know to Stay Ahead of You...
 
Measuring customer experience with social media.jan15
Measuring customer experience with social media.jan15Measuring customer experience with social media.jan15
Measuring customer experience with social media.jan15
 
Making Social Media Make Money: Ten Strategies
Making Social Media Make Money: Ten StrategiesMaking Social Media Make Money: Ten Strategies
Making Social Media Make Money: Ten Strategies
 
BrightonSEO - Jon May and how to Send Better Emails
BrightonSEO  - Jon May and how to Send Better EmailsBrightonSEO  - Jon May and how to Send Better Emails
BrightonSEO - Jon May and how to Send Better Emails
 
The Wall Street Journal - Optimizing Membership
The Wall Street Journal - Optimizing MembershipThe Wall Street Journal - Optimizing Membership
The Wall Street Journal - Optimizing Membership
 
Zappos Preso 2009-02-03 Shop.Org
Zappos Preso 2009-02-03 Shop.OrgZappos Preso 2009-02-03 Shop.Org
Zappos Preso 2009-02-03 Shop.Org
 
Zappos - National Society of Leadership and Success - 1-27-09
Zappos - National Society of Leadership and Success - 1-27-09Zappos - National Society of Leadership and Success - 1-27-09
Zappos - National Society of Leadership and Success - 1-27-09
 
Inbound Marketing: Your Competitive Advantage
Inbound Marketing: Your Competitive AdvantageInbound Marketing: Your Competitive Advantage
Inbound Marketing: Your Competitive Advantage
 
Netbase AMA Sentiment Analysis Presentation
Netbase AMA Sentiment Analysis PresentationNetbase AMA Sentiment Analysis Presentation
Netbase AMA Sentiment Analysis Presentation
 
AAN Annual Meeting - Promotions & Interactive Content Explosion
AAN Annual Meeting - Promotions & Interactive Content ExplosionAAN Annual Meeting - Promotions & Interactive Content Explosion
AAN Annual Meeting - Promotions & Interactive Content Explosion
 
Key inflection points for influencing customer behaviour
Key inflection points for influencing customer behaviourKey inflection points for influencing customer behaviour
Key inflection points for influencing customer behaviour
 
Zappos Preso 2009-01-27 Health 3.0
Zappos Preso 2009-01-27 Health 3.0Zappos Preso 2009-01-27 Health 3.0
Zappos Preso 2009-01-27 Health 3.0
 
WEBCON Resident Experience Cycle Toni Blake
WEBCON Resident Experience Cycle Toni BlakeWEBCON Resident Experience Cycle Toni Blake
WEBCON Resident Experience Cycle Toni Blake
 
Keynote Ton Wesseling at Superweek 2020: How an analyst can add value!
Keynote Ton Wesseling at Superweek 2020: How an analyst can add value!Keynote Ton Wesseling at Superweek 2020: How an analyst can add value!
Keynote Ton Wesseling at Superweek 2020: How an analyst can add value!
 
Brands the cost of being human sxsw 2012
Brands the cost of being human sxsw 2012Brands the cost of being human sxsw 2012
Brands the cost of being human sxsw 2012
 
Workshop: An Apartment Manager’s Crash Course on Social Media Marketing by Er...
Workshop: An Apartment Manager’s Crash Course on Social Media Marketing by Er...Workshop: An Apartment Manager’s Crash Course on Social Media Marketing by Er...
Workshop: An Apartment Manager’s Crash Course on Social Media Marketing by Er...
 

Mehr von Optimizely

Clover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationClover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationOptimizely
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Optimizely
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing RightOptimizely
 
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleAtlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleOptimizely
 
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Optimizely
 
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueZillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueOptimizely
 
The Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsThe Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsOptimizely
 
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Optimizely
 
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Optimizely
 
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingBuilding an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingOptimizely
 
AMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideAMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideOptimizely
 
Evolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentEvolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentOptimizely
 
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOvercoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOptimizely
 
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...Optimizely
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyOptimizely
 
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueKick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueOptimizely
 
Experimentation through Clients' Eyes
Experimentation through Clients' EyesExperimentation through Clients' Eyes
Experimentation through Clients' EyesOptimizely
 
Shipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubShipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubOptimizely
 
Test Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationTest Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationOptimizely
 
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely
 

Mehr von Optimizely (20)

Clover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationClover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive Experimentation
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing Right
 
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleAtlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
 
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
 
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueZillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
 
The Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsThe Future of Optimizely for Technical Teams
The Future of Optimizely for Technical Teams
 
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
 
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
 
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingBuilding an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team Offering
 
AMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideAMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server Side
 
Evolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentEvolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product Development
 
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOvercoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
 
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product Strategy
 
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueKick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
 
Experimentation through Clients' Eyes
Experimentation through Clients' EyesExperimentation through Clients' Eyes
Experimentation through Clients' Eyes
 
Shipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubShipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHub
 
Test Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationTest Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with Experimentation
 
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature Delivery
 

Kürzlich hochgeladen

A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024Results
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slidevu2urc
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 
Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Paola De la Torre
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...gurkirankumar98700
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
 

Kürzlich hochgeladen (20)

A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 

Test & Learn: Building Habits That Matter - How WSJ Models Subscriber Behavior to Improve Retention & Engagement

  • 1. Olivia Simon WSJ Building Habits That Matter: How WSJ Models Subscriber Behavior to Improve Retention and Engagement Tune in on Twitter: #testandlearn19
  • 2. Building Habits That Matter at WSJ
  • 3. DOW JONES OPTIMIZATION What Does Experimentation at WSJ Look Like?
  • 4. DOW JONES OPTIMIZATION Where Do We Test? Acquisition: Homepage placements Article roadblocks Shop Checkout Engagement: Onboarding Newsletter Center Platforms Center Customer Center
  • 5. DOW JONES OPTIMIZATION Where Do We Test? Product: Home page Article pages Videos / Video Center Retention: Online Cancel Winback shop Winback emails Credit Card Decline
  • 6. WSJ & BARRON’S iOS TESTING - 2016 DOW JONES OPTIMIZATION How Do We Quantify the Value of Product Testing?
  • 7. DOW JONES OPTIMIZATION Acquisition Tests Make Us $$$$ 100 People Shop Test 10% Uplift! 10 More orders! 10 orders x $2 per order = $20!
  • 8. DOW JONES OPTIMIZATION But Do Product Tests Make Us $$$? 100 People Article Test: Email Shares 10% Uplift! 10 More email shares 10 more email shares? So what?
  • 9. DOW JONES OPTIMIZATION Determining A Fair Metric For Product Testing Did they ride it again? Did they order it again? Test question: What does WSJ have that keeps members around longer?
  • 10. DOW JONES OPTIMIZATION Turning Product Actions Into Value The Habit Project: A model to determine what onsite habits make members stay longer.
  • 11. DOW JONES OPTIMIZATION Habit in a Nutshell Over 50 product actions ● Content types (section, column) ● Authors ● Video/Audio ● Sharing ● Product benefits (app, newsletter, WS+) Resulting 1-year retention rate + Likelihood Yes! :) No! >:( In a member’s first 100 days, did they do the action?
  • 12. DOW JONES OPTIMIZATION First 100 Days = Habit Magic Opportunity Window First 100 Days
  • 13. DOW JONES OPTIMIZATION Correlation vs Causation? What if people who share via email are just more engaged members than those who watch video content?” 1. Model focuses on non-habituated readers (ie, first 100 days). 2. Trends held true regardless of tenure and subscription package selected.
  • 14. DOW JONES OPTIMIZATION Our Framework for Mapping Habits Habit’s Impact on 1 Year Retention Rate CurrentAdoptionRateoftheHabit LOW HIGH HIGH LOW Unlikely and Not Sticky! Highly Likely and Very Sticky! Very Sticky, but Unlikely Very Likely, but not Sticky
  • 15. DOW JONES OPTIMIZATION LOW HIGH 2 122 11 10 4 8HIGH LOW 1 Our Framework for Mapping Habits Habit’s Impact on 1 Year Retention Rate CurrentAdoptionRateoftheHabit
  • 16. DOW JONES OPTIMIZATION A Peek At The Results WSJ App Newsletters Our Content!
  • 17. WSJ & BARRON’S iOS TESTING - 2016 DOW JONES OPTIMIZATION Actioning Sticky Habits via Testing
  • 18. DOW JONES OPTIMIZATION Testing Our Onboarding Flow Mobile Apps Card 1 Newsletter Card 2 WSJ+ Card 4 Social Card 5 PI Card Card 3 Control
  • 19. DOW JONES OPTIMIZATION Testing Our Onboarding Flow Variation App Card 1 Newsletter Card 2 Puzzle Card 4 Watchlist Card 5 Video Card 6 Podcast Card 7 WSJ+ Card 8 Social Card 9 PI Card 3 Books and Sports Newsletters Added +51% increase in average number of engagement actions being taken
  • 20. DOW JONES OPTIMIZATION Get Them to The App! Variation +4.3% app download rate Control
  • 21. DOW JONES OPTIMIZATION Get Them to The App! Part 2 Variation +7194% app download rate Control
  • 22. DOW JONES OPTIMIZATION Get Them to The App! Part 3 Variation +13% app download rate Control
  • 23. WSJ & BARRON’S iOS TESTING - 2016 DOW JONES OPTIMIZATION Evaluating Success
  • 24. DOW JONES OPTIMIZATION How Much We’ve Changed Engagement With Sticky Actions Habit Change Play Podcast on WSJ.com +371% Read Content A +118% Play Puzzle +54% Read Author A +51% Read Author B +51% Subscribe to New Newsletter +44% Post Comment +14% Read Content B +10% Visit On Weekend +7% Login App +6% Add to Watchlist -2% Read Content C -7% Read Content D -16%
  • 25. DOW JONES OPTIMIZATION Building Habits That Matter at WSJ
  • 26. Next Session How to Leverage Design to Learn & Deliver Results Quickly Angel Steger Director of Product Design, Dropbox Tune in on Twitter: #testandlearn19