In creating experiments to increase product sales or customer acquisition, the end goal is clear: the more orders placed, the more dollars back to the business. But how should we think about prioritizing tests to increase product engagement?
In a world where we can encourage users to take a wide variety of actions with unclear business benefits, it seems almost impossible to determine what is best. How can we promote behaviors that build habits and promote retention? The answer is simple: we analyze what habits matter most. In this session Olivia will share:
How she created the Wall Street Journal’s first habit model
How the team identified which actions influence member tenure
How WSJ uses habit data to inform their testing and engagement strategies
Test & Learn: Building Habits That Matter - How WSJ Models Subscriber Behavior to Improve Retention & Engagement
1. Olivia Simon
WSJ
Building Habits That Matter: How
WSJ Models Subscriber Behavior
to Improve Retention and
Engagement
Tune in on Twitter: #testandlearn19
4. DOW JONES OPTIMIZATION
Where Do We Test?
Acquisition:
Homepage placements
Article roadblocks
Shop
Checkout
Engagement:
Onboarding
Newsletter Center
Platforms Center
Customer Center
5. DOW JONES OPTIMIZATION
Where Do We Test?
Product:
Home page
Article pages
Videos / Video Center
Retention:
Online Cancel
Winback shop
Winback emails
Credit Card Decline
6. WSJ & BARRON’S iOS TESTING - 2016
DOW JONES OPTIMIZATION
How Do We Quantify the Value
of Product Testing?
8. DOW JONES OPTIMIZATION
But Do Product Tests Make Us $$$?
100 People
Article Test:
Email Shares
10% Uplift!
10 More email
shares
10 more email shares? So what?
9. DOW JONES OPTIMIZATION
Determining A Fair Metric For Product Testing
Did they ride it again? Did they order it again?
Test question: What does WSJ have that
keeps members around longer?
10. DOW JONES OPTIMIZATION
Turning Product Actions Into Value
The Habit Project:
A model to determine what onsite habits
make members stay longer.
11. DOW JONES OPTIMIZATION
Habit in a Nutshell
Over 50 product actions
● Content types (section,
column)
● Authors
● Video/Audio
● Sharing
● Product benefits (app,
newsletter, WS+)
Resulting 1-year
retention rate
+
Likelihood
Yes! :)
No! >:(
In a member’s first
100 days, did they
do the action?
13. DOW JONES OPTIMIZATION
Correlation vs Causation?
What if people who
share via email are
just more engaged
members than
those who watch
video content?”
1. Model focuses on non-habituated
readers (ie, first 100 days).
2. Trends held true regardless of
tenure and subscription package
selected.
14. DOW JONES OPTIMIZATION
Our Framework for Mapping Habits
Habit’s Impact on 1 Year Retention Rate
CurrentAdoptionRateoftheHabit
LOW HIGH
HIGH
LOW Unlikely and Not
Sticky!
Highly Likely and
Very Sticky!
Very Sticky, but
Unlikely
Very Likely, but
not Sticky
15. DOW JONES OPTIMIZATION
LOW HIGH
2
122 11
10
4 8HIGH
LOW
1
Our Framework for Mapping Habits
Habit’s Impact on 1 Year Retention Rate
CurrentAdoptionRateoftheHabit
17. WSJ & BARRON’S iOS TESTING - 2016
DOW JONES OPTIMIZATION
Actioning Sticky Habits via Testing
18. DOW JONES OPTIMIZATION
Testing Our Onboarding Flow
Mobile Apps
Card 1
Newsletter
Card 2
WSJ+
Card 4
Social
Card 5
PI Card
Card 3
Control
19. DOW JONES OPTIMIZATION
Testing Our Onboarding Flow
Variation
App Card 1
Newsletter Card 2
Puzzle Card 4 Watchlist Card 5
Video Card 6
Podcast Card 7 WSJ+ Card 8 Social Card 9
PI Card 3
Books and Sports
Newsletters Added
+51%
increase in average
number of
engagement actions
being taken
24. DOW JONES OPTIMIZATION
How Much We’ve Changed Engagement With Sticky Actions
Habit Change
Play Podcast on WSJ.com +371%
Read Content A +118%
Play Puzzle +54%
Read Author A +51%
Read Author B +51%
Subscribe to New Newsletter +44%
Post Comment +14%
Read Content B +10%
Visit On Weekend +7%
Login App +6%
Add to Watchlist -2%
Read Content C -7%
Read Content D -16%
26. Next Session
How to Leverage Design to Learn & Deliver
Results Quickly
Angel Steger
Director of Product Design, Dropbox
Tune in on Twitter: #testandlearn19