Best practices in applying metrics to optimizing your product and your business by Dan Olsen from the Startonomics Conference in San Francisco, October 2, 2008

1. Designing & Optimizing
the DNA of a Killer App
Dan Olsen, CEO & Founder, YourVersion
Startonomics ∙ San Francisco ∙ Oct 2, 2008
Copyright © 2008 Olsen Solutions LLC

2. Building the DNA of a Killer App with
Metrics and Optimization
How to integrate metrics & optimization
Using metrics to optimize value creation
Business Value
Customer Value
UI Design
Illustrate with real world examples
Copyright © 2008 Olsen Solutions LLC

3. Adding Metrics and Optimization to
your Product Process
Site Level
Business Product Prioritized
Plan Objectives Objectives Feature List
Scoping Feature
Level
Requirements
Design & Design
Code Test Launch
Develop
Metrics & User
Optimize Feedback
Copyright © 2008 Olsen Solutions LLC

4. Optimization through Iteration:
Continuous Improvement
Measure
the metric
Analyze
Learning the metric
Gaining knowledge:
• Market Identify top
• Customer opportunities
to improve
• Domain
• Usability Design & develop
the enhancement
Launch the
enhancement
Copyright © 2008 Olsen Solutions LLC

5. Optimizing
Business Value Creation
with Metrics
Copyright © 2008 Olsen Solutions LLC

6. Approaching Business as an
Optimization Exercise
Given reality as it exists today,
optimize our business results
subject to our resource constraints.
Copyright © 2008 Olsen Solutions LLC

7. Define the Equation of your Business
“Peeling the Onion”
Profit = Revenue ‐ Cost
Unique Visitors x Ad Revenue per Visitor
Impressions/Visitor x Effective CPM / 1000
Visits/Visitor x Pageviews/Visit x Impressions/PV
New Visitors + Returning Visitors
Invited Visitors + Uninvited Visitors
# of Users Sending Invites x Invites Sent/User x Invite Click‐through Rate
Copyright © 2008 Olsen Solutions LLC

8. How to Track Your Metrics
Track each metric as daily time series
Unique Page Ad New User
…
Date Visitors views Revenue Sign‐ups
4/24/08 10,100 29,600 25 490
4/27/08 10,500 27,100 24 480
…
Create ratios from primary metrics: X / Y
Example: How good is your registration page?
Okay: # of registered users per day
Better: registration conversion rate =
# registered users / # uniques to reg page
Copyright © 2008 Olsen Solutions LLC

9. Sample Signup Page Yield Data
Daily Signup Page Yield vs. Time
New Registered Users divided by Unique Visitors to Signup Page
100%
90%
80%
Daily Signup Page Yield
70%
60%
50%
40%
30% Started requiring
registration
20%
Changed Added questions
messaging to signup page
10%
0%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1
0
Copyright © 2008 Olsen Solutions LLC

10. Using Metrics to Optimize the
Equation of your Business
What are the metrics for your business?
Where is current value for each metric?
How many resources to “move” each metric?
Developer‐hours, time, money
Which metrics have highest ROI opportunities?
Metric A Metric B Metric C
Good ROI Bad ROI Great ROI
Return
Return
Return
Investment Investment Investment
Copyright © 2008 Olsen Solutions LLC

11. Case Study:
Optimizing Viral Loop Metrics
% of users
sending = 15% Invites per Invite
invites sender = 2.3 click-through rate
Active Invite Prospective Click Registration Fail
Users Users Process
% of users
who are Succeed
Don’t
active Click Conversion
rate = 85%
Users
• Multiplied together, these metrics determine your viral ratio
• Which metric has highest ROI opportunity?
Copyright © 2008 Olsen Solutions LLC

12. Maximum Upside Potential of a Metric
?
100% 100%
85%
15% 2.3
0 0 0
Registration % of users sending Avg # of invites
conversion rate invitations sent per sender
Max possible 0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%
improvement
Copyright © 2008 Olsen Solutions LLC

13. Doubling Number of Invitations Sent per Sender by
Adding Address Book Importer
Copyright © 2008 Olsen Solutions LLC

14. Optimizing
Customer Value Creation
with Metrics
Copyright © 2008 Olsen Solutions LLC

15. The Fuzzy Math of Customer Value
You create customer value with a product that
Satisfies customers’ needs
Is easy to use
Has a good price
Is better than other alternatives
Customer value = customer satisfaction
Not fuzzy: # of users and frequency of use
Understanding user needs and satisfaction
Quantitative: site analytics, usage metrics, surveys
Qualitative: support emails, usability sessions (best)
Copyright © 2008 Olsen Solutions LLC

16. What is Your Value Proposition?
Which user benefits are most important?
How well does your product deliver?
Importance Current User Upside
User Benefit to User Satisfaction Potential
Low High – +
Benefit 1 High
Benefit 2 High
Benefit 3 Low ?
opportunity
Benefit 4 Med
Benefit 5 Low ?
opportunity
Benefit 6 Low
Doesn’t matter
Copyright © 2008 Olsen Solutions LLC

17. Return (Value Created) Analyzing Product Ideas by ROI
?
4
Idea D
3
Idea A Idea B
2
Idea C
1
Idea F
1 2 3 4
Investment (developer-weeks)
Copyright © 2008 Olsen Solutions LLC

18. Return (Value Created) Prioritizing Product Ideas by ROI
Idea C
8
7 Idea B
6
5
4 Idea A
3
2
1
1 2 3 4 5
Investment (developer-weeks)
Copyright © 2008 Olsen Solutions LLC

19. The UI Design Iceberg
What most
people see
and react to Visual
Design What good
PMs and
Designers
Interaction think about
Design
Information
Architecture
Conceptual
Design
Recommended reading: Jesse James Garrett’s
“Elements of User Experience” chart, free at www.jjg.net Copyright © 2008 Olsen Solutions LLC

20. Approaching UI Design Analytically
Typical UI design question:
“When using web pages, do users scroll down?”
‐ Yes
‐ No
UI questions are never yes/no! (not binary)
Should ask: “What percentage of users …?”
UI changes impact your metrics
Impact can be positive, negative, small, large
Seek high‐ROI UI changes
Copyright © 2008 Olsen Solutions LLC

21. Put Key Conversion Actions Above The Fold
Landing Page A Landing Page B
The Fold
Key conversion action
is above the fold
Key conversion action
is below the fold
Copyright © 2008 Olsen Solutions LLC

22. The Fold Isn’t Binary Either
768 px
The chrome steals
% of Users
about 170 pixels
600 px 1024 px
Data courtesy of ClickTale
Free trial at www.clicktale.com Copyright © 2008 Olsen Solutions LLC

23. Case Study: Account Signup Process Redesign
100%
100%
Biggest
80%
drop
% of Users
62.3%
58.8%
60%
50.9%
40% 34.4% 32.7%
20%
0%
Sign in / Account Cash vs. 5 Partner 3 Partner
Registration Type Margin Pages Pages
Copyright © 2008 Olsen Solutions LLC

24. Analysis of Sign In/Registration Flow
Open
Account
55%
44% Register Registration (24% of Total)
Process
45% drop off
64%
(20% of total)
of Total Account
36% overall Selection
83% 30% drop off for
56% (46% of Total) (14% of Total) this step
Sign in
Forget 70% Change 80%
Password (32% of Total) Password (26% of Total)
17% drop off 20% drop off
(10% of total) (6% of total)
Copyright © 2008 Olsen Solutions LLC

25. Redesigned User Flow Improved
Registration Conversion Rate
Abandonment Rate (7 Day Moving Average)
Steps 1-2
80%
70%
Abandonment Rate (7 Day Moving Average)
60%
50%
40%
30% Released 37% improvement
New Design in conversion rate
20%
10%
0%
10/14/02
10/21/02
10/28/02
11/11/02
11/18/02
11/25/02
12/16/02
12/23/02
12/30/02
10/7/02
11/4/02
12/2/02
12/9/02
1/13/03
1/20/03
1/6/03
Copyright © 2008 Olsen Solutions LLC

26. Summary: Building the DNA of a Killer App
Define what success means
Customer value proposition
Equation of your business
Instrument your site and track key metrics
Identify opportunities and prioritize by ROI
Launch, learn, and iterate
Copyright © 2008 Olsen Solutions LLC

27. Questions?
www.yourversion.com
dan@yourversion.com
Copyright © 2008 Olsen Solutions LLC