In a world of ever-changing customer needs and behaviors, product experimentation has become an indispensable way for organisations to validate new ideas, de-risk delivery and drive impactful outcomes.
This opens up a whole new set of challenges and opportunities:
How do we use experiments to drive product success and business outcomes?
How do we apply experimentation methods and make it into a continuous process for our organisations?
How can we leverage the experimental mindset to take the guesswork out of product innovation?
1. Driving Product Success through
Experimentation
Talk + Workshop
Scott Si
Felicitas Seah
#ISSLearningFest
2. What we will cover today
#ISSLearningFest
A New Way of
Building Products
Workshop: How to
Get Started
Which Experiments
Matter Most?
1 2 3
Key Takeaway: Viewing product management in a more outcome-led, hypothesis-driven
way – build with intention by framing potential features as experiments
3. Your Facilitators
#ISSLearningFest
Scott SI Felicitas SEAH
Senior Lecturer &
Consultant, Digital
Products & Platforms
Practice
Principal Lecturer &
Consultant, Digital
Products & Platforms
Practice
4. Goal as Product Builders…
Deliver Value for clients and
stakeholders
#ISSLearningFest
But… What is Value?
8. #ISSLearningFest
Build 10 things, find out only
3 worked
Run 10 experiments,
build the 3 that worked
Image from https://www.hellowonderful.co/post/blast-
off-with-these-8-fun-ways-to-make-a-rocket/
OLD
NEW
9. #ISSLearningFest
“Instead of saying 'I have an idea,' what if
you said 'I have a new hypothesis, let's go
test it, see if it's valid, ask how quickly can
we validate it.' And if it's not valid, move on
to the next one.”
Satya Nadella, Microsoft CEO
https://www.businessinsider.com/microsoft-ceo-brilliant-career-advice-
2017-5?IR=T&jwsource=cl
11. Value = Driving Metrics that Matter
#ISSLearningFest
Product Initiatives User Actions/KPIs User/Business Outcome
New
activation
flow
Product
360
view
User
Comments
Seller
chat
North Star Metric Revenue/
Shareholder Value
# of sign-ups
App downloads
% of returning users
e.g. Total value
transacted,
services booked
Ave. shopping cart items
12. So what is a product experiment?
#ISSLearningFest
Drive business and user
outcome
A B
vs.
Control Treatment
14. Getting Started: Crafting your hypothesis
#ISSLearningFest
We believe that [making product change]
will lead to [meaningful outcome]
We are right if [improvement in measurable user action
versus baseline]
and we can validate this by [type of experiment]
15. Example: Page Registrations
#ISSLearningFest
We believe that [changing ‘Register’ button
to blue]
will lead to [more membership registrations]
We are right if [we see 2x improvement in
‘Register’ CTR versus the baseline]
and we can validate this by [running A/B
test for the homepage]
https://hbr.org/2020/03/building-a-culture-of-experimentation
16. Why do we experiment?
#ISSLearningFest
• We don’t have all the answers
• Drive metrics that matter - measure the impact of a change
o New features/functions may hurt KPIs or metrics!
o Differing/polarized opinions within the team on a product change
• Avoid feature creep
18. Product Experiment Mini Case Study: E-commerce
You are a product manager for an apparel e-commerce platform. While your channel
traffic is on a historic high, your team flagged that total sales (gross merchandise value)
are flat compared to the previous quarter.
You review your product backlog for the purchase page to see which of these product
changes your team should work on to drive the most business impact:
Choose one of the features above. How would you define and reframe it as a
hypothesis that you can test with experiments - with the goal of driving up purchases?
#ISSLearningFest
Delivery time estimation Additional pick-up options
Seller Q&A Product 360-degree view
A C
B D
19. Workshop Template
#ISSLearningFest
We believe that [feature or product change]
will lead to [meaningful user/business outcome]
We are right if [proof of measurable improvement on
desired user action over the baseline, usually a % increase].
and we can validate this by [the experiment you will run]
1
2
3
4
20. Example
#ISSLearningFest
We believe that [adding delivery time estimation prompt
to the purchase page]
will lead to [higher ‘add to shopping cart’ conversions]
We are right if [# of purchases increase by 20% versus
the baseline].
and we can validate this easily by [running A/B test on
Singapore user base over the next 6 weeks]
21. Product Experimentation Steps
1. Know which metrics matter
2. Ask questions and brainstorm
3. Create testable hypothesis
4. Conduct experiment and monitor data
5. Make decision and implement changes
6. Repeat!
#ISSLearningFest
22. #ISSLearningFest
What is my riskiest assumption?
What is the impact if I am proven
right?
What is the least expensive way to
test it?
Should we experiment on everything?
23. Experimentation is a Spectrum
#ISSLearningFest
Optimization
Product Discovery
Are we solving the right problem?
Is this the right solution to solve the problem?
Is this the best way to build the solution?
More risky
Assumptions
Assumptions
1
2
3
‘North Star’
