As a product manager, you probably know specific ways to gather data to inform your product decisions, like the ever-popular A/B test. But as they say, you wouldn't put a round peg in a square hole. What about the times when it doesn't make sense to A/B test, because you have too small a sample size? Do you just do it anyway, because your company's product culture requires that you have those numbers? Tim Herbig will share his hands-on approach to working with analytics in agile product management. Tim will discuss the line between being data-informed versus data-driven. The audience will leave with an analysis toolkit filled with the right data tools for every scenario. Data validation won't be an issue again.

1. How To Build Data-Informed Products
Tim Herbig Hannah Flynn
With: Moderated by:
TO USE YOUR COMPUTER'S AUDIO:
When the webinar begins, you will be connected to audio using
your computer's microphone and speakers (VoIP). A headset is
recommended.
Webinar will begin:
11:00 am, PST
TO USE YOUR TELEPHONE:
If you prefer to use your phone, you must select "Use
Telephone" after joining the webinar and call in using the
numbers below.
United States: +1 (415) 655-0052
Access Code: 985-561-257
Audio PIN: Shown after joining the webinar
--OR--

2. Revulytics gives any software producers deep and actionable insight into
who is using their software products and how they are being used, and the
out-of-box analytics that enable them to grow incremental revenue, convert
and retain customers, and make decisions about licensing and cloud
transformation strategies.

3. Click on the Questions panel to
interact with the presenters
www.productmanagementtoday.com/webinar-series/analytics-in-action
www.projectmanagementupdate.com/webinar-series/analytics-in-action
www.businessinnovationbrief.com/webinar-series/analytics-in-action

4. About Tim Herbig
Tim is a product and business leader, as well as a prolific speaker and author of the book ‘Lateral Leadership: A
Practical Guide for Agile Product Managers’. Currently, he’s responsible for the overall sales, product,
engineering, and marketing efforts behind the leading conversion optimization platform iridion.
He previously shaped digital products at XING, Gruner+Jahr, and several smaller startups where he held product
leadership roles for more than seven years. Tim also regularly co-organizes the Product Tank Hamburg meetup to
promote the local product management community.
About Hannah Flynn
Hannah went to The University of Chicago, where she majored in Environmental Studies with a concentration in
Economics and Policy. She now works with Aggregage on social media strategy and webinar production on sites
such as Product Management Today, B2B Marketing Zone, and Customer Experience Update.

5. @herbigt
Insights for Today

6. Why care about
Analytics at all?

7. To beat the HiPPO

8. Product Vision & Strategy
Product Discovery Product Delivery
Analytics
Story
Mappings
User
Interviews
Webinar: How To Build Data-Informed Products by @herbigt

9. Product Vision & Strategy
Product Discovery Product Delivery
Analytics
Story
Mappings
User
Interviews
Webinar: How To Build Data-Informed Products by @herbigt

10. Source: uxdesign.cc

13. 1. Get caught up in micro-conversions.
2. Confuse data with why people are doing things.
3. Leaving interpretation of data to everyone themselves
(confirmation bias).
4. Confuse today’s behavior with tomorrow’s potential.
Analytics DON’Ts
Webinar: How To Build Data-Informed Products by @herbigt

14. Analytics DOs
Webinar: How To Build Data-Informed Products by @herbigt
1. Sanity check your first assumptions about user
problems.
2. Use it as a foundation for your hypotheses.
3. Objectify understanding about outcomes.
4. Monitor your product’s ongoing performance.

15. Analytics Tool Landscape
Webinar: How To Build Data-Informed Products by @herbigt
Web/App
Data
A/B Testing Heatmaps
DWH/Back-
end
Tracking
Data Viz

16. Analytics Tool Examples
Webinar: How To Build Data-Informed Products by @herbigt
Web/App
Data
● Used by Product
Managers/Analytics Managers
● Mirrors current behavior

17. Analytics Tool Examples
Webinar: How To Build Data-Informed Products by @herbigt
A/B Testing
● Used by Product
Manager/Conversion Optimization
Managers
● Helps to decide which
design/version performs better
● Lots of craftsmanship required to
nail

18. Analytics Tool Examples
Webinar: How To Build Data-Informed Products by @herbigt
Heatmaps
● Used by Product Managers and UX
Designers to spot Usability gaps
● Helps to move closer to the why
behind analytics data
● Requires few “expertise” to distill
information

19. Analytics Tool Examples
Webinar: How To Build Data-Informed Products by @herbigt
DWH/Back-
end
Tracking
● Backbone behind analytics tools
● Used by data analytics managers
● Helpful to double check frontend
analytics data
● Allows to unify data from various
silos

20. Analytics Tool Examples
Webinar: How To Build Data-Informed Products by @herbigt
Data Viz
● Makes analytics data tangible
● Allows to spot similarities between
and compare multiple sources
● Unmatched flexibility for building
custom dashboards

21. Great, A/B Test
all the things, right?

23. Why not?
Webinar: How To Build Data-Informed Products by @herbigt
● People fancy A/B Testing for the sake of it
● Red flags to watch out for:
○ Lack of hypotheses
○ Used to resolve conflicts → getting consensus and
leaving potential on the table for progress
○ Only marginal differences in variations

24. The biggest A/B Testing
Misconceptions + Pitfalls

25. Sharing Results too early
Webinar: How To Build Data-Informed Products by @herbigt
● Tests take time to become valid
● Don’t give in to the nagging of stakeholders to ‘take a
look’
● Instead, make stats and significance calculation visible

26. Solely relying on calculators
Webinar: How To Build Data-Informed Products by @herbigt
● Calculators help you to objectify, but factor in all
environmental factors
● E.g. Competing A/B tests, changing discount campaigns
changing selection criteria

27. Bonus: Validity Rule of thumb
Webinar: How To Build Data-Informed Products by @herbigt
1. 2 weeks minimum duration to capture ‘seasonal’
fluctuations
2. 500 conversions per variation (ideally 1.000)
3. Significance of >95%

28. ● Don’t assume you can automatically influence bottom
line through micro-conversions at top of the funnel
● We tend to run smaller A/B tests without connecting
these KPIs to the overall company goals
● Track and compare cohorts over time to identify impact
of test change on e.g. revenue
Focussing on wrong KPI
Webinar: How To Build Data-Informed Products by @herbigt

29. Focussing on wrong KPI
Webinar: How To Build Data-Informed Products by @herbigt
+20% Top of Funnel
Doesn’t carry through
Increase in Quantity
can lead to decrease in
Quality
= Lower Bottom Line
Conversions

30. Webinar: How To Build Data-Informed Products by @herbigt
Travel Group Syndrome

31. A/B Test Results ≠ User Feedback
Webinar: How To Build Data-Informed Products by @herbigt
● ‘Version A won, so our users like this one more!’
● A/B Tests don’t tell you why people are doing sth.
● It’s the unfortunate discrepancy between what people
say they do and what they actually do.
● Check back it’s mid- to long-term impact on your
product by conducting some 1:1 user interviews digging
into what users actually made them e.g. click more

32. Don’t A/B Test entire Projects
Webinar: How To Build Data-Informed Products by @herbigt
● A/B Tests work for further evolution of existing products
● Don’t help much for new ideas or innovation
● You can’t test business models
● Instead, try quantitative or qualitative experiments to
validate individual core hypotheses

33. Quantitative vs.
Qualitative Validation

34. Webinar: How To Build Data-Informed Products by @herbigt

35. Validation Example at XING

36. It’s about indicators
not evidence

37. Low Traffic B2B Environments
● Don’t look for something what’s not there
● Focus on key metrics like weekly active users or NPS
● 10 well executed and recruited user interviews beat
obsessing over micro metrics
● Make capturing user feedback a habit and core part of
your teams processes

38. Summary

39. Thanks!
tim@herbigt.com
herbigt.com
@herbigt
herbig.blog/book

40. Q&A
Hannah Flynn
With: Moderated by:
Director, iridion
Linkedin page: linkedin.com/in/herbigt/
Twitter ID: @herbigt
Website: herbigt.com
Book: herbig.blog/book
Tim Herbig
Site editor, Product Management Today
Linkedin page: linkedin.com/in/hannahmflynn
Twitter ID: @prodmgmttoday
Email: hannah@aggregage.com
Website: productmanagementtoday.com
www.productmanagementtoday.com/webinar-series/analytics-in-action
www.projectmanagementupdate.com/webinar-series/analytics-in-action
www.businessinnovationbrief.com/webinar-series/analytics-in-action