When done right, experimentation can help you validate the product you’re building and create winning customer experiences. And it doesn’t take a big engineering team to make this happen. TrustRadius, the most trusted review site for business technology, uses experimentation to build an online community through website and server-side experimentation. The small but mighty TrustRadius team runs experiments throughout the buyer’s journey to engage different user personas and understand outcomes in real-time. Watch the webinar recording featuring Rilo Stark, product manager at TrustRadius, and Jack Peden, senior software engineer, to understand their data-driven experimentation strategy and how TrustRadius uses Optimizely Web and Full Stack products to tailor experiences to different customer segments and mitigate risk through A/B/N and painted door tests.

1. 1
Test Everything:
TrustRadius Delivers Customer
Value with Experimentation

2. 2
Rilo Stark
Product Manager
TrustRadius
Speakers
Jack Peden
Senior Software Engineer
TrustRadius
Lauren Brennan
Product Marketing Manager
Optimizely

3. ● Customize the widgets on your page
to your preference
● This webinar is recorded and you will
receive the link with the slides in the
next few days
● We will have time for questions!
Please submit them in the Q&A box
on the left side of the screen
Housekeeping

4. 4
The only solution built for your entire team: marketers, product
managers, e-commerce specialists, and data analysts.
What
We Do
A unified platform for A/B/n testing and targeted experiences
across the entire customer journey.
World’s #1 digital laboratory - Optimizely customers have
run over 1.8M experiments on our platform since 2010.
Put customers at the center of your
experience optimization & delivery process
Leader in Site Experimentation &
Optimization
Measure customer data from real users in production
environments, so that every experience delivered is high-
quality and high-value.
Built for the whole team

5. Full Stack
Optimizely Platform
Built for product, engineering, growth, and data teams
Javascript snippet experimentation
enabling optimization through an easy
to use WYSIWYG visual editor.
Enterprise Expertise
Real-time
Data & Statistics
Progressive Delivery &
Experimentation
Enablement Services Ongoing Support Training & Documentation
Governance Security Scale
Zero-latency feature-flagging and in-code
experimentation implemented via 12 SDKs
or as a microservice
Web
Rollouts:
Standalone, free feature flagging w/one
experiment
Performance Edge: Faster experiments
processed at the edge (CDN)
Stats EngineData Integration Flexible analysis
Open APIs

6. 6 #TruthSells
Test Everything:
TrustRadius Delivers
Customer Value with
Experimentation

7. 7 #TruthSells#TruthSells
Agenda
Kick-off
Getting Started with Experimentation
• North Star Metrics
• Experimentation Philosophy
• Testing Our Way Into Testing
What We’ve Learned
• When to use Full Stack?
• When to use Web?
• Tips & Tricks for starting Full Platform
Experimentation
Q&A

8. 8 #TruthSells#TruthSells
Rilo Stark
Product Manager
/rilo-stark
Jack Peden
Sr. Software Engineer
/jack-peden
trustradius.com

9. 9 #TruthSells#TruthSells
Software Reviews
You Can Trust
• Most trusted customer voice and
platform
• We help software buyers make great
decisions
• We help technology vendors get and
great customers
• Over 1 million monthly users

10. 10 #TruthSells
Audience Poll

11. 11 #TruthSells#TruthSells
North Star Metrics
Our mission is to make it easy for people to buy, use, and sell
technology.
Metrics for building our community include: REACH
Reach Monthly sessions
Keyword rankings
Impact Site satisfaction scores
Decision impact surveys
Authority Organic registration rate
Repeat rate

12. 12 #TruthSells#TruthSells
Experiments Improve Product
Decisions
“Fast, cheap, or good: You can have
any two.”
Experiments help us stay user-
centered, mitigate risk, and
increase adoption by
incorporating user feedback
early and often.

13. 13 #TruthSells#TruthSells
Program Tenets
Optimize Existing
• Find the best solution among many
possibilities
• Seize low-hanging fruit
• Avoid costly mistakes
Test Into New
• Cost:Benefit analysis of a new initiative
• Controlled releases
• Establish baselines & best practices

14. 14 #TruthSells#TruthSells
Google Optimize was good for us to get our feet
wet with experimentation, but we quickly
outgrew it.
We used our own product to find a better
solution for our business needs!
Testing Our Way Into Testing
Reviews are an important
part of the buying decision.
Unbiased, trusted reviews are
the holy grail. I want real use
cases in the reviews I read… I
want the good, the bad, and
most importantly the UGLY.
TrustRadius is the leader in
tech solution review sites. It
has the most unbiased, candid
reviews I can find.
Chris Salles
Director,
Learning
Audible, Inc.

15. 15 #TruthSells
When to use Full Stack?

16. 16 #TruthSells#TruthSells
Page Speed Metrics
Observation: TrustMaps are used by Buyers, but slow
page speed.
Hypothesis: If we reduce the page speed impact of the
TrustMap, we will improve the user experience.
Key Metrics: bounce rate, page speed, and TrustMap
interactions
Results: After experimenting with different treatments,
the static image variation improved our key UX metrics
the most!
Why Full Stack? This content is generated server-side, so
to conditionally include it we needed to deliver this
experiment using Full Stack.Tip: Full Stack can help you test improvements for page
speed metrics.

17. 17 #TruthSells#TruthSells
Capture User Inputs
Observation: Buyers seek integration details, but this info
is buried.
Hypothesis: If buyers can easily check integrations to
create a short list of viable products, our product will have
more impact on their buying decision.
Key Metrics: clicks on this CTA, # of flows completed, # of
user inputs
Results: After a series of different locations and flows, an
insufficient % of the community was engaging so we
abandoned the feature.
Why Full Stack? Each iteration either depended on user
input, or prompted an “I’m sorry” modal after the click
occurred.
Tip: Use Full Stack for tests that require user input or new
workflows.

18. 18 #TruthSells#TruthSells
Repurpose Server-Side Element
Observation: Half of all software Buyers want a vendor
Demo, but this CTA doesn’t persist as they scroll the product
Hypothesis: If we make it easier for buyers to find the Demo
CTA, we will increase the click-through-rate and the page
score.
Key Metrics: CTA clicks, bounce rate, time on page
Results: this experiment is currently live! The precursor
experiment resulted in a 2X increase in clicks.
Why Full Stack? The test location is a part of our code that is
rendered server-side, so being able to run the test there was
option.Tip: Check your logs for Optimizely messages when testing
server-side, including which test bucket a user is assigned
into.
Precursor Web Experiment
Current Full Stack Experiment

19. 19 #TruthSells#TruthSells
Three Use Cases for Full Stack

20. 20 #TruthSells
When to use Web?

21. 21 #TruthSells#TruthSells
Modify Existing CTA
Observation: Vendors want to claim their listing & engage
with buyers researching their product – but the claim link
is buried!
Hypothesis: If we make it easier for vendors to find the
claim link, we will increase the number who claim their
profile.
Key Metrics: clicks on claim CTA, completed claims,
bounce rate, time on page
Results: Increased weekly vendor profile claims over
200%
Why Web? Moving a pre-existing link did not check any
of the boxes where server-side makes more sense.
Tip: Set client-side console log levels via the
optimizely_log=info query parameter.
Original CTA (in Sidebar)
Experiment location (in header)

22. 22 #TruthSells#TruthSells
Painted Door
Observation: Buyers often request a synthesized version
of our content and data.
Hypothesis: If we make it easier for buyers to synthesize
the pros and cons content in reviews, page engagement
and satisfaction will increase.
Key Metrics: word cloud clicks, pageviews/session,
bounce rate, session duration
Results: +150% pageviews/session & +157% session
duration on mobile; but desktop was flat other than an
increased bounce rate.
Why Web? We were able to minimize dev time by
manually creating word clouds and targeting individual
pages.
Tip: Build the minimum functionality needed to validate
hypotheses.

23. 23 #TruthSells#TruthSells
Repurpose Client-Side
Element
Observation: Buyers seek authoritative, third-party
content throughout their journey – and we have this with
Buyer’s Guides!
Hypothesis: If we merchandise our Buyer’s Guides
effectively, we will increase the number of buyers who
benefit from this content.
Key Metrics: conversion rate of timed modal
Results: 2x conversion rate from first iteration to current
baseline.
Why Web? This content does not need to be present on
page load, and this modal already exists.Tip: Use existing elements to test new value propositions.

24. 24 #TruthSells#TruthSells
Three Use Cases for Web

25. 25 #TruthSells#TruthSells
Tips for Starting Full Platform Experimentation
• Don’t be afraid to fail! A “loss” is better than a neutral result.
• Keep your experiments as simple as possible. What is the one thing you’re trying to learn?
• Use the right tool for the job. Being able to choose Optimizely Web or Full Stack based on the
nature of the experiment will increase test velocity.
• Share experiment results with your company - wins AND losses.
• Do >1 experiment per key metric. You will accumulate learnings over time, which – combined
with other inputs – improve interpretation of each experiment, and decision to fish or cut bait.
• Identify ALL key metrics that could be impacted by each experiment. Which metrics might be
adversely impacted by this change? Incorporate them in your test plan.

26. 26 #TruthSells
Q&A
Visit trustradius.com to learn more

27. 27
Questions?

28. 28
What’s Next
Watch our Full Platform On-Demand Demo
Pump up your road map with our
Product Experimentation Boot Camp