1.
Building a Culture Of
Testing
SPENCER MANN - VP OF GROWTH AT LUCID SOFTWARE

2.
@Kissmetrics
#KissWebinar
@thuelmadsen

3.
Thue is the Kissmetrics Webinar Wizard and Marketing
Ops Manager. Before joining forces with Kissmetrics, he
was a Lyft driver in SF, which is also how he ended up as
a Kissmetrics marketer. Whenever Thue is not trying to
automate everything around him, you can find him hiking
in the Sierras.
THUE MADSEN
Marketing Operations Manager, Kissmetrics
@ThueLMadsen
Spencer Mann is the VP of Growth at Lucid Software.
Prior to working at Lucid Software, he worked in the
chemical industry and with alternative sources of energy.
He has a BS and MS in biological engineering and an
MBA. Spencer is passionate about AB testing and
spending time with his family (only occasionally do these
passions mix).
SPENCER MANN
VP of Growth, Lucid Software
@spencer_mann

4.
@lucidchart
#KissWebinar
@spencer_mann

5.
1 THE GROWTH TEAM AT LUCID
2 HOW DID WE BUILD A CULTURE OF TESTING?
3 WHERE TO FIND GOOD IDEAS
WHAT WE’LL COVER
4 HOW TO GET “BUY IN” FOR BIG TESTS

6.
The Growth Team
at Lucid

7.
THE GROWTH TEAM AT LUCID
Payments!
Registrations!
Payment Quality!
Retention!
Team Growth!
Payments
40%
Registrations
20%
Payment Quality
18%
Retention
18%
Team
Growth
18%
EXPERIMENT FOCUS AREAS
Over 200 experiments in 2016…
around 35% success rate.
NOTABLE TESTS
Credit card required for trial!
Multiple paywalls!
Several price increases!
Onboarding!
Personalization!
Registration experience!

8.
How Did We Build a
Culture of Testing?

9.
SOME EXPERIENCES THAT DROVE OUR CULTURE OF TESTING
Four experiences…
§ The first time data suggested we do something everyone hated
§ Why would we test something we already know is a good idea?
§ Few things are worse than a test that can’t fail
§ “Let’s save time by just doing this all at once.”

10.
THE FIRST TIME DATA SUGGESTED WE DO SOMETHING EVERYONE HATED
The Hypothesis “If we shorten our trial length from 2 weeks to 1
week we will significantly increase the number of users who
experience that high conversion event.”
The Problem Many of the biggest opportunities involved changing
something put in place by a founder.
§ Good leadership is more interested in being successful than
being right.
§ Most scary tests can be de-risked by testing a low value portion
of the population, or only testing a component of the principle.
§ Data is the best source for high value test ideas.
Learnings
TotalPayments

11.
WHY WOULD WE TEST SOMETHING WE ALREADY KNOW IS A GOOD IDEA?
The Hypothesis “If we increase our price by 20% today, tomorrow we will make 20%
more money.”
The Problem In the early days of a startup you are forced to make gut decisions that
keep you moving fast. Transitioning from that to a culture of deliberate testing and data
based decisions can be uncomfortable at first.
§ People are wrong a lot.
§ Transitioning to a culture of testing is
painful, but obvious in retrospect.
§ AB tests will provide the “what” but not
always the “why”
Learnings

12.
FEW THINGS ARE WORSE THAN A TEST THAT CAN’T FAIL
The Hypothesis “Updating our engineering architecture will improve
site performance, which will drive user engagement and payments.”
The Problem This was not a traditional A/B test. Setting up the
experiment was very expensive so casually failing it was not an
option. Additionally, this was a strategic initiative for the company.
§ Ask yourself, “What will we do if this doesn’t win?”
§ Testing engineering architecture is a great way to diagnose.
§ Analyzing another team’s experiences can be “your Vietnam.”
Learnings
Browser – OS
Drop in
Engagement
Chrome - Linux 34%
Firefox - Mac 23%
Firefox - Linux -2%
Chrome - Mac 2%

13.
LET’S SAVE TIME BY JUST DOING ALL THIS AT ONCE . . .

14.
LET’S SAVE TIME BY JUST DOING ALL THIS AT ONCE . . .
Flight to moon took
many iterative launches.
The Hypothesis “We can save time when testing a change to our
editor by changing both the design and the underlying architecture at
the same time.”
The Problem Hundreds of things changed at once, and it was nearly
impossible to identify the problematic elements.
§ Simplicity is better than complexity in testing.
§ Test components independently if possible.
§ It’s extremely unlikely that a whole-scale change will beat a
multi-year, optimized experience on it’s first try.
§ Long running complex tests can be a “tax” on engineering and
QA resources, sometimes you need to call it and move on.
Learnings

15.
Where to Find Good Ideas

16.
WHERE TO FIND GOOD IDEAS
§ Mapping out flows and conversion rates
§ Focus on areas with high volume and high potential for
improvement
§ Take “growth hacks” with a high degree of skepticism
§ Don’t just “throw things against the wall”
§ Discuss successes and failures with other credible
growth leaders
§ Bi-weekly brainstorming with entire growth team, with
focus area provided
§ Talk with customer facing groups to identify pain points
§ Learn from you failures and iterate
30% of our wins
started as failed
experiments

17.
How To Get
“Buy In” for Big Tests

18.
HOW DO I GET “BUY IN” FOR BIG TESTS?
§ Credibility is a core to success
§ Visually diagram proposed test flow
§ Come prepared with all the relevant data
§ Describe risk and upside potential
§ Calculate test length
§ Break the test into smaller low risk components
§ Call out your own failures and NEVER bias your data
§ Benchmark against other similar companies
§ Provide updates on tests early, but be cautious about calling wins
§ Balance low test investments with not screwing it up
Pro Tips

19.
SOME TOOLS WE HAVE FOUND USEFUL
§ Google Sheets for tracking all tests
and pre-test work
§ Flow chart of test implementation
§ AB Test configuration tool
§ Kissmetrics
Access our AB testing template at:
http://bit.ly/2qjEGv1

20.
HOW WE STRUCTURED OUR GROWTH TEAM
Growth Manager Senior Engineer
2 additional engineers
• Data-focused
• Curious
• Challenges assumptions
• Not afraid to fail
• Expert on payment system
• Business minded
• Enjoys experiments
• High learning agility
• Balanced customer/ business mindset
Analyst
• Curious
• Disciplined
• Communicates well

21.
HOW TO SET YOUR GROWTH TEAM UP TO FAIL
§ No Executive buy-in. Many ideas you test will challenge long-held beliefs
§ No Independence. Continuous optimization requires freedom to be innovative
§ No Engineers. “Borrowing” engineering time is frustrating and doomed to fail
§ No Data. Without good data you can’t identify success or focus areas
Beware these mistakes which can set your growth team up to fail…

22.
SPENCER MANN
VP of Growth, Lucid Software
@spencer_mann
THUE MADSEN
Marketing Operations Manager, Kissmetrics
@ThueLMadsen
Questions?
(Illustrations)
Libby Thomas,
Creative Content Specialist
Lucid Software