The document discusses the lean analytics cycle of metrics, hypothesis, experiment, and act. It provides examples of how Hello Bar used this process to improve their installation rate. They found a low installation rate in metrics, hypothesized that more options would increase installations, tested this in an experiment, and achieved a 40% increase. Through dozens of experiments, their rate increased by 89%. The document encourages analyzing metrics to find opportunities, forming hypotheses through research, rigorously testing hypotheses, and making data-driven decisions.

1. Lean Analytics
Lean Product & Lean UX Silicon Valley - July 2014
Hiten Shah
hnshah@gmail.com
Download the slides: h p://kiss.ly/leanprod2014

3. Continuous Learning & Improvement

4. Don’t know where to start?

5. Wondering if you’re working on the right things?

6. Or, you just want to hear me babble!

7. HTTP://KISS.LY/LEANLYTICSBOOK

8. Lean Analytics helps you Optimize for Learning

9. Lean Analytics Cycle
AVINASH KAUSHIK HTTP://KISS.LY/LEANAC

10. Metrics
→
Hypothesis
→
Experiment
→
Act

11. HTTP://HELLOBAR.COM

12. What is Hello Bar?

21. Metrics
Figure
out
what
to
improve
Hypothesis
Make
an
educated
guess
Experiment
Test
your
guess
Act
Decide
what’s
next

22. Our funnel showed a low installation rate
Hello Bar Funnel Metrics
Steps Visitors Homepage
URL Entries
Created
First Bar
Completed
Registration
Installed
Totals 15068 1868 716 688 398
% prev step 12.40% 38.33% 96.09% 57.85%
% drop off -87.60% -61.67% -3.91% -42.15%
% of visitors 100% 12.40% 4.75% 4.57% 2.64%

23. We
achieved
a
40%
increase
for
our
installa2on
rate
with
just
this
one
first
experiment.
The Lean Analytics Cycle At Work

24. Original Hello Bar installation screen

25. Metrics
Figure
out
what
to
improve
Hypothesis
Make
an
educated
guess
Experiment
Test
your
guess
Act
Decide
what’s
next

26. Find out why people don’t install Hello Bar

27. Didn’t Meet My Expectations

28. I Had Trouble Installing It

29. I Don’t Have A Website

30. Not Ready To Install

31. Other

32. Hello Bar Experiment Hypothesis
If
more
installa2on
op2ons
are
given,
then
more
people
will
add
Hello
Bar
to
their
website
due
to
the
need
for
a
WordPress
plugin
and
ability
to
email
the
code
to
someone
else.

33. Metrics
Figure
out
what
to
improve
Hypothesis
Make
an
educated
guess
Experiment
Test
your
guess
Act
Decide
what’s
next

34. Created a new flow to test

35. I can install code myself.

36. 41.67% responded with “WordPress Plugin”

37. 25% responded with “email my developer / web designer”

38. I need help.

39. Metrics
Figure
out
what
to
improve
Hypothesis
Make
an
educated
guess
Experiment
Test
your
guess
Act
Decide
what’s
next

40. Full funnel metrics for the experiment
Variation
Steps Visitors Homepage
URL Entries
Created
First Bar
Completed
Registration
Installed
Totals 7456 906 342 329 276
% prev step 12.15% 37.75% 96.20% 83.89%
% drop off -87.85% -62.25% -3.80% -16.11%
% of visitors 100% 12.15% 4.59% 4.41% 3.70%
Control
Steps Visitors Homepage
URL Entries
Created
First Bar
Completed
Registration
Installed
Totals 7469 925 353 341 197
% prev step 12.38% 38.16% 96.60% 57.77%
% drop off -87.62% -61.84% -3.40% -42.23%
% of visitors 100% 12.38% 4.73% 4.57% 2.64%

41. Resulted in a 30% installation rate increase
HTTP://ISVALID.ORG
Experiment resulted in a 40% improvement

42. ACer
a
dozen
more
experiments,
our
installa2on
rate
is
now
hovering
between
5%
and
7%
That’s
an
addi;onal
89%
increase!!!
The Lean Analytics Cycle At Work

43. Metrics
→
Hypothesis
→
Experiment
→
Act
How You Can Put It Into Practice

44. Metrics
Figure
out
what
to
improve

45. “What gets measured, gets managed.” Peter Drucker

46. Baselines
DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA

47. Funnels
KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL

48. SaaS Funnel
KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL

49. eCommerce Funnel
KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL

50. Revenue
BUFFER BAREMETRICS HTTP://KISS.LY/BUFFERBARE

51. Spreadsheets and Calculations
HELLOBAR HTTP://HELLOBAR.COM
Steps Visitors Homepage
URL Entries
Created
First Bar
Completed
Registration
Installed Created
Second Bar
Totals 15068 1868 716 688 398 193
% prev step 12.40% 38.33% 96.09% 57.85% 48.49%
% drop off -87.60% -61.67% -3.91% -42.15% -51.51%
% of visitors 100% 12.40% 4.75% 4.57% 2.64% 1.28%

52. Hypothesis
Make
an
educated
guess

53. “A solid test hypothesis is an informed solution to
a real problem – not an arbitrary guess. The more
research and data you have to base your
hypothesis on, the be er it will be.”
— Michael Aagaard, h p://kiss.ly/abthypo

54. Research
Tools
&
Tac;cs

55. Customer Development
HTTP://WWW.CUSTDEV.COM

56. Jobs To Be Done #JTBD
HTTP://JOBSTOBEDONE.ORG

57. Usability Testing
HTTP://USERTESTING.COM

58. Live Chat
HTTP://OLARK.COM

59. Mouse Tracking
HTTP://CRAZYEGG.COM

60. Long Form Surveying
HTTP://SURVEYMONKEY.COM

61. On-Page Surveying
HTTP://QUALAROO.COM

62. Personalized Messaging
HTTP://INTERCOM.IO

63. Email a survey to people
HTTP://SURVEYMONKEY.COM

64. Ask people why they canceled
HTTP://SURVEYMONKEY.COM

65. Get feedback right when they cancel

66. Exit Surveys
HTTP://QUALAROO.COM

67. Discover what’s missing
HTTP://QUALAROO.COM

68. Find out what words people use to describe your product
HTTP://QUALAROO.COM

69. Learn about people’s expectations
HTTP://INTERCOM.IO

70. Learn what frustrates people
HTTP://INTERCOM.IO

71. More actionable reading…
QUALITATIVE DATA HTTP://KISS.LY/QUALABT

72. Experiment Hypothesis Formula
DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA

73. Experiment
Test
your
guess

74. Only
1
out
of
5
tests
win.
4
out
of
5
tests
cost
you
money.
$

75. 1)
Which
top
metric
are
we
trying
to
improve?
2)
What
can
you
anchor
your
metric
against?
3)
What
is
your
hypothesis
about
why
this
metric
is
at
x%?
4)
What
do
we
expect
from
this
experiment?
5)
What
was
the
result
and
what
did
we
learn
from
it?
HTTP://KISS.LY/6QSAB
Answer key questions for each experiment

76. Prioritize your experiments
HTTP://KISS.LY/ABPIE

77. Don’t copy “best practices” blindly

78. Learn what works best for you

79. Act
Decide
what’s
next

80. Explore
the
results
and
make
data
informed
decisions.
THE ANALYTICS THAT MATTER TO FACEBOOK HTTP://KISS.LY/FBINFORM

81. KISSmetrics A/B Testing Report
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

83. We know that people love Google Analytics

84. KISSmetrics Experiment Hypothesis
If
people
authen2cate
with
Google,
then
more
people
will
sign
up
due
to
their
exis2ng
familiarity
with
Google
Analy2cs
and
alignment
with
our
winning
headline.

85. KISSmetrics Homepage Test Control

86. KISSmetrics Homepage Test Variation

87. KISSmetrics Homepage Test Signup Conversion Rate
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

88. KISSmetrics Homepage Test Received Data
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

89. KISSmetrics Homepage New Control

90. Inspiration

91. KISSmetrics Homepage New Test Variation

92. KISSmetrics Homepage Test Signup Conversion Rate
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

93. KISSmetrics Homepage Test Received Data
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

94. Growth!
HTTP://KISSMETRICS.COM

95. Metrics
Figure
out
what
to
improve
Hypothesis
Make
an
educated
guess
Experiment
Test
your
guess
Act
Decide
what’s
next

96. Hiten Shah
hnshah@gmail.com
@hnshah
You will get all you want in life if you help enough
other people get what they want.
Zig Ziglar
Download the slides: h p://kiss.ly/leanprod2014