The document discusses A/B testing and optimization. It covers why companies do A/B testing, common mistakes in testing like poor data quality and not following statistics properly, and how to properly conduct tests. Key recommendations include testing everything, prioritizing high impact tests, understanding customer problems by analyzing metrics and flows, removing friction from the user experience, and building a data-driven culture.

1. By Mariia Bocheva, OWOX BI
A / B testing,
Optimization and Results Analysis

2. ‫שלום‬‫לכולם‬
Mariia Bocheva
Head of Sales and Marketing
5 years @ OWOX
Ecommerce, Analytics, BI, CRO
m.bocheva@owox.com
@BochevaMariia
@mariia.bocheva

3. Why A/B Testing

4. Why A/B Testing?
1. We want to earn more

5. Kano Model

6. 1. We want to earn more
2. We’re normally wrong about what we think is ‘good’
Why A/B Testing?

7. It’s Quiz Time :)

8. Blue Pill or Red Pill
A
Left Hand
B
Right Hand

9. Which One Was Better?

10. Which One Was Better?
L

11. Which One Was Better?

12. Which One Was Better?
R

13. Which One Was Better?

14. Which One Was Better?
R

15. Which One Was Better?

16. Which One Was Better?
R

17. Which One Was Better?

18. Which One Was Better?
LL

19. Which One Was Better?

20. Which One Was Better?
L

21. Which One Was Better?

22. Which One Was Better?
R

23. Which One Was Better?

24. Which One Was Better?
L

25. Common mistakes

26. 1. Quality of data
What Could Go Wrong?

27. 1. Is our GTM/GA set up properly?
2. Do we filter out unnecessary traffic?
3. What do we capture?
4. Where do we store it?
5. How do we track refunds and ROPO-effect?
BS in – BS out

28. A/B Testing Ad Campaigns
Campaign A
Campaign B

29. A/B Testing Ad Campaigns
Campaign A
Campaign B
150 Clicks
100 Clicks

30. A/B Testing Ad Campaigns
Campaign A
Campaign B
150 Clicks
100 Clicks
$1,000
$2,000

31. A/B Testing Ad Campaigns
Campaign A
Campaign B
150 Clicks
100 Clicks
$1,000
$2,000
-$100 in Returns -$500

32. A/B Testing Ad Campaigns
Campaign A
Campaign B
150 Clicks
100 Clicks
$1,000
$2,000
-$100 in Returns -$500

33. 1. Is our GTM/GA set up properly?
2. Do we filter out unnecessary traffic?
3. What do we capture?
4. Where do we store it?
5. How do we track refunds and ROPO-effect?
6. Do we trust the data we have?
BS in – BS out

34. Want to Dig Dipper?
Use this checklist

35. 1. Quality of data
2. Not doing your statistics right
What Could Go Wrong?

36. Doing Statistics Wrong
Calculate test duration

37. 1. Quality of data
2. Not doing your statistics right
3. Big Shot A/B testing
What Could Go Wrong?

38. 1. Quality of data
2. Not doing your statistics right
3. Big Shot A/B testing
4. HIPPO opinion
What Could Go Wrong?

39. HIPPO Opinion

40. HIPPO Opinion
“If we have data, let’s look at
data. If all we have are opinions,
let’s go with mine.”
– Jim Barksdale, former Netscape CEO

41. 1. Quality of data
2. Not doing your statistics right
3. Big Shot A/B testing
4. HIPPO opinion
5. Optimizing for a micro conversion
What Could Go Wrong?

42. What Could Go Wrong?
1. Quality of data
2. Not doing your statistics right
3. Big Shot A/B testing
4. HIPPO opinion
5. Optimizing for a micro conversion
6. Assumed reproducibility

43. 1. Quality of data
2. Not doing your statistics right
3. Big Shot A/B testing
4. HIPPO opinion
5. Optimizing for a micro conversion
6. Assumed reproducibility
7. Copying your competitors
What Could Go Wrong?

44. “Stop Copying Your Competitors:
They Don’t Know What They’re
Doing Either”
– by Peep Laja

45. 1. Quality of data
2. Not doing your statistics right
3. Big Shot A/B testing
4. HIPPO opinion
5. Optimizing for a micro conversion
6. Assumed reproducibility
7. Copying your competitors
8. Unclear hypothesis and expected results
What Could Go Wrong?

46. Hypothesis Kit
1. Because we saw (data/feedback)
2. We expect that (change) will cause (impact)
3. We’ll measure this using (data metric)

47. How to Do it Right
[in my opinion]

48. 1. Test everything
2. Test atomically
3. Prioritize testing
Doing it Right

49. “All models are wrong but some
are useful”
– George Box

50. Prioritize Testing

51. 1. Test everything
2. Test atomically
3. Prioritize testing
4. Focus on the right thing
Doing it Right

52. Focus on the Right Thing

53. 1. Test everything
2. Test atomically
3. Prioritize testing
4. Focus on the right thing
5. Question things you don’t understand
Doing it Right
6. Build data-driven culture

54. “You can have data without
information, but you cannot have
information without data.”
– Daniel Keys Moran

55. 1. Test everything
2. Test atomically
3. Prioritize testing
4. Focus on the right thing
5. Question things you don’t understand
6. Build data-driven culture
Doing it Right
7. Automate testing
8. Understand cross device

56. What Should I Do
Tomorrow?

57. 1. Remove the friction
DIY Recommendations

58. Remove the Friction – Stroop effect
RED BLUE GREEN
RED
YELLOW BLUE RED
YELLOW
GREEN RED BLUE
BLUE

59. Stroop effect – Name the Color
RED BLUE GREEN
RED
YELLOW BLUE RED
YELLOW
GREEN RED BLUE
BLUE

60. Remove the Friction

61. 1. Remove the friction
2. Understand the problem
DIY Recommendations

62. 1. Who is this page for (age, gender, geo)?
Understand the Problem

63. Men or Women?

64. Older or Younger?

65. 1. Who is this page for (age, gender, geo)?
2. Where do customers come from?
Understand the Problem

66. Which Channel Shows the Highest Bounce
Rate?

67. 1. Who is this page for (age, gender, geo)?
2. Where do customers come from?
3. What do we want our customers to do next and what they actually
do?
Understand the Problem

68. User a Flow Report

69. 1. Who is this page for (age, gender, geo)?
2. Where do customers come from?
3. What do we want our customers to do next and what they actually
do?
4. What are the key metrics (conversion rate, bounce rate, time on
page, etc.)?
5. What’s the mix of devices and resolutions?
Understand the Problem

70. How Does it Look Like on Different Devices?

71. Google Analytics It

72. 1. Who is this page for (age, gender, geo)?
2. Where do customers come from?
3. What do we want our customers to do next and what they actually
do?
4. What are the key metrics (conversion rate, bounce rate, time on
page, etc.)?
5. What’s the mix of devices and resolutions?
Understand the Problem
6. How does the page look like on those devices?
7. Is the page load time okay?
8. What’s going on with JS errors

73. Correlation Between JS Errors and Revenue

74. How we deal with JS Errors
Google Analytics
Real Time API
BigQuery
Rest API
Google
APPs Script
Slack Push
User Email

75. App Script

76. Slack Message and Instant Answer

77. “Your customers’ experience is
only as good as its shittiest
part.”

78. 1. Remove the friction
2. Understand the problem
3. See what they see (Hotjar, Webvisor)
DIY Recommendations

79. Record Sessions, Check Heat Maps

80. 1. Remove the friction
2. Understand the problem
3. See what they see (Hotjar, Webvisor)
4. 5 seconds test
DIY Recommendations

81. Takeaways

82. 1. Eliminate friction
2. Understand the problem
3. Don’t go with HIPPO
4. Make sure you have enough data
5. Build the culture
6. Prove that your/others ideas were wrong
7. Learn from the failure
TL;DR

83. Download the Presentation Here
www.owox.com/c/atd2018
m.bocheva@owox.com

84. ?‫שאלות‬
‫תודה‬‫על‬‫תשומת‬‫לבכם‬