Joel Harvey details new ways to look at quality assurance and outlines the approach his company, Conversion Sciences ensures it's tests don't break client websites

1. OPTIMIZATION QA:
How to Avoid
Launching Tests that
Break Your Site
When you break shit you
pollute data, lose money
and eventually get fired.
Here’s how to avoid that
pain and look like a hero in
the process.
@joeljharvey #convcon
Joel Harvey, COO
Conversion Sciences

2. @joeljharvey #convcon
Goal

3. @joeljharvey #convcon
What’s QA?

4. @joeljharvey #convcon
Let’s say that …

5. @joeljharvey #convcon
30% of your
visitors
get a broken
experience on
your site

6. @joeljharvey #convcon
Y
O
U
R
SITE IS
BROKEN ON ONE OF
T
H
E
S
E
SCREENS

7. @joeljharvey #convcon
No One On Your
Team Knows

8. @joeljharvey #convcon
20% RPV
IE BUG

9. @joeljharvey #convcon
INTERNET
EXPLORER

10. @joeljharvey #convcon
You’re about to hire this guy as your CPA

11. @joeljharvey #convcon
QA MATH

12. @joeljharvey #convcon
Internet Explorer Traffic: Feb 2013
Internet Explorer Traffic: Feb 2014

13. @joeljharvey #convcon
 120% Increase Internet Explorer (IE)
Revenue
 20% Increase in All Revenue
 20% Increase in IE Average Order Value
 167% Increase in IE Conversion Rate
$1,500,000 Per Year

14. @joeljharvey #convcon
Who Are
You?

15. @joeljharvey #convcon
Who Am I?
- COO & Managing Partner,
Conversion Sciences
- Conversion Scientist
- Data Geek
- Entrepreneurial MBA

16. @joeljharvey #convcon

17. RE-DEFINING QA
#1
@joeljharvey #convcon
STOP STEALING FROM YOURSELF

18. @joeljharvey #convcon
30% of your
visitors
get a broken
experience on
your site

19. QA DRIVES
REVENUE
@joeljharvey #convcon

20. OLD BORING
DEFINITION
@joeljharvey #convcon

21. @joeljharvey #convcon

22. QA is just a Cost
Center
We can’t
support
everything!
QA prevents us
from being agile
People on really old
devices probably
aren’t buyers
QA is an IT /
Development Task
@joeljharvey #convcon

23. QA IS A
MARKETING
FUNCTION
@joeljharvey #convcon

24. QA STRATEGY
#2
@joeljharvey #convcon
INFUSE QA INTO YOUR OPTIMIZATION DNA

25. @joeljharvey #convcon
1. CULTURE
3. TECH PARADIGMS
2. REAL USER USE CASES

26. QA IS
OPTIMIZATION
@joeljharvey #convcon

27. 1. Understand your visitors
@joeljharvey #convcon

28. 2. Make sure control is not
broken
@joeljharvey #convcon

29. 3. Generate great hypotheses &
test them
@joeljharvey #convcon

30. @joeljharvey #convcon
4. Don’t
pollute
your test
data

31. 5. Do what the data tells you to
do
@joeljharvey #convcon

32. BUILDING A
QA CULTURE
@joeljharvey #convcon

33. @joeljharvey #convcon
#QAHAMMER

34. @joeljharvey #convcon
Build Destroy

35. HAMMER UP
@joeljharvey #convcon

36. BUY IN
@joeljharvey #convcon

37. @joeljharvey #convcon

38. @joeljharvey #convcon
20% RPV
IE BUG

39. @joeljharvey #convcon

40. REAL USE
CASES
@joeljharvey #convcon

41. 90%
@joeljharvey #convcon

42. VISITORS
DON’T USE
YOUR SITE THE
WAY YOU
THINK THEY DO
@joeljharvey #convcon

43. @joeljharvey #convcon
SCREEN RECORDING

44. @joeljharvey #convcon
USER TESTING

45. DOCUMENT
AND QA ALL
USE CASES
@joeljharvey #convcon

46. YOUR JOB IS TO
BREAK YOUR
SITE
@joeljharvey #convcon

47. @joeljharvey #convcon

48. WHAT
SHOULD YOU
SUPPORT?
@joeljharvey #convcon

49. 5%
@joeljharvey #convcon

50. @joeljharvey #convcon
Old New
BROWSER SUPPORT
 Internet Explorer 8
 Chrome 7
 Firefox 4
 Safari 4.1
 iOS 5.1 *
 Internet Explorer 11
 Chrome 33
 Firefox 27
 Safari 7.0
 iOS 7.1 *

51. @joeljharvey #convcon
OS SUPPORT
Old New
 Windows XP
 OS X 10.4 (Tiger)
 Android 2.3
 iOS 5.1
 Windows Phone 7
 Windows 8
 OS X 10.9 (Maverick)
 Android 4.4
 iOS 7.1
 Windows Phone 8.1

52. 5% MINIMUM
@joeljharvey #convcon

53. @joeljharvey #convcon

54. @joeljharvey #convcon

55. @joeljharvey #convcon

56. @joeljharvey #convcon

57. @joeljharvey #convcon

58. @joeljharvey #convcon

59. @joeljharvey #convcon

60. @joeljharvey #convcon

61. @joeljharvey #convcon

62. QA TOOLS
#4
@joeljharvey #convcon
BUILD YOUR QA’TION STATION

63. @joeljharvey #convcon
SEE YOUR
SITES LIKE
YOUR
VISITORS

64. iOS 7.1?
@joeljharvey #convcon

65. @joeljharvey #convcon
SOFTWARE
vs.
HARDWARE

66. @joeljharvey
#convcon
iPad,
iOS 5
iPad,
iOS
7.1
Windows
Vista
Android 4.3
iPhones:
iOS4, 5, 7
Android:
2.3.6, 4.0.4,
Windows
8, no
updates
OS X 10.6 Windows 8,
regular updates
+ VM’s
OS X 10.8
OS X
10.9
OS X 10.7
+ VM’s
Windows XP
Windows
7

67. @joeljharvey #convcon

68. @joeljharvey #convcon
?

69. @joeljharvey
#convcon
1. Browser
2. Operating System
3. “Other”
- Visitor Internet Speed
- Browser Plugins
- Flash
- Resolution
QA’TION STATION CHECKLIST

70. QA PROCESS
#4
@joeljharvey #convcon
BE THE HAMMER

71. QA
EXPERIENCES,
NOT PAGES
@joeljharvey #convcon

72. @joeljharvey #convcon

73. @joeljharvey #convcon

74. @joeljharvey #convcon

75. TEST QA
PROCESS
@joeljharvey #convcon

76. 1.
Build Your Test
@joeljharvey #convcon

77. 2.
Use Cookie, Query
String and/or Location
Targeting so Only You
See Experience
@joeljharvey #convcon

78. 3.
Pause / Disable All
But One Experience
@joeljharvey #convcon

79. 4.
Review Live
Experience In All
Supported
Browser/OS Combos
(like REAL USERS)
@joeljharvey #convcon

80. 5.
Clear Cookies, Pause
Good-To-Go
Experience and
Unpause the Next One
@joeljharvey #convcon

81. 6.
Repeat
@joeljharvey #convcon

82. A FEW TIPS

83. RANDOMIZE
YOUR
APPROACH

84. CAUTION WITH
CLASSES

85. WATCH SCRIPT
STACKS

86. FEAR THE
WYSIWYG

87. OWNERSHIP

88. @joeljharvey #convcon
Need to Borrow a QA Hammer?
Call 1-888-961-6604
and mention this
session for a chance
to win a Free QA
Audit ($15,000 Value)
Conversion Conference San Francisco, 2014
Joel Harvey, Conversion Sciences
Joel@ConversionSciences.com