Analytics are an increasingly important capability of any large web site or application. When a user selects an option or clicks a button, dozens—if not hundreds—of behavior-defining “beacons” fire off into a black box of “big data” to be correlated with the usage patterns of thousands of other users. In the end, all these little data points form a constellation of information your organization will use to determine its course. But what if it doesn’t work? A misconfigured site option or an errant variable might seem insignificant, but if 10,000 users are firing 10,000 incorrect values concerning their click patterns, it suddenly becomes a problem for the QA department―a department which is often left out of conversations involving analytics. Join Marcus Merrell to learn how analytics work, how to get involved early, and how to integrate analytics testing into the normal QA process, using Selenium and other open source tools, to prevent those misfires from slipping through.

1. T18
Special Topics
5/8/2014 1:30:00 PM
Automated Analytics Testing
with Open Source Tools
Presented by:
Marcus Merrell
RetailMeNot, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Marcus Merrell
RetailMeNot, Inc.
A test architect at RetailMeNot, Inc., Marcus Merrell has written UI and API test frameworks for
several products since 2001. Marcus is obsessed with code design and holds sacred the
philosophy that test frameworks should be approached, developed, and tested just as carefully
as the production software they seek to examine. The successful demonstration of these
principles has led to an intense focus on collaboration between testers and developers, and to a
deep appreciation for code that is at once simple and sophisticated―an API for testers which is
reusable, interface-agnostic, and universal to many problem sets.

3. 4/26/2014
1
Web Analytics
Testing
Marcus Merrell,
RetailMeNot, inc
@mmerrell
Web Analytics – The Basics
Traffic:
Google, Woopra, Yahoo, Crazy Egg
Optimization and Performance:
Optimizely, Google
Competitive Analysis:
Compete.com, watchthatpage.com,
faganfinder.com
Social:
Facebook Insights, Twitalyzer

4. 4/26/2014
2
Web Analytics – Custom
WebTrends
Google
Adobe (formerly Site Catalyst/Omniture)
IBM, formerly CoreMetrics
…This is what we’ll be talking about
Web Analytics - Advanced
Cookies on steroids
Custom key-value pairs
Data Storm!!
Where else can you learn, correlate, and tell
stories from seemingly disparate points of data?
“If you are not paying for it… you are the product
being sold”
--Andrew Lewis (blue_beetle)

5. 4/26/2014
3
A Major Newspaper
Click on a story on nytimes.com – what have
they learned about you?
How far down the page was the story?
How long were you there before you clicked it?
Did you “bounce” to another page before
clicking it?
Were you logged in?
Have you looked at other stories from that
category/keyword/tag cloud before?
How long before you click another link?
If we popped up a story suggestion, did you
click on it?
Abusive Comments
Once you’re on a story, are there
comments?
Did you add a comment?
Did you click a “show more” link to expand
a comment?
Did you do that multiple times in a thread?
Across multiple threads?
Did you click an ad?
Was it before of after you looked at
comments?

6. 4/26/2014
4
Web Analytics – Telling a Story
Now extrapolate for 100,000 users
Are people consistently clicking on stories
“below the scroll”?
If so, should that story be further up?
Is something wrong with the top stories?
Are stories with pictures more popular?
Are certain keywords more likely to get
comments? Abusive comments?
How does ad click-through change based
on abusive comment %?
Making a Hypothesis
Do abusive comments drive down the
likelihood of users’ clicking on ads?
Question for A/B test: Will ad revenue
increase if we disable comments on
stories about politics?
A path: disable comments on stories with
keyword “politics”, “election”, or “pundit”
B path: keep comments enabled

7. 4/26/2014
5
Drawing a Conclusion
Test B, “variation”, won
Disable comments when “abusive
comment” rate rises above threshold
Now, refine the threshold with further A/B
tests
Real-world Examples
“People who bought this also bought…”
Shopping cart—shipping & tax calculation
Pairing account creation with check-out
Suggesting products and content based
on cookie, not login

8. 4/26/2014
6
Why You Should Care
What if the beacons they’re sending
contain the wrong information?
But furthermore…
This is everywhere
It is only growing
Companies are becoming smart
(Really really smart)
You do not want to miss this opportunity to
provide value
Why You Should Really Care
As a tester:
There is a whole team of people working on
this
It gets worked into features as they are
developed
It is rarely called out separately in a
scheduled task
It rarely receives QA outside of the PM and
BI people who really care about it

9. 4/26/2014
7
Fortunately, It’s Easy
All of this comes down to an extra HTTP
request or two, made during a some
navigation event
As long as you can intercept this request
and read from it, you can verify the data
within it
Examples
Wells Fargo (s.gif)
Amazon (a.gif)
Netflix ([….]?trkId=xxx, beacon?s=xxx)
The New York Times (pixel.gif, dcs.gif)
OpenTable (/b/ss/otcom)
(and RetailMeNot)

10. 4/26/2014
8
Classic Approach
Marketing asks the BI team to figure out
our ROI on TV ads during a period of time
BI requests PM to create a series of
analytics
PM gives Dev the particulars
Dev assigns the code task to the newest
person on the team
If anyone tests it, it’s also the newest
person on the team
Classic Approach
Manual testing of web analytics is about
as exciting as reconciling a large column
of data with another large column of
data
…what if it’s wrong?
…what if it changes?
…why not let the software do it?

11. 4/26/2014
9
What We Do
Testing the web component:
Launch the browser
Do the navigation
Launch the debugger window
Parse the source code for “key=value”
pairs
Look for the one you want
What We Also Do
Proxy FTW
Safer, more reliable, probably faster
Examine ALL HTTP requests, looking for
some token
Parse out the request body for
“key=value” pairs
Look for the one you want

12. 4/26/2014
10
Tech Stack
Maven
Spring
TestNG
Selenium
BrowserMob Proxy
Features
Scalability
Autoscaling Grid!
Data-driven tests
Database
Validation
Test data creation

13. 4/26/2014
11
Execution
TeamCity kicks off…
…Maven job, which executes…
…TestNG tests
Reporting
Report to a dashboard
Indicates “PASS”, “FAIL”, and a “Staleness
factor”

14. 4/26/2014
12
Conclusion
Open source = robust, well-supported,
tight control
Scalable, particularly from a license point
of view
Deliver real value—million-dollar decisions
are made with this data