Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Automated Analytics
Testing
MARCUS MERRELL, RETAILMENOT, INC
@MMERRELL
What We’re
Talking about
 Conservancy
 Overview of Web/User
Analytics
 Explanation of A/B
Testing
 Why this matters to...
Who Is Conservancy?
 Software Freedom Conservancy
 Protects the branding, copyrights, and licenses for over
40 free and ...
Overview of User Analytics
 Primarily B2C: Banks, shopping, high-volume apps
 Applies to all platforms, including video ...
User Analytics
Hits
Sessions
Users
“If you are not paying for it… you are the product being sold”
—Andrew Lewis (blue_beet...
A/B Testing
 Don’t change everything at once
 Major source of regressions in analytics:
 Analytics will need to change ...
Real-life RMN Example
...&b21=17&…
...&b21=18&…
...&b21=0&…
(wait, shouldn’t that say 18?)
How Is This Related?
 Analytics have historically not been included in test
design
 Major source of regression defects
...
Real-world Examples
 Shopping cart—shipping & tax calculation
 Suggesting products and content based on cookie, not
login
Real-world Examples
 RetailMeNot’s email program
 Category and Merchant “affinity”
Classic Approach
Marketing
Business
Intelligence
Product
Manager
EngineeringQA
(Usually the newest
person on the team)
Why You Should Care
 What if the analytics contain the wrong information?
 But furthermore…
 This is everywhere
 It is...
Why You Should Care
Why You Should Really Care
 As a tester:
 There is a team of people working on this
 It gets worked into features as th...
Fortunately, It’s Easy
 Usually one extra HTTP request, made during a
navigation event
 Intercept this request, then ver...
Process
 Break down into key-value pairs
 Build a Map of “expected results”
 Assert on expected vs actual results
seque...
Test Cycle
@Test
Launch
Browser
Navigate
to Position
Start Proxy
Perform
Main
Action
Stop Proxy
Clean-up
Test
Execution
 Send JSON to the “expectation” end-point
 Includes a “timer” and an alert email
 Later (asynchronously), mor...
Reporting
 Report to a dashboard
 Indicates “PASS”, “FAIL”, and “Staleness”
Alternatives
 Anand Bagmar
 https://github.com/anandbagmar/WAAT
 Tool for testing Omniture Analytics
 Uses Omniture’s ...
Conclusion
 User Analytics are your CEO’s favorite subject!
 Deliver real value—million-dollar decisions are made with
t...
Questions?
 Once again, thank you so very much
 Twitter: @mmerrell
 mmerrell@gmail.com
 I’m also on LinkedIn
Automated Analytics Testing
Nächste SlideShare
Wird geladen in …5
×

Automated Analytics Testing

2.094 Aufrufe

Veröffentlicht am

Selenium Committer Day 2017
Marcus Merrell さんによる講演資料です。
ご本人の好意により公開許可を頂きましたので共有いたします。

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Automated Analytics Testing

  1. 1. Automated Analytics Testing MARCUS MERRELL, RETAILMENOT, INC @MMERRELL
  2. 2. What We’re Talking about  Conservancy  Overview of Web/User Analytics  Explanation of A/B Testing  Why this matters to QA personnel  Examples  How to get started http://bit.ly/2u6tJMa http://bit.ly/2urc80O
  3. 3. Who Is Conservancy?  Software Freedom Conservancy  Protects the branding, copyrights, and licenses for over 40 free and open source projects  Includes Selenium, git, Samba, PyPy, and many others  www.sfconservancy.org  Non-profit organization  The reason Jim, Manoj, and I are here
  4. 4. Overview of User Analytics  Primarily B2C: Banks, shopping, high-volume apps  Applies to all platforms, including video game consoles  Tells the ”story” of user interaction with a product
  5. 5. User Analytics Hits Sessions Users “If you are not paying for it… you are the product being sold” —Andrew Lewis (blue_beetle)
  6. 6. A/B Testing  Don’t change everything at once  Major source of regressions in analytics:  Analytics will need to change as a result of a test, but sometimes those changes are forgotten (  We used to just release changes after basic QA  “The Highball Incident” http://bit.ly/2ttj1RK
  7. 7. Real-life RMN Example ...&b21=17&… ...&b21=18&… ...&b21=0&… (wait, shouldn’t that say 18?)
  8. 8. How Is This Related?  Analytics have historically not been included in test design  Major source of regression defects  In this case, no A/B testing was done  Data loss was permanent, irrecoverable
  9. 9. Real-world Examples  Shopping cart—shipping & tax calculation  Suggesting products and content based on cookie, not login
  10. 10. Real-world Examples  RetailMeNot’s email program  Category and Merchant “affinity”
  11. 11. Classic Approach Marketing Business Intelligence Product Manager EngineeringQA (Usually the newest person on the team)
  12. 12. Why You Should Care  What if the analytics contain the wrong information?  But furthermore…  This is everywhere  It is only growing  Companies are becoming smart  (Really really smart)  QA is being left behind
  13. 13. Why You Should Care
  14. 14. Why You Should Really Care  As a tester:  There is a team of people working on this  It gets worked into features as they are developed  It is rarely called out separately in a scheduled task  It rarely receives QA outside of the Product Management and Business Intelligence people who really care about it http://bit.ly/2u6j4AW
  15. 15. Fortunately, It’s Easy  Usually one extra HTTP request, made during a navigation event  Intercept this request, then verify the data within it
  16. 16. Process  Break down into key-value pairs  Build a Map of “expected results”  Assert on expected vs actual results sequence=12 sequence=12 category=food category=food type=code type=sale
  17. 17. Test Cycle @Test Launch Browser Navigate to Position Start Proxy Perform Main Action Stop Proxy Clean-up Test
  18. 18. Execution  Send JSON to the “expectation” end-point  Includes a “timer” and an alert email  Later (asynchronously), more JSON is sent to the “result”  If there’s no match before the timer runs out, ALERT Expectation Action Result
  19. 19. Reporting  Report to a dashboard  Indicates “PASS”, “FAIL”, and “Staleness”
  20. 20. Alternatives  Anand Bagmar  https://github.com/anandbagmar/WAAT  Tool for testing Omniture Analytics  Uses Omniture’s built-in web interface to examine attributes emitted by page, rather than using a proxy
  21. 21. Conclusion  User Analytics are your CEO’s favorite subject!  Deliver real value—million-dollar decisions are made with this data  Can be implemented with just as many bugs as any other kind of software
  22. 22. Questions?  Once again, thank you so very much  Twitter: @mmerrell  mmerrell@gmail.com  I’m also on LinkedIn

×