End-to-end browser testing with a manageable number of annoyances

1. End-to-end browser testing
with a manageable number of annoyances
— Steven Noble, 2019

2. About me
• Creating software since Dick
Smith added a Basic cartridge
and cassette data storage to
the Wizard
• Working in the digital
economy since 1994:
journalism, PR, marketing,
consulting, research, etc
• React/Node software
developer with GorillaStack:
we’re like IFTTT for your
enterprise cloud infrastructure

3. End-to-end browser testing:
the theory
• Confirm that your services and units work
together as required
• Test your app the way your users test it —
through the user interface
• Spend less time on manual testing

4. End-to-end browser testing:
the reality
…that’s all true, but…
• It’s really slow
• It’s hard to debug
• It can be flakey
…so here’s how to get those benefits with a
manageable number of annoyances

5. But first, your stack
Headless
web
browser
Chrome
only
Chrome
Firefox
etc
Browser
automation
driver
Chromedriver
Chromedriver
Marionette (Firefox)
etc
Browser
automation
server
DevTools
protocol
Selenium
Browser
testing
framework
Google
Puppeteer
Webdriver.io
Nightwatch
TestCafe
etc

6. It looks like this
browser.url('/login');
$('#login-form').waitForVisible();
$('#username').setValue(WDIO_USERNAME);
$('#password').setValue(WDIO_PASSWORD);
$('#login').click();
$('#dashboard').waitForVisible();
expect(browser.getUrl()).to.equal('/dashboard');
expect($('h1')).getText().to.equal('Dashboard');
expect($('#header').getText())
.to.include(`Welcome ${WDIO_USERNAME}`);

7. TIP: Employ software developers,
not automation testers
So you can quickly
tell whether a failure
is a bug in the app
or a bug in the test
So developers will
write or maintain tests
whenever they write
or maintain features
aka, be GorillaStack, not the ABC

8. TIP: Employ software developers,
not automation testers
• So you can define semantic selectors:
• Testing your own code:
• button#reset-password
• Testing someone else's code:
• //div[@class="col-md-12"]/
form[@id="login"]/button[2]

9. TIP: Employ software developers,
not automation testers
So you can change the behaviour of the app being
tested:
• Disable Google Analytics, Rollbar, Intercom etc
• Use test keys for Google Recaptcha, Stripe, etc
• Execute next steps immediately instead of
sending them to a message queue
• Load/unload database fixtures

10. TIP: Lock down Google Chrome
If you are just using the Google Chrome binary that you
use for everyday browsing, then you are liable to find:
• Test behaviour changes randomly after automatic
Chrome version updates
• Test behaviour differs on different machines that are
running different versions of Chrome
Instead, use a Docker container or similar that lets you
lock your tests to a specific version of Chrome

11. TIP: If you Dockerise Selenium, don’t
treat the container as a black box
• If you get weird Selenium errors in your tests, start by
dropping down a level:
• Visit http://localhost:4444/wd/hub/static/resource/
hub.html
• Send commands to Selenium directly via its HTTP
API
• You’ll probably see errors related to a known
issues with the official SeleniumHQ containers,
with possible solutions in GitHub issues

12. TIP: If you Dockerise Selenium, don’t
treat the container as a black box
• If not, drop down a second level:
• docker exec -it <constainerID>
/bin/bash
• Send commands directly to Selenium via its
CLI
• You’ll probably see errors related to a known
issues with the official SeleniumHQ containers,
with possible solutions in GitHub issues

13. TIP: Create forgiving pipelines
in your CI
For example, in Buildkite, we:
• Run the WDIO step after all other steps, as they all
provide useful feedback more quickly
• Make the WDIO step mandatory when merging a
feature branch in develop
• Make the WDIO step optional in all other scenarios
• Make good use of the retry attributes in
pipeline.yml