Jay Luker will be presenting an introduction to Ghost Inspector, a cloud-based web UI testing service that takes some (some!) of the pain away from creating browser-based, web application tests. Think Selenium, but for projects that are short on the resources, infrastructure and/or coding expertise to confidently develop and manage a suite of fully automated, “good-enough” UI tests.
Jay is a Senior Software Engineer at Harvard DCE where he works on back-end applications and software for analytics data collection, deployment automation, and integration testing for the Extension School’s video processing and delivery system. Previously he has been an IT Specialist at the Smithsonian Astrophysics Data System, and a Software Developer at Ex Libris.
1. ui testing with ghost inspector
a capstone for your testing pyramid
jay luker | @lbjay
senior software engineer, harvard dce
2. about.me
“self-taught”
full-stack
not a testing "expert"
free, open-source
small teams, smallish orgs
lazy
likes: python, rowing, jaywalking, mondays
dislikes: tea, endless scroll, mis-implemented http status codes
3. this is not that other talk
● pytest
● pytest plugins
● ghost inspector
● pytest-ghostinspector
https://www.youtube.com/watch?v=qs1mbTg2kds
4. let's talk about...
why we do automated testing
the ideal "testing pyramid"
ghost inspector
how we're using ghost inspector at dce
5. why write automated tests?
● continuous feedback throughout the development cycle
● did this change break something?
● catch bugs earlier and faster
● help inform the writing of code and application architecture
● help specify code and application behavior
7. test scope continuum
small (unit) large (e2e)
fast
cheap
reliable
isolated
low test:value ratio
run frequently
verifies code behavior
owned by developers
created early
short feedback loop
"are we building the system right?"
👈
👈
👈
¯_(ツ)_/¯
👉
👈
¯_(ツ)_/¯
👉
👈
👈
¯_(ツ)_/¯
slow
expensive
flakey
integrated
high test:value ratio
run infrequently
verifies real world behavior
owned by anyone*
created late
long feedback loop
"are we building the right system?"
8. the testing pyramid
source: http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-more-end-to-end-tests.html
9. pyramid vs ice cream cone
source: https://watirmelon.com/2012/01/31/introducing-the-software-testing-ice-cream-cone/
11. ghost inspector
● cloud-based ui testing service
● record tests
● manually build/edit tests
● execute tests
● schedule tests
● view & monitor test results
● receive notification of test outcomes
12. who the what now?
● it's like a browser*...
● in the cloud…
● that accesses your site…
● and does stuff…
● and verifies everything worked right…
● and then lets you know if it didn't
* more specifically, a headless browser under the control of a webdriver
13. great things about ghost inspector
● free / cheap
● easy to get started
● high initial effort:value ratio
● well-written documentation
● intuitive interface
● api for running tests
● nothing to install locally
● tests allow custom variables & re-use
14. not so great things about ghost inspector
● slow, particularly on failures
● missing concurrency
● no test versioning
● documentation can be shallow
● test recorder tries hard but has limitations
● tests run remotely; firewalls / access control require hoop-jumping
● support is currently via e-mail only
15. things you will need
● a website or web application to test
● chrome (optional)
● knowledge of html and javascript
● knowledge of how to use chrome or firefox developer tools
● knowledge of how to interact with restful apis (optional)
● $$ for > free pricing tier (optional)
16. free tier details
● 100 tests / month
● 3 tests * daily schedule: boom
● all the bells & whistles
● link a (free) runscope account* and
get 500 tests / month
* https://www.runscope.com/ghostinspector
17. let's do this thing
1. create a ghost inspector account
2. install the chrome extension
3. login through the extension
4. record a test
5. edit the test steps
6. execute the test
7. view the results
8. repeat 5-7 as needed
19. test creation & editing tips
● adjust css selectors to be less brittle
● adjust css selectors to remove transient values
● chrome / firefox developer tools are your friend
● use variables for transient values
● remove unnecessary form input 'click' steps
● something not working? drop down to javascript
● start urls can contain variables!
● try changing the browser version
20. ghost inspector at dce
● dev team of ~10, including 2 full-time qa'ers
● cluster application on aws opsworks
● more ice cream cone than pyramid
● e2e testing objective is to validate production releases
21. test automation tools
● mh-opsworks: command-line cluster manager
● mh-ui-testing: command-line ui testing tool
● pytest-ghostinspector: pytest plugin for the ghost inspector api
https://github.com/harvard-dce/mh-opsworks
https://github.com/harvard-dce/mh-ui-testing
https://github.com/harvard-dce/pytest-ghostinspector
I'd like to touch briefly on the value of automated tests at all levels
Then we'll talk about the concept of the "testing pyramid" which is a pattern / way of thinking about what kinds of automated tests make the most sense for a web application, how many of each type, etc.
Next I'll give a demo of ghost inspector, show you what it's capable of and how, in my opinion, it can give you some immediate value for relatively low up-front effort.
Then finally a bit about how dce is managing and executing our ghost inspector tests
tests that run automatically at every step, from the developer's IDE to the deployment process
just to quickly review and nail down some of the slippery testing taxonomy
unit tests are code-level tests. run primarily by developers and continuous integration services. they attempt to test the smallest possible unit of functionality. ideally they are run in complete isolation from other concerns, such as other, non-trivial parts of the code, network, databases, the file system, etc. other characteristics of unit tests are that they are (or should be) very fast, because their scope is limited they are also less fragile than higher level tests and help to isolate bugs more quickly
Example: does my method for calculating a unique, indexable string based on an metadata record return a correct result for all variety of inputs
integration tests are directed at at the next higher level of functionality. do the smaller pieces align correctly when stitched together? Here you are still frequently trying to isolate your concerns, e.g., through mocking, but maybe the setup phase of the test spins up an external service that the System Under Test (SUT) interacts with.
Example: does my session manager class correctly utilize Redis's key expiration controls
end-to-end tests operate at a holistic level, testing the entire system from the perspective of a user or customer. most common example is tests that exercise some kind of GUI, like using Selenium to drive a browser which interacts with a web site/application.
Example: can a user log in and change their display name; can a user add and remove items from a shopping cart
testing pyramid: illustrates a generally accepted recommendation for the distribution of your testing efforts.
the bottom, the foundation, are your unit tests
then integration tests
finally e2e or UI tests
the width of the pyramid represents the volume of tests
as you go up, the tests
here's another interesting take on do vs. don't by testing expert, Alister Scott
Scott points out that what is missing from most testing pyramid representations is the practice of manual, exploratory tests. This is apparently a whole thing, and is not simply a fancy name for ad hoc "clicking around".
Scott also identifies the anti-pattern that occurs when your distribution of test types gets inverted. This can be detrimental to your project as it means your testing efforts are not playing to the strengths and weaknesses of the different types of tests.
Manual testing, while valuable, has the same downsides as large, end-to-end automated tests, but with the additional challenges that it can be even harder to orchestrate, capture and reproduce results, etc.
It's that end-to-end / ui type that I believe ghost inspector can help with
bells & whistles = video capture, screenshot compare, API, integrations