On March 18th, 2015 Excella's Dan Davis presented Automated Acceptance Testing from Scratch to the Tech Talk DC Meetup group.

1. Automated Acceptance Tests
from Scratch
Daniel Davis

2. • Daniel Davis
• Software Developer for 8 years
• Fun Fact:
– I recently went to see “Laser Cat” in DC
Who Am I?

3. 3

4. Really though, who are you?
• Came from Java world
• Python developer for 2 years
• DevOps
– Lots of work with automation and quality
• First introduced to AAT about 2 years ago
– Now I use it at my client!

5. 5
Deploy with CI
Website

6. Now that it’s up…
how do I test that it works?
6

7. • Go to the website, click around, make sure
things look right…
• Problem: Scaling
– What happens as the site continues to grow?
– What happens as we deploy more frequently?
• We need a way to test the functionality of
the site
Simple!

8. Automated Acceptance
Testing Can Help!!!

9. • Tests written at the functional/business
level
• Can be read by non-technical people
• Doesn’t have to be automated to provide
value!!!
What is Acceptance Testing

10. Unit – 70%
Integration
– 20%
AAT
10%
Where does AAT fit in?

11. Consider it this way…
11
Unit Testing Integration Testing

12. 12

13. Automated Acceptance Testing
• Great for covering a lot of ground
• Not so good for very specific things

14. Ok, so how do I get
started???

15. Popular Frameworks

16. • Java/JVM world: Cucumber-JVM/JBehave
• Ruby: Cucumber-Ruby
• Ruby on Rails: Cucmber-Rails
• Node.js/Javascript: Cucumber-Javascript
– AngularJS: Protractor  not the same
• C#/.NET: SpecFlow
• Python: Behave
Cucumber
16

17. • Cucumber’s syntax and style is virtually
the same across all the platforms!
• Tonight’s examples will be in Python
Good News Everyone…
17

18. • One example, from start to finish
– This is how we solve problems
• Learning Objectives:
– Write an acceptance test
– Automate that test using Selenium
– Incorporate that test into your Continuous
Integration
• Stay high level
– Not enough time to go deep
Let’s get started!
18

19. Example:
theCatAPI.com
19

21. • Objective: Want to test Upvote/Downvote
functionality
• Follow Along:
– https://github.com/Ooblioob/aat_demo/
theCatAPI.com

22. Step #1: Write an
Acceptance Test
22

23. • Install your cucumber tool of choice
• Directory structure:
project-root/
|-- features/
|-- steps/ (we’ll talk about this soon)
|-- pages/ (we’ll talk about this soon)
Project Setup
23

24. 3 Layers to Cucumber
24
Gherkin (Feature)
“Glue” (Step)
Selenium Code

25. • A Human-readable syntax describing the
test
• Consists of a main “feature” and scenarios
describing the feature
Writing Gherkin

26. Feature: Title (one line describing the story/feature)
As a [role]
I want [feature]
So that [benefit]
Scenario: Title1 (for Behaviour 1)
Given [context or setup]
And [some more context]...
When [event occurs]
Then [expected outcome]
And [another outcome]...
Scenario: Title2 (for Behaviour 2)
…
Writing Gherkin
26

27. TheCatAPI.com
27
https://github.com/Ooblioob/aat_demo/blob/master/browser_tests/
features/upvote.feature

28. • Value even when not automated
– Clear, concise way to manually test
– Developer friendly
• Not extremely specific…
– What does it mean “a new cat image should
be loaded…”?
– Declarative vs Imperative
– Don’t get bogged down in
precision/verboseness
Observations

29. Step #2: Automating your
Acceptance Tests
29

30. 3 Layers to Cucumber
30
Gherkin (Feature)
“Glue” (Step)
Selenium Code

31. • Create an “environment.py” file in the
features directory
– Defines the test runner setup
• (optional) Create an environment.cfg file in
the features directory
– Holds editable configurations for your setup
• Create a “steps.py” file in the steps directory
– Contains the implementation of your steps
Project Setup
31

32. Environment.py
32
Full Example:
https://github.com/Ooblioob/aat_demo/blob/master/browser_tests/fe
atures/environment.py

33. • Setting up Selenium
– Select a driver, like Chromedriver, PhantomJS
or SauceLabs
• Setting up a logger
• Initializing the context object
• Set defaults
What goes in Environment.py?
33

34. • Parses each line of Gherkin
• Decide how to implement
Glue code (steps)
home.feature
steps/steps.py

35. Writing your implementation
Example:
https://github.com/Ooblioob/aat_demo/blob/master/browser_tests/featu
res/steps/steps_home.py

36. • Making references to context
– Way of passing information between steps
• What is “Home” in the context?
– Class representing the homepage
– Part of “Page-object model”
Observations
36

37. That pretty much it…
When do we talk about Selenium?
37

38. Not the best practice…
38
Glue (step)
One Giant Selenium Class

39. Page-Object model
39
Glue (step)
Home
page
Docs
page
Reference
page
Utility
functions
Base Page

40. So what goes in a
“Page” object?
40

41. “Hate It” Button
“Like It” Button
Title
Main
Image

42. • For each page element:
– Selectors: Id, CSS, XPATH, etc
– get() method
– commonly used attributes
• Define navigation to the page
– go()
• Define other actions / helpers
– Storing info, waits, etc.
Writing your pages

43. Writing Selenium Code
43

44. One more small
detail…
44

45. • Need to setup the web driver
– Let’s use ChromeDriver
• https://sites.google.com/a/chromium.org/chromedri
ver/getting-started
Environment.py revisited
45

46. Demo Time!
(Making it all work)
46

47. • Use page-object model to organize code
• Write simple, reusable methods to
implement pages
• Installed Chromedriver
Recap

48. Step #3: Put it into Continuous
Integration!
48

49. • Now that we have tests
– Validate site is up and running
– Validate deployments
– Ensure new code doesn’t break existing
functionality (regression)
Why Continuous Integration?

50. Challenges

51. • PhantomJS
– Headless browser
– Based on Webkit
– Fast
• Works with Selenium
Solving the problem
51

52. • Black Box
– Can’t see problems when they happen
• Can’t handle certain things
– e.g. alerts
• Buggy, but getting better
Problems with PhantomJS
52
At this time, I don’t recommend it…

53. • “Cloudified” browsers
Sauce Labs
53

54. • Test different browser / os configurations
– Don’t need to own that hardware
• Integration with many CI tools
– Jenkins, Travis, CircleCI, Bamboo, Teamcity…
• Video recording/playback
– No longer a black box!!!
Advantages
54

55. Yes, but what about
my Dev severs???
55

56. • Sauce Connect: test your Dev server too!
Advantages
56

57. So how do I set it up?
57

58. • Sign Up for an account:
– https://saucelabs.com/signup/trial
• Receive an API key
Sauce Labs Setup

59. • Modify your environment.py file to use
Sauce instead of Chrome
– Provide username and api key
– Select a browser and OS to use
Sauce Labs Setup
59

60. Sauce Labs Setup
60
https://github.com/Ooblioob/aat_demo/blob/master/browser_tests/f
eatures/environment.py

61. Sauce Labs Setup
61
https://github.com/Ooblioob/aat_demo/blob/master/browser_tests/f
eatures/environment.py

62. Demo Time!
62

63. One last topic

64. • Can only test what’s available through the
browser
• Selenium can be painful at times
– May not wait for things to load
– Debugging can be hard
Limitations of AAT

65. • Data
– Not a great choice for data-heavy applications
– Not impossible, but trickier:
• Launch a separate test server?
• Create a “testing mode” where transactions are
rolled back
– Might be better covered by Integration Testing
Limitations of AAT
65

66. • https://github.com/Ooblioob/aat_demo
• https://cukes.info/
• http://jenisys.github.io/behave.example/
• https://docs.saucelabs.com/reference/sauc
e-connect/
Resources
66

67. Questions?
67