Brian Bayer QA or the Highway 2016 talk

1. test automation strategy
Developing a

2. why are we here?

3. regression testing
the greatest cost of development

4. regression testing
you have already paid for this at least once

5. why regression test?

6. why regression test?
quality ensures what was working, still works

7. why regression test?
quality
risk
money
lost revenue, market, contractual
ensures what was working, still works
deliverables, release, schedule

8. why regression test?
a project without
thorough and continuous regression testing
is like a road trip
where you do not look out the window
until the end of the trip

9. why automate? (vs. manual)

10. why automate? (vs. manual)
TimeMoney

11. why automate? (vs. manual)
Money

12. why automate? (vs. manual)
Time

13. automation strategy
what to consider...
tests are living specifications of the system

14. automation strategy
what to consider...
ARE THE TESTS GOING TO BE USED?

15. automation strategy
what to consider...
business needs
return on investment
organizational structure and culture

16. automation strategy
business needs...
determine areas of focus of the automation

17. automation strategy
return on investment...
limits and prioritizes the areas of focus of the
automation
determines at which level to automate a test
determines the automation framework

18. determines the success or failure
determines the speed, ability & openness to change
determines the communication
determines the ability to learn
automation strategy
organization and culture...

19. automation strategy
organization and culture...
determines who uses and maintains the tests
determines the automation framework too

20. automation strategy
what else to consider...

21. automation strategy
what else to consider...
development process and lifecycle
maintenance of the tests
technology stacks

22. automation strategy
development process and lifecycle...
legacy (system without tests) to greenfields

23. automation strategy
development process and lifecycle...
system under test development process
methodology: waterfall agile xp ad hoc
is automation development part of the same process?

24. automation strategy
development process and lifecycle...
is the automator part of the development team?
how much access to the system developers?

25. automation strategy
maintenance of the tests...
who and how much can they maintain?
how are the system developers involved?

26. automation strategy
maintenance of the tests...
how timely can/must tests be maintained
what reporting needed to maintain tests

27. automation strategy
finally, technology stacks...
languages and technology already used
unit and integration testing
source code control, continuous integration
devops or operations

28. where to automate?

30. who is going to use the tests?

31. developers, maybe...
depends

32. where to automate?
https://msdn.microsoft.com
“Its essential point is that you should
have many more low-level unit tests than
high level end-to-end tests running
through a GUI.” Martin Fowler

33. where to automate?
UNIT LEVEL
INTEGRATION LEVEL
GUI INTEGRATION LEVEL
XPtest driven development
ACCEPTANCE LEVEL
BDD

34. ACCEPTANCE LEVEL
BDD
where to automate?
acceptance tests
only if the business is engaged and
going to use them
ACCEPTANCE LEVEL
BDD

35. where to automate?
what to consider...
DRY

36. look for low-hanging fruit
risk-based testing
what computers do well, humans not so much
fragility of test
where to automate?
what to consider...

37. which automation framework to use?

38. which automation framework to use?
Gherkin/Cucumber
Robot Framework
Selenium Webdriver
Spock/Geb
JUnit
Jasmine
TestNG

39. which automation framework to use?
what to consider...
ORGANIZATIONAL STRUCTURE AND CULTURE

40. which automation framework to use?
what to consider...
who is reading the tests?
who is writing and maintaining the tests?
balance of right tool with existing technology,
knowledge, and ability to learn
ORGANIZATIONAL STRUCTURE AND CULTURE

41. which automation framework to use?
what else to consider...
natural language for high level specification
goal is to develop tests NOT framework
and...

42. grids...

43. integration...

45. reporting...

46. teh sexy

47. gherkin/cucumber/ruby

48. but JRuby (JVM), Java, Groovy, JavaScript,
.NET (using SpecFlow)
libraries (Selenium, Watir)
Ruby has the richest ecosystem with gems
gherkin/cucumber
generally ruby based...

49. Feature: Cash Withdrawal
Scenario: Successful withdrawal from an account in credit
Given my account has been credited with $100
When I withdraw $20
Then $20 should be dispensed
And the balance of my account should be $80
Given /^my account has been credited with (#{CAPTURE_CASH_AMOUNT})$/
do |amount|
my_account.credit(amount)
end

51. also runs on Jython (JVM) & IronPython (.NET)
libraries (Selenium, Watir, SOAP, etc.)
keyword based
supports “given when then”
supports tabular/data driven
robot framework
python based...

52. *** Settings ***
Library Selenium Library
Suite Setup Start Selenium Server
Suite Teardown Stop Selenium Server
Test Setup Setup Test
Test Teardown Close All Browsers

53. *** Testcases ***
Login Should Succeed When the Correct
Username and Password are Entered
Enter Username AUser
Enter Password TestPass
Click the Login Button
Login is Successful
*** Keywords ***
Setup Test
Open Browser http://zdiles.chaosnet.org/ googlechrome
Maximize Browser Window
Enter Username [Arguments] ${username}
Input Text uname ${username}
Login Is Successful
Page Should Contain Welcome

54. *** Testcases ***
Login Should Succeed When the Correct Username and
Password are Entered
Given I Enter Username AUser
And I Enter Password TestPass
When I Click the Login Button
Then Login is Successful
given when then

55. spock/geb/groovy

57. developing in java or grails?

58. groovy-based browser automation tool
built on Selenium WebDriver
works with JUnit, TestNG, Spock
page models and jQuery selector syntax
spock/geb/groovy
geb...

59. groovy-based test and specification framework
supports “given when then”
tabular/data driven
spock/geb/groovy
spock...

60. import geb.spock.GebSpec
class GebHomepageSpec extends GebSpec {
def "can access The Book of Geb via homepage"() {
given:”One is on the Geb Home Page”
to GebHomePage
when: “I click the jQuery Link”
highlights.jQueryLikeApi.click()
then: “I should be at the jQuery Section”
highlights.jQueryLikeApi.isSelected
}
}

61. import geb.Page
class GebHomePage extends Page {
static url = "http://gebish.org"
static at = { title == "Geb - Very Groovy Browser Automation" }
static content = {
highlights { $("#sidebar .sidemenu").module(HighlightsModule) }
sectionTitles { $("#main h1")*.text() }
}
}

62. FRAMEWORKS

63. other options...

64. raw Selenium (Java)
excel spreadsheet data driven scripts
script capture record/playback
flat files in source control
other options...

65. which automation framework to use?
the simplest thing that could possibly work

66. tests
are specifications of the system...

67. how does it fit?

68. how does it fit...
design your approach into the organization

69. how does it fit...
find the gaps and tight squeezes

70. the specifics...

71. the mechanics...

72. so what’s left?

73. documentation of the strategy...

74. execution of the strategy...

75. if project management...

76. estimation...

77. resources...