Automated testing can be difficult, slow to implement, involve expensive and non-compatible software, and require a high level of technical expertise to use. Join Matt Joste as he presents Ryerson University's Automation Framework, put together using best-in-class open source software. The framework allows software developers, product owners, and testers without a technical background to write and run automated scripts. This modular framework addresses both functional and nonfunctional automated tests—performance, security, and accessibility—and is both agile and expandable. Matt explains the benefits and constraints of using an integrated set of open source tools such as JMeter (for performance), Robot Framework and Selenium (for functional testing), ZAP (for security), Appium (for mobile) and Pally (for accessibility)—all connected to Jenkins (for continuous testing). Matt discusses his team’s journey and key learnings in defining the framework and provides some ideas for using tests in production for proactive monitoring and alerting.

1. T3
Session
10/27/2016 10:15:00 AM
How to Build a Fully Open Source Test
Automation Framework
Presented by:
Matt Joste
Ryerson University
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888-­‐268-­‐8770 ·∙ 904-­‐278-­‐0524 - info@techwell.com - http://www.starcanada.techwell.com/

2. Matt Joste
Ryerson University
A software quality assurance and testing specialist for Ryerson University in
downtown Toronto, Matt Joste has more than fifteen years of experience in
software quality assurance including two years as a ScrumMaster. Passionate
about helping teams become as agile and self-sufficient as possible, Matt has
journeyed through the finance, banking, security, media, and now education
sectors.

3. 10/17/2016
1
Presentation: How to Build a Fully Open Source Test
Automation Framework
Presenter: Matt Joste
Date: October 27th 2016
‐ Software > 15 years
‐ Industries: Banking, Finance,
Software Security, Media
‐ Mostly QA, apart from a stint as a
Scrum Master
‐ Currently QA & Testing Specialist @
Ryerson

4. 10/17/2016
2
Software Test Automation
In software testing, test automation is the use of
special software (separate from the software being tested) to control
the execution of tests and the comparison of actual outcomes with
predicted outcomes
Kolawa, Adam; Huizinga, Dorota (2007). Automated Defect Prevention: Best Practices in Software Management. Wiley‐IEEE
Computer Society Press. p. 74. ISBN 0‐470‐04212‐5.
Why Automate?
• Half the labor expended to develop a working program is typically spent on testing
activities‐ (Boris Beizer, Software Testing Techniques,1990. + IDT Survey responses)
• Automation can increase the depth and breadth of testing. Specific benefits include:*
• Improved build verification testing(smoke testing)
• Improved regression testing
• Multiplatform compatibility and configuration testing
• Improved execution of mundane tests
• Improved focus on advanced test issues
• Testing beyond manual: e.g. security or memory leak testing
Abilit t d d f t i t tl• Ability to reproduce defects consistently
• Extending the hours of operation
*Automated Software Testing(Dustin et al)

5. 10/17/2016
3
Agile Software Development
Agile software development is a group of software development methods in
which requirements and solutions evolve through collaboration between self-
organizing, cross-functional teams. It promotes adaptive planning, evolutionary
development, early delivery, continuous improvement, and encourages rapid and
flexible response to change
"What is Agile Software Development?". Agile Alliance. 8 June 2013
Benefits of Agile Software Development
Benefits Obtained from Implementing Agile ‐VersionOne ‘State of Agile Summary 2011’

6. 10/17/2016
4
Agile Adoption
2014 Product Development Methodology Adoption Rates
http://www.actuationconsulting.com/product‐development‐methods‐popular/>
Types of Automation
a. Code‐Driven, Data‐driven, keyword‐driven
b. Unit‐testing, Test‐driven, Acceptance Tests
c. Testing Tasks
i. Mobile
1) Web, Native, Hybrid
ii. Load
iii. Functionality
iv GUI Testingiv. GUI Testing
v. Security
vi. Accessibility
vii. Test Execution

7. 10/17/2016
5
Automation Tools Overview
Open Source vs Proprietary
1. Support levels
V i d b t d d t l( )Varied, but depends on tool(o) vs
Guaranteed(for a price)
2. Cost of tools
Free vs hundreds to thousands
3. Cost of personnel
Open Source/in house could be much
more expensive
4. Vendor lock‐in
Systems and 'Vendorscript' *Lessons
learned in Software Testing(Kaner et al)
Available Automation Tools(selection)
Selenium(o), QTP(p), AutoITi.
a. GUI Testing
b. Load Testing
Jmeter(o), LoadRunner(p)i.
Appium(o), MobileAnywhere(p),i.
c. Mobile Testing
STAF/STAX(o) i.
d. Desktop App testing
OWASP ZAP(o), BurpSuite(p), WebScarabi.
e. Security Testing
FireEyes(o), Compliance Sheriff(p)i.
f. Accessibility Testing
FireEyes(o), Compliance Sheriff(p)i.
Jenkins CI(o), Quality Centre(p), Robot Framework(o),
Excel(p)
i.
g. Test Case Management, Execution and Reporting
FireEye/ACTS from NIST(o)i.
h. Test Data Generation
RegEx Coach(o), Fiddler(o), FireRobot(o)i.
i. Automation Support

8. 10/17/2016
6
Cloud‐based Services
Browserstack Sauce Labs
Browserstack Keynote DeviceAnywhereBrowserstack Keynote DeviceAnywhere
Appthwack BrowserSwarm
Automation Frameworks
• Create Structure for Automation Efforts(format, execution, reporting)
• Allow for Abstraction of Test Cases( e.g. Keyword‐driven Testing)
• Facilitate Growth of Test Repositories
• Enable Many Contributors
• Should Support Integration of Multiple Tools
• Should Support Testing without having to install Framework on Every Computer

9. 10/17/2016
7
Pros and Cons of Automation
Pros
Repeatable consistentRepeatable, consistent
Fast(once implemented)
Possible(e.g. Load tests)
Vital to realize Agile and Continuous Delivery
Less tedious
Cons
Time consuming(3‐10 times manual testing)Time consuming(3 10 times manual testing)
More skills required
Expensive tools or personnel or both
Maintenance
Slower to start
Doesn't fully replace manual testing
How We Automate @ Ryerson
• Open Source
• Small QA team
• Leverage developers
• Build toward Agililty

10. 10/17/2016
8
The Ryerson CCS Automation Framework
Robot Framework Test Example
RF Test Case Folder

11. 10/17/2016
9
h k f h !Thanks for watching!
https://www.linkedin.com/pub/matt‐joste/6/abb/b89
mjoste@ryerson.ca