Transitioning to agile development isn't any easier for the test team than it is for the development team. New features and fixes appear daily, priorities change fluidly as customer feedback shapes the backlog, and design documents are frequently sparse as developers use iterations to zero in on design solutions. On top of all this, with agile testing there is rarely time to build the giant spreadsheets of test cases that management loves to use for tracking testing progress. So, how do we handle the constant flood of change, balance exploratory testing and automated testing—and communicate progress to senior management? Matt Attaway shares his experiences developing a successful lightweight agile testing process that helped his team move from testing two waterfall projects at once to six agile projects simultaneously. Matt discusses lightweight test plan development, the use of BDD tools such as Cucumber, and methods for measuring testing progress.

1.
nt Session
Presented by:
Matthew
Brought to you by:
340 Corporate Way, Suite Orange Park, FL 32073
888‐2
T11
Concurre
4/8/2014
2:00 PM
“Balancing Exploratory and
Automated Testing in Agile”
Attaway
Perforce Software
300,
68‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Matthew Attaway
Perforce Software
A jack of all trades and master of some, Matt Attaway has worked as a tester,
developer, researcher, designer, manager, DevOps engineer, and elephant trainer. He
currently manages the open source development group at Perforce Software, but
prior to that he led a test team working on the highest profile projects at Perforce.
When not tinkering with the latest software development pipeline tools, Matt enjoys
experimenting with cocktails and playing games of all sorts—preferably
simultaneously.

3. 2/18/2014
1
Balancing Exploratory and
Automated Testing in Agile
Matt Attaway | @mataway

4. 2/18/2014
2
• Projects varied in size from 3-20 people
• Multiple languages, multiple stacks
My World
• GUIs, CLI tools, servers, and APIs
• Remote and co-located teams
• Scrum, Kanban, Scrumban, Waterscrumfallban
• Managed ~10 testers on 6 simultaneous projects
• Almost all "bet the company" projects
• Automation or bust!
My Constraints
• Exploratory testing a must
• Volatile projects; few design docs
• Extremely high bar for quality
• Executive demand for measurability and metrics
• Do more with less

5. 2/18/2014
3
Must handle the constant flood of change
My Process Requirements
• Must handle the constant flood of change
• Must integrate automated testing without sacficing
exploratory testing
• Must communicate progress both to the product
team and the executive teamteam and the executive team
Must not die

6. 2/18/2014
4
Ruthlessly optimize
Exploratory testing
T t l
Automated testing
Test plans

7. 2/18/2014
5
Exploratory testing
T t l
Automated testing
Test plans
Helm th on Moltke the Elder
“No plan survives contact with the enemy.”
–Helmuth von Moltke the Elder

8. 2/18/2014
6
Needed test plans per epic story feature and fix• Needed test plans per epic, story, feature and fix
• Easy report generation and APIs
• Easy access for testers
Cheap both in time and in money• Cheap, both in time and in money
Issue tracker

9. 2/18/2014
7
Focused around epic story feature and fix
Strengths
• Focused around epic, story, feature and fix
• Built-in report generation and APIs
• Testers already lived in it
Already installed and paid for!• Already installed and paid for!
Test plans highly visible to the developers
Bonuses
• Test plans highly visible to the developers
• Naturally supports multiple editors
• Extremely easy to extend
• Notifications of updates via email and RSS with diffs• Notifications of updates via email and RSS, with diffs

10. 2/18/2014
8
• Added a "Testing Notes" field to issues
Testers added test cases as they did their
Process
• Testers added test cases as they did their
exploratory testing
• + used to flag completed test, - to flag pending test
• Anyone could add test cases to an issue
• Tweet sized test cases; should be clear for
knowledgable tester
Example
- Attempt to create user with HTTP special characters
- Attempt to create user with XML special characters
+Attempt to create user that already exists
Example

11. 2/18/2014
9
• Devs handed off work by merging to mainline
Process & Measuring
• Issue not closed until all test cases complete
• Outstanding test work = issues in 'fixed' state
• Test work displayed on burndown charts; easy to
spot when test was having a hard time keeping up

12. 2/18/2014
10
Automated testing
• Numerous automated test technologies and languages
CLI t t t l b t t t l d GUI t t t l• CLI test tools, web test tools, and GUI test tools
• Poorly written code; copy/paste gone wild
• Each product had one (or more!) test frameworks
• Too slow to add tests• Too slow to add tests
• Tests were dependent on each other

13. 2/18/2014
11

14. 2/18/2014
12
Given I am speaking at StarCanada
When I start my talk
Then my heart rate will go up dramatically
• Forced code modularization
Why Cucumber?
• Test cases are legible and shareable
• Can drive any tool; can use most languages
• Tests are easy to add; frequently require no "real" code
• Works out of the box; huge development community
W it t t ti• Write tests anytime
• Everyone can automate
• Cucumber is, dare I say, fun

15. 2/18/2014
13
• Hour meeting at the beginning of the sprint; more as
Bugstorming
needed during sprint
• Wrote Gherkin as a team coming up with different
attacks based on slotted stories
• Helped to share system knowledge and automated
testing knowledge
• Track your hit rate; failed automated test is a bug
T k h t t t d h t d 't ( d h !)
Automated testing advice
• Track what you automate and what you don't (and why!)
• Aim to be able to add a test case in 15 minutes
• Always Reset State
• Automate all the things• Automate all the things
• Plan for parallelized tests from day one

16. 2/18/2014
14
Ruthlessly optimize
Avoid documents that do nothing
(or make your documents do more)

17. 2/18/2014
15
Embrace your issue tracker
Detach writing automated tests from
implementing automated tests

18. 2/18/2014
16
Regression tests == automated tests
The Cucumber Book by Aslak Hellesøy
Books
• The Cucumber Book by Aslak Hellesøy
• Cucumber & Cheese by Jeff Morgan
• Continuous Delivery by Jez Humble & David Farley
Exploratory Software Testing by James Whittaker• Exploratory Software Testing by James Whittaker