Test Automation Strategies for the Agile World

TR
PM Tutorial
10/14/2014 1:00:00 PM
"Test Automation Strategies for the
Agile World"
Presented by:
Bob Galen
RGalen Counsulting Group, LLC
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Bob Galen
Velocity Partners
An agile methodologist, practitioner, and coach based in Cary, NC, Bob Galen helps guide
companies in their adoption of Scrum and other agile methodologies and practices. Bob is a
principal agile evangelist at Velocity Partners, a leading agile nearshore development partner;
president of RGCG; and frequent speaker on software development, project management,
software testing, and team leadership at conferences and professional groups. He is a Certified
Scrum Coach, Certified Scrum Product Owner, and an active member of the Agile and Scrum
Alliances. In 2013 Bob published Scrum Product Ownership–Balancing Value from the Inside
Out. Reach him at bob@rgalen.com.

1
Test Automation Strategies for the
Agile World
Bob Galen
President & Principal Consultant
RGCG, LLC
bob@rgalen.com
Copyright © 2014 RGCG, LLC 2
Introduction
Bob Galen
Independent Agile Coach (CSC) at RGCG, LLC
Principle Agile Evangelist at Velocity Partners
Somewhere ‘north’ of 30 years overall experience ☺
Wide variety of technical stacks and business domains
Developer first, then Project Management / Leadership, then
Testing
Senior/Executive software development leadership for 20 years
Practicing formal agility since 2000
XP, Lean, Scrum, and Kanban experience
From Cary, North Carolina
Connect w/ me via LinkedIn and Twitter @bobgalen
Bias Disclaimer:
Agile is THE BEST Methodology
for Software Development
However, NOT a Silver Bullet!

2
Outline
Traditional Automation – Business Case & ROI
3-Pillars
Agile Test Automation Pyramid
Agile Automation – Business Case & ROI
Implementation Strategy
Communication
Wrap-up

3
Let’s start with…
Traditional Automation Strategy
What are your current strategies towards:
Test Automation
Frameworks
Tooling
Maintenance
ROI
Team structure
Get together in “pairs” and chat about this for 20
minutes. List your approaches and tactics.
Then leverage SWOT analysis to capture your current
position. Then we’ll gather your results
Let’s start with…
Traditional Automation Strategy
SWOT – Often used to analyze strategic direction or
positions. List the “position”, then identify aspects of:
Strengths
Weaknesses
Opportunities
Threats
Often in Quadrants
Force Field Analysis – again, list a position
Forces (FOR)
Forces (AGAINST) 

4
Traditional Automation
Business Case & ROI
It’s simple really…
Manual testing is:
Slow
Labor intensive
Not intellectually stimulating
Error prone
Iterative
Did I say slow?
And we’re testing software right?
So we ought to be able to automate the testing.

5
It’s simple really…
Automated tests are:
Fast
Low/no errors
Free after initial development
Run continuously
Did I say fast?
Did I say ‘free’?
And we’re testing software right?
So we ought to be able to automate ALL of the testing.
10Copyright © 2014 RGCG, LLC 10
Capture size of the team
Cost per tester – usually hourly
# of Test cases to be run
Time to run them
Cyclical nature of the testing cycles (regression, integration,
system, etc.)
Configurations
Number of tests run per release
Number of releases per year
Do the math to come up with iterative testing run-time
investments and costs
Test Automation
ROI Sample Points – Manual Testing

6
Then contrast that against the cost for automation:
Time / cost to establish automation infrastructure
Time / cost to establish automated tests
Time / cost for ongoing maintenance and results analysis
Cost for automation tooling – depends on ‘type’, ex: Keyword
Driven
Cost per automation developer vs. manual tester
Automation replacement target for manual test cases
Do the simple, discrete math to come up with cost & time
to replace manual testing run-time and ROI savings
Test Automation
ROI Sample Points – Automation Development
Run more tests, faster and with fewer people (lower
costs)
VALUE being driven by: Running tests!
http://www.aspiresys.com/testautomationroi/index.php?action=__
reset
Did I say with fewer people (testers)?
We can hire more developers with the cost savings! And get
MORE done
We can even reduce the size of the bathrooms ☺
Traditional
Business Case

7
http://www.aspiresys.com/testautomationroi/index.php#ROI
http://www.aspiresys.com/testautomationroi/index.php#ROI

8
But often there is underestimation of risks…
So the ROI was always a ‘fuzzy’ estimate
Underestimating the nature of a software project
Architecture & design, complexity, risks, unknowns, ambiguity,
etc.
Underestimating parallel software projects (product +
automation) complexity
Integration, dependencies, coordination
Underestimating ongoing support & maintenance
Repairs & updates
Designing and adding new tests
Impact of new product technologies
Underestimating skill levels for automation development
It’s also very dangerous!
Will Robinson…
It’s commoditizing your testing activity.
Could you imagine –
Measuring developer value by LOC and doing ROI?
Measuring architectural integrity by complexity analysis and doing
ROI?
Measuring product owner value by number of backlog elements and
doing ROI?
Based on some sort of automation introduction? Of
course not. It’s insane.
Then why do we do it with tests and testers?

9
We’re all using them, so they must be good and the
approaches must be ‘correct’ right?
However, there are challenges—
Training
Size
UI-centric
COST
Can’t test everything (Back-end database code)
One-size fits all strategy
Limited developer / whole team usage
Traditional Test Automation Tools
Just a quick stop…
Typical Deployment
Level of effort
Initial design
& Architecture
Initial Test
Case Automation
Technology Disruption
Ongoing Maintenance
MORE
Typical Time & Budget
Investment
LESS
Level of Effort -
Investment

10
Now it may sound…
Like I’m opposed to test automation.
Truly, I’m not. I think automation
rocks! Or like I’m opposed to gaining
a ‘return’, absolutely not.
Automation is clearly a foundation
element for increased quality and going
faster – releasing more often
But, this traditional model or
approach of focusing on ROI is
sort of insane
3-Pillars of Agile
Quality & Testing

11
3-Pillars
Genesis
I’ve learned that “Balance” is important
A sad tale of:
Thousands of ATDD testing; Gherkin run amok
All of them are working; continuously testing; increasing
“coverage’ and life is Good!
BUT
These same teams couldn’t write a cohesive User Story to save
their life
So, where were the Acceptance Tests coming from?
3-Pillars of Agile Quality
Copyright © 2014 RGCG, LLC
22
Development & Test
Automation
• Pyramid-based Strategy:
(Unit + Cucumber +
Selenium)
• Continuous Integration
• Attack technical
infrastructure in the Backlog
• Visual Feedback –
Dashboards
• Actively practice ATDD and
BDD
Software Testing
• Risk-based testing:
Functional & Non-Functional
• Test planning @ Release &
Sprint levels
• Exploratory Testing
• Standards – checklists,
templates, repositories
• Balance across manual,
exploratory & automation
Cross-Functional Team
Practices
• Team-based Pairing
• Stop-the-Line Mindset
• Code Reviews & Standards
• Active Done-Ness
• Aggressive Refactoring of
Technical Debt
• User Stories, “3 Amigo”
based Conversations
• Whole Team Ownership of “Quality”
• Knowing the Right Thing to Build; And Building it Right
• Healthy – Agile Centric Metrics
• Steering via: Center of Excellence or Community of Practice
• Strategic balance across 3 Pillars; Assessment, Recalibration, and Continuous Improvement

12
Foundation of the 3-Pillars
23
• Whole Team Ownership of
“Quality”
• Knowing the “Right” thing to
Build AND Building it “Right”
• Healthy – Agile Centric Metrics
• Steering Required – CoE or CoP
• Strategic balance across 3
Pillars; Assessment,
Recalibration, and Continuous
Improvement
• Whole team view includes building it right,
everyone tests, everyone demo’s, etc.
• Focus on features/stories, confirmation,
conversation, and getting them staged
properly OVER testing
• 4-tier metrics: Quality, Value, Prediction, Team
• Agile strategies need light-handed “steering”;
establish a CoE (heavier weight) or a CoP
(lightweight)
• Consider finding an assessment framework
and then tying it to your strategy
measurement, recalibration, and continuous
improvement.
• Make the foundation visible thru information
radiators and metrics
24
Development &
Test Automation
• Pyramid-based
Strategy: (Unit +
Cucumber + Selenium)
• Continuous Integration
• Attack technical
infrastructure in the
Backlog
• Visual Feedback –
Dashboards
• Actively practice ATDD
and BDD
A central part of agile adoption is focusing on CI, 3-
tiered Automation development, and Dashboards to
begin incrementally building coverage for faster
feedback on changes.
100% automation is NOT the Goal!
In the interim, Hardening or Stabilization Sprints and
having a risk-based Release Train concept help
It’s important that Test or QA not ‘own’ the tooling or
all of the automation efforts. The strategy can come
from QA, but the tactical automation development is
best left to the team.
Mature teams invest in Automation, Tooling, and
Technical Debt reduction as part of Done-ness and
continually add it to their backlogs

13
25
Software Testing
• Risk-based testing:
Functional & Non-
Functional
• Test planning @
Release & Sprint levels
• Exploratory Testing
• Standards – checklists,
templates, repositories
• Balance across
manual, exploratory &
automation
Exploratory Testing (SBET with pairing) can be an
incredibly effective way to establish a whole-team,
collaborative view towards quality and testing. It also
emerges new tests.
Leverage ‘plans’ as a whole-team collaboration-
conversation mechanism; at Sprint and Release
levels.
Do not measure testing or tester progress; instead,
measure throughput, output, sprint outcomes, and
done-ness escapes at a team level.
You need a balanced test team; not everyone needs
to be able to program. But everyone needs to be
passionately skilled testers with curiosity.
Agile testing is a Risk-Based play in every Sprint and
across a release sequence.
26
Cross-Functional
Team Practices
• Team-based Pairing
• Stop-the-Line Mindset
• Code Reviews &
Standards
• Active Done-Ness
• Aggressive Refactoring
of Technical Debt
• User Stories – 3 Amigo
based Conversations
One of the hardest areas to get ‘right’ culturally. It
needs leadership alignment from Quality/Testing to
Product to Development and a consistent voice of
whole-team approaches.
This is where LEAN Thinking lives, where whole-
team collaboration happens, where professionalism
and craftsmanship are held dear.
I like the view of testers becoming the VOC,
champions of quality, and consistent questioners of
what is being build. Are we solving the right
problems as simply as possible. Notions of Minimal
Viable Product / Feature help with focus.
And yes Virginia, there ARE standards, templates,
and a focus on x-team consistency!

14
Agile Test Automation
Pyramid
Pyramid
Coined by Mike Cohn, ~ 2005
Establishes the validity of turning Traditional Automation
– upside down
Invests:
Most effort (%) in Unit Tests
Moderate effort (%) in Middle-tier tests (business logic, API,
component, feature)
Least effort (%) in UI-centric, top down tests
Granularity of the tests is part of the strategy

15
Pyramid
Often:
Tied to Definition-of-Done from a maintenance
perspective
You break it, you fix it
Percentages vary; intent does not
Whole-team ownership of automation
Although it skews at either end of the pyramid
Run as much as possible
Check-in, Overnight, Cyclically
Prime Directive = Real-time Feedback
Pyramid
+80%, UI Centric Automation
0-10%, Service or Middle Tier
Automation
0-10%, Unit level Automation
0-10%, UI Centric Automation
20-40%, Service or Middle Tier
Automation
50-60%, Unit level Automation
Traditional
Agile

16
Agile Test Automation Pyramid
Mike Cohn; Lisa Crispin & Janet Gregory
http://behaviordrivendevelopment.wikispaces.com/Testing
Brainstorm…
Agile, Multi-tiered Automation
Get together in small groups of 4-6 to discuss
Take a few minutes and think about your current
automation approaches:
Tooling, approaches & strategies, strengths, weaknesses,
opportunities, maintenance challenges, future technology, etc.
What sorts of adjustments would you need to make to
take this approach?
What would be the largest challenges in taking this
approach? How might you overcome them?
Do you “buy” the whole-team view to automation?

17
ROI
What agile has exposed to me…
wrt/Automation & Testing
Testers vs. a Whole Team view
Build teams in ratios – developers vs. testers
Instead invest in cross-functional teams composed of the skills to design,
construct, test, and deliver excellent products
Only testers write test automation
The entire team writes test automation and, oh by the way, can test as
well
Only testers test the application
The whole team participates in writing / running automation, functional
testing, exploratory testing, regression testing, etc.

18
Traditional vs. Open Source
Tools
Instead of leveraging singular, UI-centric tools
Leverage multiple tools at all-tiers of your application stack
Higher cost
Lower cost and a sense of “Community”
Developers won’t use them; costs
Everyone is leveraging the same tool-sets; AND integrating them in
Continuous Integration and Continuous Deployment
People are your primary value
proposition
Instead of looking at testers as a commodity
Consider good testers to be as valuable as any other team member
Instead of thinking that testing is a by-rote exercise with little intellectual
pursuit
Testing is a profession. Testers are valuable craftspeople. Testing is an
intellectual exercise.
Instead of thinking the value is in the test cases
The value is in the testers (people) continuously reevaluating the right
things to test at the right time for valuable feedback

19
http://outrage.typepad.com/crisisanalysis/2010/11/the-tyranny-of-roi.html
The NEW Test Automation
Business Case
So, don’t
Count pennies
Count heads
Count cycles
Work as part of a TEAM
Focus in on testing continuously – just the right things at just the
right time
Providing feedback, transparency, and reaction
Allowing your teams time to learn and improve
Solving your customers’ problems; delivering value

20
Run more tests, faster, unattended, continuous
feedback, learning & adjustments
Yes, saving time. Now what to do with it???
Drive your new VALUE by:
Early feedback for adjustments—risk avoidance, learning, solving
real problems
Having more time to continuously refine your test cases
(automated and not)
Having more time to focus on building Quality into your products
Creatively testing the application (Exploratory Testing, Test Idea
Generation, Test Design)
The NEW Test Automation
Business Case
So, with the additional time…
INVEST in your testers & quality
In building In Quality
In attacking the customers REAL problems
In better testing
In more balanced automation
In team training; in slack time
In better collaboration
In doing more than thought possible
In creative solutions
In adapting to new technologies
In having FUN

21
How do we “Sell”
this change?
You don’t sell it from a testing
perspective. Nor from a
bean-counting perspective.
You allow the results to begin speaking for themselves
as we begin focusing on and talking about—
How one builds truly innovative and creative software teams
That understand and solve customer problems
With simple solutions that fundamentally work
And do so iteratively & quickly, while nimbly adjusting to
customer feedback
But Bob…
That doesn't work in the “Real World”!
My boss would never go for it without ROI justification
and hard measures
Testing IS a commodity
We’ve done automation perfectly – seeing none of the
issues/risks you’ve pointed out

22
But Bob…
Think about what I’m saying; taking
the time and
Invest in people
Invest in better quality & testing
Invest in better product solutions to customer problems
Invest in improved workflow and delivering high customer value
WOW – how crazy is that?
Meet in small groups of 2-3 (pro / con) and chat for 10
minutes about the “sides” and the possibility for change?
Then we’ll debrief
Automation is one of the foundational investments of effective Agile
Adoptions. It’s not a choice, it’s an imperative! It’s a cost of doing business.

23
Typically a 5-Phased Strategy
1. Strategy – This session: 3 Pillars + Pyramid + Team
2. Team Formation
3. Tools Selection
4. Training & Infrastructure Development
5. Automation Development
6. Care & Feeding
Phase 2
Team Formation
GOAL whole team ownership
Skill-set discussion; SDET vs. Test Professional?
May influence
Development focus on Unit Testing
Automation team to “gain momentum”
Agile execution (Scrum, Kanban, transparency, demo, etc)
Connect the dots architecturally
Bias: I like to lead automation with a Test Automation
Architect
Rather than with someone from the development team

24
Phase 3
Tools Selection
Open Source vs. Commercial
Singular, all-purpose vs. Tool-kit
Cobble together or Integrate disparate tools
Always perform a “Bake off” within your teams
If Open Source, consider support implications
Connect to development tooling
Including CI and CD
Additional Considerations: Test data, virtualization,
DevOps
Be prepared to iterate, learn, and possibly fail
Phase 4
Training & Infrastructure
Design, coding, and documentation standards
Templates
Community of Practice
Example code, standards, forum, collaborative groups
Models:
Keyword, Data-driven, Table-driven, DSL, Model-driven, others
Test Suite; Hierarchy, Naming
Commercial: invest in training and/or consulting
Open Source: hire and/or acquire direct expertise
Also can build the expertise

25
Phase 5
Automation Development
Move it to your teams ASAP
Make it a part of Definition-of-Done
It should become a natural outcome and not a choice or question
Complete it within each sprint
SAFe as a goal model for completing work in the same sprint!
Salesforce as well
It needs to be a negotiated part of your Product Backlog
Have % targets;
For every story:
The TEAM decides the requisite automated tests at each level of
the pyramid?
Phase 6
Care & Feeding
Automation evolution roadmap
Merge with your architectural look-ahead
Merge with your Product Roadmap
Factor in ongoing automation investments
Refactoring & Repairs as part of your Technical Debt
handling
Consider it at the same level of investment as your
application code
Stop-the-Line
Commitment

26
Communication
Leadership Vision
High-level Plans, Strategic sharing with your teams
Leadership collaboration (Development + QA)
Intent to doggedly pursue automation
Information Radiators
Roadmaps & Product Backlogs
Sprint & Release Reviews
Even though it’s hard to show “working software”
Let the “process” transparently communicate your progress,
impediments, plans, challenges
Let’s end with…
Agile Automation Strategy
Based on what you’ve heard, discussed, learned
Get together in “pairs” and chat about this for 20
minutes. List your strategic approaches. Then leverage
FFA to capture your driving forces FOR and AGAINST
moving towards Agile Automation
Most importantly, focus on strategies to overcome your
obstacles.
Then we’ll gather your results

27
Wrap-up
THANK YOU!
I hope I’ve added to your previous
assumptions and understanding
Inspired you to change your view towards Automation
ROI and investment
Introduced you to models for
attacking Agile Automation Strategy
and ROI
Final questions or discussion?
5454

28
Contact Info
Bob Galen
Principal Consultant,
RGalen Consulting Group, L.L.C.
Experience-driven agile focused training,
coaching & consulting
Cell: (919) 272-0719
bob@rgalen.com www.rgalen.com
bgalen@velocitypartners.net www.velocitypartners.net
Blogs
Project Times - http://www.projecttimes.com/robert-galen/
BA Times - http://www.batimes.com/robert-galen/
Podcast on all things ‘agile’ - http://www.meta-cast.com/
References

29
Links
http://www.agileengineeringdesign.com/2012/01/7-
deadly-sins-of-automated-software-testing/
Link to PDF of Pillar-1 chapter
Links to “old” STP 3-part automation series
Link to Mary’s – Agile Test Manager article
Open Source Tools
Unit Level
x-Unit
Junit, nunit, phpunit,
ATDD
FitNesse, Robot Framework
BDD
Cucumber, JBehave
UI
Selenium, Watir, HP-UTF

30
…DD
much from the XP space
TDD - Test Driven Development
Test-first; Unit Tests as a design aid
Debate around creating the unit tests “first”
ATDD - Acceptance Test Driven Development
User Story – Acceptance Tests
Executable Requirements
BDD - Behavior Driven Development
jBehave, Cucumber; Gherkin – Given, When, Then
More useful phrasing
Tests as functional documentation
Product Owners, Customer – writing tests
Agile Testing Quadrants
Brian Marick; Lisa Crispin & Janet Gregory
Exploratory testing
Scenarios
Usability testing
UAT
Alpha / Beta
Unit tests
Component tests
API tests
Functional tests
Story tests
Examples
Prototypes
Simulations
Performance testing
Load testing
Security testing
Non-functional requirements
Q1
Q2 Q3
Q4
Automated &
Manual
Automated &
Manual
Manual
Automation,
Tools, and
Manual
Business Facing
Technology Facing
SupportingtheTeam
CritiquetheProduct

Test Automation Strategies for the Agile World

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Test Automation Strategies for the Agile World

Ähnlich wie Test Automation Strategies for the Agile World (20)

Mehr von TechWell

Mehr von TechWell (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Test Automation Strategies for the Agile World