Agile Testing Transformation is as Easy as 1, 2, 3 by Michael Buening

1. Agile Testing
Transformation is as Easy
as 1, 2, 3
Presented by: Mike Buening
Venue: QA or the Highway 2019
Date: February 19, 2019

2. 2
Currently serving as Director, Digital Assurance and Testing in
the East Ohio Region for Sogeti US.
Most recently working as a QA and Testing consultant with
gas and electric utility companies in Columbus, OH. My focus
is on testing discipline uplift in support of an agile
transformation.
Previous roles include information analyst, business analyst,
test lead, project manager, test manager, and department
head, over a 20 year IT career focused mostly on quality
assurance of technology projects and products.
I obtained a Ph.D. in biochemistry from The University of
Notre Dame, which has nothing to do with IT but naturally
aligns with quality assurance and testing. Science is all
about acquiring information through experimentation.
Software testing is exactly the same thing.
During my first 30 years I spent time in Japan, Alabama,
Ohio, Okinawa, Indiana, Kansas, South Dakota, and Indiana
(again). Along the way I met my wife and brought her with
me.
For the last 20 years I have lived in Central Ohio, currently in
Lewis Center with my wife, three kids, and two dogs.
Outside work, my interests include music and living
vicariously through my kids.
Introduction

3. 3
Agenda
• What is Agile Testing Transformation?
• A Roadmap for Transformation
• Three Phases of Transformation
• Workforce Transformation
• Common Gotcha’s

4. 4
What is Agile Testing Transformation?
Manual Tests Through UI
Automation
Suites
Unit Testing
Automated
UI Tests
Automated
Functional/API
Tests
Automated Unit Tests
Manual &
Exploratory
Testing
Agile Transformation
Traditional
(find bugs)
Agile
(prevent bugs)

5. 5
Roadmap to Agile Testing Transformation
• Establish TEC
• Test Expertise
• Test Process
Standards
• Metrics
• Test Data &
Environment
Requirements
Optimize Test
Fundamentals
• Functional and
Non-functional
test case
automation
• CI/CD build
automation
• Environment
Virtualization
• Test Data
Management
Implement
Basic
Automation
• Analytics & AI
• Model-Based
Testing
• Automated Test
Design
• AI-Driven Test
Set Design
Smart QA
Automation and especially smart test automation is poised to bring about significant changes in the way QA
and testing is done over the next two to three years and organizations need to have a strategy and roadmap
in place if they want to reap its benefits.
We recommend a phased approach in three stages, i.e. first, the optimization of testing, second,
implementation of basic automation, and third, the adoption of intelligent and self-adaptive test automation
solutions to make automation "smarter."
1 2 3

6. 6
Step 1: Optimize Test Fundamentals
• Establish TEC
• Test Expertise
• Test Process Standards
• Metrics
• Test Data & Environment Requirements

7. 7
Test Fundamentals
• Establish Test Excellence Center (TEC)
• Maintains standards for processes and procedures
• Supports tools used for testing
• Provides specialty testing services
• Provides consulting on best practices and troubleshooting complex testing concerns
• Does NOT provide or manage testing staff – testing expertise is decentralized and assigned to individual teams
• Supports developing solutions around business domains where data and environment requirements have commonality
TEC
Team 1 Team 2 Team 3 Team 4 Team 5 Team n
Business
Domain A
Business
Domain B
IT

8. 8
Unit Test
Plan
Unit Test
System Test
Acceptance Test
The V Model
Development
Levels Test Levels
Business
Requirements
Acceptance
Test Plan
System
Requirements
System & E2E
Test Plan
Integration
Test Plan
Design
Documents
Program
Specifications
Static
Testing
Dynamic
Testing
Integration
Test
Development IT Test Org
Business
Org
Coding
Solution Requirements
Functional Reqs
Non-Functional Reqs
Regression
Testing
Performance
Testing
Operational
Readiness Testing
Application
Operations
Test Fundamentals

9. 9
Test Strategy &
Planning
Test Design
Test Execution
Test Closure
 Starts at project inception – waterfall, agile, hybrid, other
 Enable strong business and functional requirements elicitation
 Initiate work on test strategy and planning
 Key roles: Test Lead
 Refine requirements and enable strong solution designs
 Use a variety of test design techniques using risk as a driver for
coverage and depth of testing
 Design and build automated testing framework and test scripts
 Key roles: Test Analyst, Automated Testing Engineer
 Manage test execution by a variety of resources
 Monitor test execution status – adjust plans accordingly
 Manage defect lifecycle including triage, resolution, retest cycles
 Key roles: Tester, Test Lead
 Summarize test results – quality of the software being tested;
success and efficiency of the testing process
 Build and groom reusable regression test suites
 Key roles: Test Lead, Test Analyst
Test Fundamentals

10. 10
Test Strategy
& Planning
Test Design
Test Execution
Test Closure
Test Strategy
& Planning
Time
Time
Waterfall Agile
Test Design
Test Execution
Test Closure
Strategy & Planning consider full project
scope and anticipate all testing needs for
complete project
Strategy is a living, reusable, driver relevant
for a standing product team
Planning focuses on current and near term
sprint objectives
Design is requirements driven using risk to
determine depth of coverage
Design flows from story card acceptance
criteria and definition of done. Automate
everywhere it makes sense to do so.
Testing follows the plan. Often reliant on manual
execution, only automating where ROI is clear.
Testing is tightly coupled with development
and proceeds until definition of done is
achieved and product owner is satisfied.
Closure summarizes test results and provides
reusable test assets to the application team.
Closure proceeds via sprint review & retrospective.
Automation is critical as this sprint’s functional
tests are next sprint’s regression.
10

11. 11
Step 2: Implement Basic Automation
• Functional and Non-functional test case automation
• CI/CD build automation
• Environment Virtualization
• Test Data Management

12. 12
Implement Basic Test Automation
 Automation of web services testing for internal and 3rd party calls
 Virtualization of services for early integration testing and defect identifications
 Cross browser testing with single script
 Mobile testing capabilities across various Hardware and Software
 Non Functional Testing tools and approaches
 Orchestration platform for Performance Testing and Engineering
 Centralized test management, reporting and monitoring
 Leverage automation to minimize manual test case execution
 Incrementally build and execute regression and ETE tests
 End of sprint checkpoints for regression suite optimization and review
 In-Sprint functional automation design and execution
 Enable CI/CD pipeline
Functional,
Regression &
DevOps
Services Testing,
Virtualization, Cross
Browser and Mobile
Testing
Performance & Non
Functional Testing

13. 13
Step 3: Smart QA
• Analytics & AI
• Model-Based Testing
• Automated Test Design
• AI-Driven Test Set Design

14. 14
Smart QA: Intelligent Analytics – “What Matters Most ?”
 Tester efficiency
 Test case analytics
 Test requirements
analytics
 Test design analytics
Test Management
data
 Root cause analysis
 Defects by module
 Defects by release
 Defects found by tester
 Defects introduced by
developer
Defect data
 Heat Map analysis
 Automated Incident
Analysis
 Smart Root cause
analysis
 Performance analytics
Production Data
How do I apply
Predict what to test
Predict efficiency of workforce
Predict bottlenecks to release
Predict speed of release
Predict quality of release
 Conduct
Static/Dynamic code
analysis
 Generate Technical
Debt, Code Coverage
& Lines of Code
Metrics
Code data

15. 15
Smart QA: Emerging trends envision an AI driven future for QA
Role of BOTS in Testing
• Integrate tests as part of DevOps pipeline
• Test environment setup
• Test data generation
• Test automation
Trends for AI in Testing
• Intelligent automation
• Predictive analytics
• Descriptive analytics
• Self-learning, cognitive platforms

16. 16
Workforce Transformation

17. 17
The tester role is split into two: SDETs and Quality Engineers. Quality Engineers with their deep product knowledge and
test/quality domain from an E2E view can provide expertise & focus on what should be tested.
SDETs become the go-to engineers for projects that need TECHNICAL expertise in test automation and
integration. SDET constantly designs better and more creative ways to identify weak spots.
SDET Responsibilities
 Build frameworks and packages
required to implement automation
 Write code to test: UI and API
level testing
 Create code and tools so developers
could test their own code
 Design focused on testability,
robustness, and performance – code
review, component testing,
continuous testing
SDET Technology Skills
 Selenium, JUnit, Nunit, TestNG, and other
similar tools
 Knowledge beyond Automation tools like
QuickTest Pro, TestDirector, SilkTest
 Webservice testing & virtualization
 Agile/Scrum and DevOps knowhow
 Basic knowhow with high-level OO
language like Java | Python
 RDBMS, SQL
Quality Engineers Skills
 Good depth of knowledge for the
application
 Domain, Manual Testing, UI Testing
 Mobile/Cross browser testing
 Automation Enabled testers
 Basic BDD, Basic Java
 Basic Continuous Integration Test
Software Development Engineer in Test (SDET) vs. Quality Engineers

18. 18
QA COE
Independent Teams
QADevelopment
BA Build & Deploy
QA TEC
Hybrid Teams
Initial State
Silo Teams with Defined Roles & Responsibilities
Intermediate State
Hybrid teams, SDET as catalyst to drive
transformation
End State
Integrated full stack teams
Single Skill I Profile Teams
A Primary and Secondary Skill T Profile Teams
• User Story Analysis
• Technical Design
• Code Development
• Code Review
• Unit Testing
• Build Support
• User Story Analysis
• Test Plan and Design –
Manual, Automation,
Data and Non Functional
• Test Execution
• Defect Management
• Business Engagement
• User Story
• Acceptance Criteria
Definition
• Code Integration/Merge
• Create Build
• Deploy Build in Target
• Build Checkout
System Team
Scrum 1
Scrum 2
Scrum N
DEV/BA
Overlap
Zone SDET
E2E Testing Performance DevOps
TDMAccessibility TestingApp Scan
QA TEC
Integrated Team
Pi Profile Team
System Team
Performance DevOps
TDMAccessibility TestingApp Scan
Scrum 2
Scrum N
E2E Testing
Cross
Functional
Team (Dev /
BA / SDET)
Scrum 1 • Business Engagement
• Model Based Impact Analysis
and Test Design
• Test Harness Dev.
• Code Development
• Continuous Integration;
Automated Build, Deploy
• Sprint Demo
Workforce Transformation: Evolution of Roles to Support this Journey

19. 19
Common Gotcha’s
• An emphasis on test automation without fundamental testing discipline in place leads to waste
in the form of inefficiency.
• Every functional test case does not need to be automated
• Regression testing is still the largest and costliest testing type over the lifespan of a product
• Functional test automation should minimize the cost and increase the speed of regression testing
• Across the industry, test data and test environment management are the top blockers for
effective progress in QA and testing.
• Automated testing needs dependable testing environments
• Automated testing needs consistent, well defined, test data
• External dependencies can wreck havoc on Sprint commitments
• The more the team can control their own destiny the more efficient they will be
• Service Virtualization is a powerful tool used to decouple external dependencies
• Support! All of this automated testing is software that needs run support. Don’t forget to build
it into the plan and make sure the support staff are included in the workforce transformation.

20. 20
Common Gotcha’s
Automated
UI Tests
Automated
Functional/AP
I Tests
Automated Unit Tests
Manual &
Exploratory
Testing
Fully Integrated End-to-End Business Process Testing
Integration Testing - Functional, Performance,
Regression for Impacted Systems
The full scope of testing required for high quality
delivery often needs additional considerations beyond
what a team is capable of addressing, reasons include:
• Knowledge
• Skill
• Environment/Data/Infrastructure
• Access
• Tools
• Capacity
• Time
Coordination and input from
people and systems outside the
Scrum team might be needed to
perform integration testing and
end-to-end business process
testing.

21. 21
Common Gotcha’s
When several Agile teams are sharing
dependencies on common enterprise systems,
testing becomes a cross-team challenge
requiring a high degree of coordination and
communication.
Enterprise Data
Sources
This may seem at odds with Scrum’s emphasis on
the team owning all the work needed to deliver
to production. However, if each of the teams are
testing the middle tier and enterprise data
systems independently they will likely collide
with each other, generating uncertainty which
can delay delivery and introduce defects to
production.
Automated
UI Tests
Automated
Acceptance
Tests
Automated Unit Tests
Automated
UITests
Automated
Acceptance
Tests
AutomatedUnitTests
Automated
UITests
Automated
Acceptance
Tests
AutomatedUnitTests
Automated
UITests
Automated
Acceptance
Tests
AutomatedUnitTests
Enterprise Data
Sources
Middle Tier Integration

22. 22
Summary
• Organizations embarking on an Agile Transformation need to plan the testing transformation
• Agile Testing Transformation should be built in a solid, risk-based, fundamental testing process
• Establish basic functional and non-functional test automation coupled with a continuous
integration process
• Smart QA uses analytics, AI, and advanced automation techniques to optimize testing
• Agile Testing Transformation requires organizational changes and workforce transformation