Geoff Meyer presented on establishing an automation culture within agile organizations. He discussed challenges that Dell encountered when adopting test automation, such as overemphasis on UI testing and lack of automation skills. Meyer outlined opportunities for automation beyond test cases, including environment setup. He emphasized the importance of identifying focus areas, establishing standards and communities, developing workforce skills, and operationalizing automation. Maintaining the culture requires continuous integration, skills development, and recognizing automation as a shared responsibility.

1. AW4
Concurrent Session
11/13/2013 10:15 AM
"An Automation Culture:
The Key to Agile Success"
Presented by:
Geoff Meyer
Dell, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888 268 8770 904 278 0524 sqeinfo@sqe.com www.sqe.com

2. Geoff Meyer
Dell, Inc.
A test architect in the Dell Enterprise Solutions Group, Geoff Meyer has
more than twenty-seven years of experience as a software developer,
manager, test architect, and business analyst. Geoff co-chairs the Agile
Steering committee within Dell Enterprise Solutions Group which guides
the software development practices of more than 600 development, test,
and UX engineers across three Global Design Centers. He is an active
member of the Agile Austin community.

3. 9/3/13%
Automation Culture:
Essential to Agile Success
Agile East 2013
Geoff Meyer, geoffrey_meyer@dell.com
November 2013
Last updated: August, 28, 2013
Session Objectives
• Challenges to expect when adopting test
automation during your Agile transition
IS
• Practical solutions to what & when to automate
• Real-world examples − from a large organization
perspective
• Challenges automating in a HW-dependent
environment
• For organizational leaders driving strategy
IS
•
A Tools Discussion
NOT
•
Focused on how to develop Test Automation
1%

4. 9/3/13%
Introductions
3
Geoff Meyer
• Dell Inc, 1998 – present
– Test Architect
› Agile Test & Automation Strategy
– Agile Steering Committee Co-Chair
– Global Projects
› 15 Scrum teams (2 regions)
› 9 scrum teams (2 regions)
› 7 Scrum teams (4 regions)
• NCR Corp. 1984 – 1998
– SW developer, Project Lead, SW Manager
• B.S. Computer Science, San Diego State University
• Masters Engineering Management - NTU
4
2%

5. 9/3/13%
Agenda
• Why Automation is essential to Agile
• The Dell Landscape and Agile @ Dell
– Organizational Context Matters
• Challenges and Common Pitfalls
• The Automation Landscape
• Foundations of an Automation Culture
• Care and Feeding of the Automation Culture
5
The Need,
The Challenge,
The Pitfalls
6
Ø
3%

6. 9/3/13%
Why is Automation So Important in Agile?
• Near-term
– Ensures that you don’t break what you just built
– Provides safety net for developers & rapid feedback to new changes
– Continuous Integration and use of Build Verification Test (BVT)
• Long-term
– Maximizes velocity of Scrum team
– Creates capacity for Exploratory and ad-hoc Testing
– Enables activities that can’t be done cost-effectively by humans
And if you don’t…
Quality is at risk from an unmanageable regression suite
Differences with Automation in Waterfall?
In Waterfall…
• Automated tests are derived from the backlog of
completed features
– In Agile, Automation can be incorporated in the requirement
• Testing and automation is performed after Development is
“complete”
• Focus is on first-time discovery of defects and optimizing
your test coverage
vs. Agile… where automation development is ongoing
and provides immediate feedback throughout
development
8
4%

7. 9/3/13%
The
Context @ Dell
“Apply Test Automation in the
context of your organizational
automation needs”
http://amazngwallpapers.blogspot.com
--- Bob Galen, Agile Coach
9
Dell Enterprise Solutions Group
Global Design and Development using Agile Scrum
Silicon Valley
Design Center
Austin
Design Center
Noida Design
Center
Bangalore
Design Center
10
5%

8. 9/3/13%
Dell Enterprise Solutions
Software Products
• Server Systems Management
• Converged Infrastructure Systems Management
• Private Cloud Server Management
• Console Plug-ins (i.e. for SCCM, vCenter…)
Common Product Characteristics:
• Large hardware support test matrix
• Software is installed in the Data Center
• Enterprise update cycles ~6 months
• Products must function even as underlying HW, FW,
BIOS, and Drivers are upgraded - SUSTAINING
11
Agile @ Dell Roles/Responsibilities
Product
Owner
• Small teams
Product Owner Proxy
1 per Scrum
• Co-located
Scrum Master
1 per Scrum
• Distributed Projects
Development
4-5 per Scrum
• Teams formed from
functional silos
Test
2:1
Tech Pubs
1 per 3 scrum team
UI
1 per Scrum
Scrum teams
12
6%

9. 9/3/13%
Agile @ Dell with Adaptations
Feature
Complete
Release
Plan
Release
Exit
Code Freeze
Sprints
1
1
2
2
1
2
3
3
…
3
N-1
N-1
N
N
N-1
…
N
Hardening
Stability
Extended Sprint Test
Software System Test
OLP
Define
Plan
Develop
Launch
13
Pitfalls encountered at
• Development didn’t historically automate unit tests
• Build teams were staffed with non-Build practitioners
• Minimal guidance to Test beyond “Go forth and automate”
• Architecture(s) not optimized
for Automatability
• Automation was interpreted by
many as “Automate the UI”
• Insufficient SW engineering
background across Test teams
14
7%

10. 9/3/13%
Which Resulted In…
• Over-emphasis on UI automation
• Test Automation not keeping up within the sprint
• Feature Devotion1
• Multiple automation tools & licenses
• Test scripts not designed for re-use
1 – “A nasty condition where people start valuing ticking off features more than tracking the real outcome of the
project.”--- Martin Fowler
15
The Automation
Landscape
http://amazngwallpapers.blogspot.com
16
8%

11. 9/3/13%
Opportunities for Automation
• It’s not just about automating test cases
• Can also provide efficiencies to:
– Test Preparation, Setup and Configuration
– Non-Functional Testing (“ility”)
› Longevity, Scale and Performance Characterization
– Compatibility Testing (Solution and Device)
17
Test Preparation
• Bare-metal Deployment
• Setup & Configuration
– OS Provisioning and Configuration
• Test case staging
• Environment Cleanup/Baseline
• Virtualization-based test
environments
18
9%

12. 9/3/13%
Test Content Development
Whole team ownership of QA & Automation
• UI automation on Customer Usage
workflows
• Automated CI (UT, Build,
BVT)
• Web-services (or CLI) Test
Automation for Functional
Testing
• In-Sprint, Automated
Acceptance Tests
• Unit Test Automation
2
CAUTION: Application Architecture can be an enabler or inhibitor
2 - Agile Testing: A Practical Guide for Testers and Agile teams
19
Application Architecture Matters
• Does it facilitate testing at the API/Services level?
• Does the Business/Error Logic reside below the services
level?
• Does the UI architecture support test automation other
than record-playback?
For Large organizations:
– Standardize UI architectures across the product portfolio
› HTML5, Flex, Silverlight…
– Standardize Service architectures across the product portfolio
› SOAP, REST, CLI, API
20
10%

13. 9/3/13%
Prioritize and Identify what NOT to Automate
Prioritize High ROI
1.
BVT Candidate Acceptance Test
–
Test of core functionality that executes in a short duration
2.
Core Functionality to be run in Nightly Regression
3.
Sustaining Test Candidate Test
–
Functional test that verifies the application can withstand subsystem
changes
What NOT to Automate
•
Everything
•
GUI – Except for High-value customer usage flows after
the UI is stable
•
Tests that would only find low severity bugs
–
Where the product is unchanging
21
Non-Functional Testing
• Performance Characterization
• Longevity
3
• Stress
• Scale
• Concurrency
Often times analysis of
these areas are simply
too cost-prohibitive to
be done manually
3- http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/
22
11%

14. 9/3/13%
Foundations of
an Automation
Culture
Images.yahoo.com
23
Keys To A Culture Transition
• Development vs. Test
• A different “School” of Test4
• Evolve from Functional Responsibility
…to
Whole Product Ownership
4 – Scott Barber “Approaches to Software Testing: An Introduction “
24
12%

15. 9/3/13%
Establishing an Automation Culture
• Identify
• Near and long-term automation focus areas
• Inventory the culture and skillset of organization
• Establish
– Tooling and Infrastructure Standardization
– Your ‘community’
• Develop
– Workforce transition plan
• Organize
– Embedded vs. Specialized Automation team
– Re-align project staffing
• Operationalize
25
Identify
Focus Areas for Automation
4
4 - http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/
26
13%

16. 9/3/13%
Establish
• Tooling and Infrastructure
– Tooling Standards and Frameworks
– Team to develop/manage home-grown tools
– Lead tool evaluations to meet new architectures &
technologies
Multiple automation Tools/FW’s may be
needed to automate at layers of testing:
•
•
•
Unit level
Middle tier
UI-driven
• Community: Internal and External
27
Develop & Organize
• Adjust Hiring Practices
• Re-align Project staffing
• Embedded Automation vs. Specialized Team
• Training
– New-hire
– Ongoing
28
Confidential
14%

17. 9/3/13%
Operationalize
• Whole team commitment in words and actions
• Include Test Automation in Acceptance Criteria
– Unit
– Acceptance
• Continuous Integration and BVT
• Establish Metrics, Governance & Rewards
29
Automation @ PG Enterprise Solutions Group
How did we go about it?
Process/Project
Non-Functional
Scale, Longevity
Automated BVT
UI Automation
Service-level
automation
(CLI & Web-services)
Automation
Day
Setup &
Config
2012
2013
Automation
Content
Libraries
Automation
Architects
Culture/Organization
IVT
Device
IVT
Networking &
Servers
2011
2010
Established ALT
In-Sprint
Automation
Internal
Training
Programs
“Automate-First”
Accelerate
Automation
Skillset
2014
Maximize
Utilization
TDD &
Automated
UT
30
15%

18. 9/3/13%
Care and
Feeding of the
Automation
Culture
31
Maintaining the Automation Culture
• Operationalize Automation
– Include Test Automation in the Acceptance Criteria
• Encourage Community Practice and Participation
– External - Industry User Groups & Conferences
– Internal - Brown-bag sessions, Showcases, Mini-conference
• For Large organizations:
– Overcome skillset deficiencies by adjusting staffing strategy
– Automation Leadership Team, Automation Architect(s)
– Continually monitor the alignment of Architecture, Dev and Test
– Develop reward systems
32
16%

19. 9/3/13%
Recognizing a Successful Automation Culture
• Automation is a shared responsibility
• Automatability is a key architectural
consideration
• Continuous Integration & BVT
• Teams are staffed to include an
automation skillset
• Automation is operationalized
Resources
• Agile Manifesto - http://agilemanifesto.org/
• Articles:
– http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environment/ SmartBear
– http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ - Lisa Crispin
– http://www.stickyminds.com/sitewide.asp?
Function=edetail&ObjectType=COL&ObjectId=17793&tth=DYN&tt=siteemail&iDyn=2 – Rajini
Padmanaban
– Beyond Agile Testing: http://www.utest.com/int-v1/a/beyond-agile-testing - uTest
• Books:
– Scaling Software Agility: Best Practices for Large Enterprises – Dean Leffingwell
– How Google Tests Software – James Whittaker, Jason Arbon, Jeff Carollo
– Agile Testing: A Practical Guide for Testers and Agile teams – Lisa Crispin, Janet Gregory
– http://www.dorothygraham.co.uk/automationExperiences/index.html - Dorothy Graham
• Presentations:
– Approaches to Software Testing: An Introduction – Scott Barber
– Agile Testing: Challenges Beyond the Easy Contexts – Bob Galen
34
17%

20. 9/3/13%
Questions?
35
18%