How We Built Test Automation within a Manual Testing Organization

How We Built Test
Automation within a Manual
Testing Organization
Randy Pellegrini, Andy Doan

The Regence Group
randy.pellegrini@regence.com
Session Track Number: 1608

June 5–9 Orlando, Florida

The Premier Event for Software and Systems Innovation

Bios

Randy Pellegrini
Software Quality Engineer at Regence. Portland State
University graduate in Business Information Systems with
a Minor in Technical Writing.

An Doan
Software Quality Engineer at Regence. Oregon State
University graduate with a Major in Computer
Engineering and a Minor in Computer Science.

2 © 2011 IBM Corporation


About Regence

The largest health insurer in the Northwest/Intermountain states
 Washington, Oregon, Idaho, Utah

Largest provider networks in most markets
 Nearly 6,000 employees
 2.5 million members
 Gross revenue of $8.9 billion
 Net revenue: 1% of premium because we are nonprofit

If we were for-profit, Regence would place about 293 on Fortune 500, above
Nordstrom and Ebay.



Three Phases to Automation Maturity – Agenda

Phase 1 – Basic Test Automation - Record and Playback
 Make the switch to automated testing

 Setup test automation infrastructure

Phase 2 – Code-driven Testing and Communicating Results
 Develop report structure for the automated tests

 Initiate a common coding methodology that started moving away from record
and playback scripting

Phase 3 – Integrate code-driven testing across the enterprise
 Refined a common coding methodology to allow group java coding, automated
test execution and enhanced reporting

 Implement Agile processes



Phase 1 – Basic Automation

Automated
Testing Phase 1 – Basic Automation Setup

Manual
Testing

Low test High test
frequency frequency



Phase 1 – Basic Automation Overview

Shortcomings of Manual Testing

Testing Methodology Executing and Reporting
 Manual Regression Testing  Unable to report quality of daily builds.
– Large amounts of time & labor
– Impractical to do every build
– Difficult to test everything



Phase 1 - Switch to Automated Testing

Phase 1 – Basic Automation Setup
Make the switch to automated testing
 Testing leadership chooses to promote automated testing
 Promote staff positions to push automation in organization
 Create communication channels for disseminitation of information
 Train, Train and Train some more
Setup test automation infrastructure
 Choose the automation tool set
 Get dedicated testing hardware




Testing Leadership Supports Automated Testing
 Testing leadership commits to and plans for automated testing path
 Software tool is chosen




Promote Key Staff Position(s) to Push Automation in Organization
 Dedicate position fulltime to set up communication, training and
infrastructure
 Develop/hire staff with a balance of testing and development skills



Create Communication Channels for Dissemination of Information




Train, Train, and Train Some More
 Create a training process for testing staff to learn automation tools
 Implement a curriculum of object oriented methodology for QA staff
 Set the time aside to train staff



Phase 1 - Setup Test Automation Infrastructure

Choose Automation Tool Set

Test Development – Rational Functional Tester
• Rational Tools already in Enterprise

Source Control – CA Harvest
• Largest Enterprise foot print

Test Management – Rational CQ Test Manager
• Integrated in a non-standard method to ensure loose
coupling between testing tools.



Phase 1 - Setup Test Automation Infrastructure

Dedicate Hardware to Execute Test Automation

2nd desktop for every tester.
• Easier then test lab.

Citrix desktop
• Quick deployment

Virtual Machines
• Multiple Configurations and Browsers



Phase 1 – Basic Automation Summary
What We Accomplished in Phase 1
 Record and Playback  Manual Execution of scripts
 Easy to Learn  Simple
 Fast Development Time

Shortcomings Phase 1

 Record and Playback  Manual Execution of scripts
– Difficult to Maintain – Requires user interaction
– Cannot Record Complex Test Cases  Manual Emailing of results
– Cannot Share Work Effort – Very simple reporting
– Difficult to locate old test logs



Phase 2 – Code Driven Automation

Automated Phase 1 – Basic Automation Setup
Testing

Manual
Testing

Low Test High Test
Frequency Frequency



Phase 2 – Why Advance to Code Driven Automation
Solving the Shortcomings of Phase 1

Testing Methodology Testing Methodology
 Record and Playback  Script using java coding
– Difficult to Maintain  Improve Maintainability
– Cannot Record Complex Test Cases  Reduce Repetition of Code
– Cannot Share Work Effort  Improve Test Coverage

Executing and Reporting Executing and Reporting
 Manual Execution of scripts  Use Clear Quest Test Manager
– Requires user interaction  Combine multiple scripts into test
 Manual Emailing of results suites
– Very simple reporting  Execute scripts in succession
– Difficult to locate old test logs  Build A Test Results Web Interface
 Allow historical reporting



Phase 2 – Improve Maintainability

Project
Project
Super Helper Common
Constants
Library

Login
Feature
Feature
Common
Constants
Library

Fix One Time
Test Scripts V.S.
Many Times



Phase 2 – Reduce Repetition of Code

2.Enter Text Into Form
3.Click Submit
4.Validate Text on Page




Popular Manual Test Actions

Manual Steps: Methods:

3.Enter Text Into Form 3.setTextByProp

5.Click Submit 5.clickObjectByProp

7.Validate Text on Page 7.validateObjectWithProp




Place Constants in Common Location



Phase 2 – Improve Test Coverage - Example

•Search for a Name

•Verify that a minimum
of 3 results are returned



Phase 2 – Improve Test Coverage - Example
Drive The Test With Code



Phase 2 – Improve Test Coverage

Increase Test Coverage By Using Code-Driven Tests

Test Coverage •Generated Data
•Datapools, spreadsheets
•Manual Testing



Phase 2 – Improve Test Complexity

The Cost Of Switching To Using Methods

Increase Test Coverage
Test Coverage

Switch From Armament of
Record/Playback to Methods
using Methods

Write The
Methods

Maturity



Phase 2 - Develop Automation Reporting

Communicate Success - Daily Automated Test Results

• Email reports

• Web reports



Phase 2 – Code Driven Automation Summary
What We Accomplished in Phase 2
 Executing
 Individual Java Coding  Can run in batches
 More Complex Test Cases
 Reporting
 Easy to Maintain
 Results Logged in CQTM
 Code Driven Testing
 Automatically Emailed daily
 View by calendar date

Shortcomings Phase 2
 Individual Java Coding  Executing
 Longer Development Time – Requires user interaction to run and
commit results
 Cannot Share Code Across Projects
– Time Consuming
 Repetition of work
 Repetition of roadblocks



Phase 3 – Enterprise Code Driven Automation

Automated Phase 1 – Basic Automation Setup
Testing
Phase 3 – Enterprise Code Driven Automation

Manual
Testing

Low Test High Test
Frequency Frequency



Phase 3 – Enterprise Code Driven Automation Overview
Solving the Shortcomings of Phase 2

Testing Methodology Testing Methodology
 Script using java coding  Script using java coding
– Longer Development Time  Shorter Development Time
– Cannot Share Code  Share Code Across the Enterprise
– Repetition of Work  Reduce Repetition of Work
– Repetition of Roadblocks  Implement Agile Process
Executing and Reporting Executing and Reporting
 Execution of scripts  Use In-house Test Harness
– Requires user interaction  Develop a Common Process
– Time Consuming  Execute scripts in succession



Phase 3 – Share Code Across The Enterprise

Common Methods Shared
Across Multiple Projects

Using javadocs to share code



Phase 3 - Develop A Common Process
1. Scheduled Task calls Perl
script

3. RFT script interacts with
the Application Under Test

5. Reports results to html log
file

7. Perl script parses html log
for pass/fail

9. Perl script logs pass/fail
into CQTM database

11.Web Site reports results

13.Emailing service sends
results daily



Phase 3 – Implement Agile Process for Test Automation
Use Agile Methodology to Keep Automation on Track

Benefits of Using Scrum

 Creates a cohesive team environment

 Allows incubation of automation concepts

Challenges You Will Face

 Time Onboarding New Testers

 Environment Instability

 Tool Instability

 Tool Upgrades

 Upgrade of OS & Browser on Test PCs




Implement Basic Agile Methodologies for Test Automation

 15-minute Daily Scrums

 Beginning of Sprint Planning

 End of Sprint Demo

 Story Done Criteria




Success Over the Past Year Show by Increased Velocity

Sprint 1: March 2010
11 User Stories
3 Sprint Participants
Automated 15 simple Build Verification Tests (BVTs) for one project

Sprint 18: February 2011
51 User Stories
13 Sprint Participants
Automated 45 Functional Tests Across 8 projects



Phase 3 – Enterprise Code Driven Automation Summary
What we accomplished in Phase 3
Shared Methods Across The Enterprise
Developed A Common Automation Process
Expanded Test Environments

Implemented Agile processes
What type of testing to expect in Phase 3
 Group Java Scripting  Scheduled Task
 Got more done as a team  Fully Automated
 Easier to learn  No user interaction
 Shorter development time  Modular Architecture
 Sharing methods  Reporting v2
 Common architecture  Links to RFT logs
 Easier to maintain then phase 2  Links included in emails
 Easier to understand phase 2
 Everyone must participate



Conclusion

Benefits of automation we have seen

 Actual reduction in manual effort

 Improved, increased and consistent test coverage

 Create robust automation infrastructure for all projects to use

Next Steps…

 Process proven effective enterprise-wide, now would like to take this to a
wider audience



Daily iPod Touch giveaway
SPONSORED BY
 Complete your session surveys online each day
at a conference kiosk or on your Innovate 2011 Portal!

 Each day that you complete all of that day’s session
surveys, your name will be entered to win the daily
IPOD touch!

 On Wednesday be sure to complete your full conference evaluation
to receive your free conference t-shirt!



www.ibm.com/software/rational

© Copyright IBM Corporation 2011. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of
any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to,
nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing
the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release
dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment
to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the
International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.


How We Built Test Automation within a Manual Testing Organization

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (18)

Ähnlich wie How We Built Test Automation within a Manual Testing Organization

Ähnlich wie How We Built Test Automation within a Manual Testing Organization (20)

How We Built Test Automation within a Manual Testing Organization

Hinweis der Redaktion