SlideShare ist ein Scribd-Unternehmen logo
1 von 82
Downloaden Sie, um offline zu lesen
MB
Full-day Tutorial
4/29/13 8:30AM

The Challenges of BIG Testing:
Automation, Virtualization,
Outsourcing, and More
Presented by:
Hans Buwalda
LogiGear

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
Hans Buwalda
An internationally recognized expert in testing, Hans Buwalda is a pioneer of keyword-driven test
automation, an approach now widely adopted throughout the testing industry. Originally from the
Netherlands, Hans is the CTO of California-based LogiGear, directing the development of the successful
Action Based Testing™methodology for keyword-driven test automation and its supporting
TestArchitect™ toolset. Prior to joining LogiGear, Hans served as project director at CMG (now Logica).
Hans speaks frequently at international conferences on concepts such as Soap Opera Testing, Three
Holy Grails of Test Development, Testing in the Cold, and Jungle Testing. Hans is coauthor of Integrated
Test Design and Automation: Using the TestFrame Method.
4/11/2013

STAREAST 2013, Tutorial MB
Orlando, Monday April 29

The Challenges of

BIG Testing
Automation,
Virtualization,
Outsourcing, and More

Hans Buwalda
LogiGear
© 2013 LogiGear Corporation. All Rights Reserved

Introduction

− industries
− roles in testing

© 2013 LogiGear Corporation. All Rights Reserved

1
4/11/2013

About LogiGear

www.logigear.com
www.testarchitect.com

Software testing company, around since 1994
Testing and test automation expertise, services and
tooling
− consultancy, training
− test development and automation services
− "test integrated" development services

Aims to be thought leader, in particular for large and
complex test projects
Products:
− TestArchitect™, TestArchitect for Visual Studio™
− integrating test development with test management and automation
− based on modularized keyword-driven testing

© 2013 LogiGear Corporation. All Rights Reserved

About Hans

www.happytester.com
hans @ logigear.com

Dutch guy, living and working in California since 2001, as
CTO of LogiGear
Background in math, computer science, management
Original career in management consultancy, since 1994
focusing on testing and test automation
− keywords, agile testing, big testing, . . .

© 2013 LogiGear Corporation. All Rights Reserved

2
4/11/2013

Topics for today
Automation
Designing and organizing tests
Executing tests
Team, organization and process
Off-shoring, globalization

© 2013 LogiGear Corporation. All Rights Reserved

What is "BIG"
Big efforts in development, automation, execution and/or follow up
It takes a long and/or capacity to run tests (lot of tests, lot of versions,
lot of configurations, ...)
Scalability, short term and long term
Complexity, functional, technical
Number and diversity of players and stakeholders
− pigs, chicken, elephants, ankle biters, ...

Various definitions of "big" possible... and relevant...
− "10 machines" or "10 acres"
− "1000 tests" or "1000 weeks of testing"

Big today means: big for you
− "non trivial", you need to think about it

"Windows 8 has undergone more than
1,240,000,000 hours of testing"
Steven Sinofsky, Microsoft, 2012

© 2013 LogiGear Corporation. All Rights Reserved

3
4/11/2013

Existential Questions
Why test?
Why not test?
Why automate tests?
Why not automate tests?

© 2013 LogiGear Corporation. All Rights Reserved

Why test?
People expect us to do
Somebody wants us to
Increases certainty and control
− Showing absence of problems

Finds faults, saving time, money, damage
− Showing presence of problems

© 2013 LogiGear Corporation. All Rights Reserved

4
4/11/2013

Why not test?
It costs time and money
You might find problems . . .
We forgot to plan for it
We need the resources for development
It is difficult
It's hard to manage

© 2013 LogiGear Corporation. All Rights Reserved

Why Automate Tests?
It is more fun
Can save time and money
− potentially improving time-to-market

Can capture key application knowledge in a reusable way
Consolidates a structured way of working
− when established as integral part of system
development process

Can speeds up development life cycles
Execution typically is more reliable
− a robot is not subjective
© 2013 LogiGear Corporation. All Rights Reserved

5
4/11/2013

The Power of Robot Perception
FINISHED FILES ARE THE RE
SULT OF YEARS OF SCIENTI
FIC STUDY COMBINED WITH
THE EXPERIENCE OF YEARS...

© 2013 LogiGear Corporation. All Rights Reserved

Why not Automate?
Can rule out the human elements
− promotes "mechanical" testing
− might not find "unexpected" problems

More sensitive to good practices
− pitfalls are plentiful

maintenance can crush automation...

Creates more software to manage
Needs/uses technical expertise in the test team
Tends to dominate the testing process
− at the cost of good test development

© 2013 LogiGear Corporation. All Rights Reserved

6
4/11/2013

The Power of Human Perception
Olny srmat poelpe can raed tihs.
I cdnuolt blveiee taht I cluod aulaclty uesdnatnrd
waht I was rdanieg. The phaonmneal pweor of the
hmuan mnid, aoccdrnig to a rscheearch at
Cmabrigde Uinervtisy, it deosn't mttaer in waht
oredr the ltteers in a wrod are, the olny iprmoatnt
tihng is taht the frist and lsat ltteer be in the rghit
pclae. The rset can be a taotl mses and you can
sitll raed it wouthit a porbelm. Tihs is bcuseae the
huamn mnid deos not raed ervey lteter by istlef,
but the wrod as a wlohe.

© 2013 LogiGear Corporation. All Rights Reserved

About tests in big projects
Regular tests may be activities, complex tests are
products. In fact any test that you want to run
more than once is a product
Every test that is written down with sufficient
detail should be automated
Automation
− No longer an option in most situations
− Also a key prerequisite of most agile approaches

How tests are written and automated can make
or break large scale testing

© 2013 LogiGear Corporation. All Rights Reserved

7
4/11/2013

Keywords, essential for scalability
Distinguish tasks for test development and for automation
The test developer creates tests using "actions" (my term)
Each action consists of a keyword and arguments
The automation task focuses on automating the actions
Each action is automated only once
fragment from a test with actions
number

actions, each with a
keyword and
arguments

name

new product

P-9009

Sledge Hammer 5

number

quantity

add quantity
add quantity
add quantity

P-9009
P-9009
P-9009

20
3
6

number

check quantity P-9009

quantity

34

quantity

read from top
to bottom

"34" is the
expected value
here
© 2013 LogiGear Corporation. All Rights Reserved

Potential benefits of keywords
More tests, better tests
− more breadth
− more depth

Fast, results can be quickly available
− the design directly drives the automation

Separates the tests from the technical scripting language
− easier to involve business subject matter experts
− the action format allows for easy readability

Less efforts for automation
− "script free" in most cases

Automation more stable and maintainable
− limited and manageable impact of changes in the system under test

Develop tests more early in the life cycle
− deal with execution details later

...

© 2013 LogiGear Corporation. All Rights Reserved

8
4/11/2013

Risks of keywords
Often seen as silver bullet, complications are
underestimated
− often treated as a technical "trick"
− testers can get squeezed and marginalized
• developers and users dictating tests
• automation engineers dictating actions

− or testers get the automation responsibility, thus becoming pseudo
programmers

The method needs understanding and experience to be
successful
− pitfalls are many, and can have a negative effect on the outcome

Lack of method and structure can risk manageability
− maintainability not as good as hoped
− results can be disappointing, approach will be blamed
© 2013 LogiGear Corporation. All Rights Reserved

Case: International Financial Project
One of the largest projects to date with action words
Over 10 000 windows, meant for use in 85 countries
Long development cycle (400 pp, 4 years and counting)
Maintenance very hard
Testing major bottleneck
Much investment in automation techniques were needed
to become successful
Also a lot of attention for team and work environment
helped the success
Team of 35 test developers, 2 automation engineers
© 2013 LogiGear Corporation. All Rights Reserved

9
4/11/2013

Keywords need a method
By themselves keywords don't provide much scalability
− they can even backfire and make automation more cumbersome
− a method can help tell you which keywords to use when, and how to
organize the process

Today we'll look at Action Based Testing (ABT)
− addresses test management, test development and automation
− large focus on test design as the main driver for automation success

Central deliveries in ABT are the "Test Modules"
− developed in spreadsheets
− each test module contains "test objectives" and "test cases"
− each test module is a separate (mini) project, each test module can
involve different stake holders

© 2013 LogiGear Corporation. All Rights Reserved

Example of an ABT test module
Consists of an (1) initial part, (2) test cases and (3) a final part
Focus is on readability, and a clear scope
Navigation details are avoided, unless they're meant to be tested
TEST MODULE

Car Rental Payments
user

start system

john

TEST CASE

TC 01

Rent some cars

first name

last name

car

John
John

Doe
Doe

Ford Escape
Chevvy Volt

last name

amount

Doe

140.4

rent car
rent car

check payment
FINAL
close application

© 2013 LogiGear Corporation. All Rights Reserved

10
4/11/2013

Example of a "low level" test module
In "low level" tests interaction details are not hidden, since they are
the target of the test
The right level of abstraction depends on the scope of the test, and is
an outcome of your test design process
TEST MODULE

Screen Flow
user

start system

john

TEST CASE

TC 01

"New Order" button

first name

control

main

new order

click

window

check window exists new order
FINAL
close application

© 2013 LogiGear Corporation. All Rights Reserved

Re-use actions to make new actions
In the below example we use another sheet, but if you code actions,
you could do something similar
Often low level tests are re-used into these action definitions

ACTION DEFINITION check payment
user

check

control

value

main

last name

# last name

control

main

view balance

window

click

Jones

window

enter

default value

last name
amount
window

argument
argument

control

expected

main

balance

# amount

© 2013 LogiGear Corporation. All Rights Reserved

11
4/11/2013

Action Based Testing
Test Development Plan
Break down

Test Module 1

Test Module 2

Test Module N

Test
Objectives

Test
Objectives

Test
Objectives

Test Cases

Test Cases

...

Test Cases

Automate

Actions

ACTION AUTOMATION
© 2013 LogiGear Corporation. All Rights Reserved

Case: Stock Exchange
Transition from floor-based to screen-based trade

The Four Daltons
(French comic book characters)

Created on basis of an existing standard package
− result: very little specifications

Consisting of four major, different, systems that need to work in realtime
Failures and bugs are not an option:
− core of the financial system of the country, 100K revenue per second
− traders not necessarily following rules

In-depth knowledge limited to four people
− nicknamed "The Four Daltons", after characters in a French comic book series about the
wild west
− none of the four Daltons was involved in testing, testing was in a vacuum

Three months to go...
−
−
−
−

test development (and scripted automation) had failed
test department not cooperating well with developers and domain experts
internal and external auditors had raised the alarm
and... the Dutch Crown Prince was scheduled to put the system into use!!
© 2013 LogiGear Corporation. All Rights Reserved

12
4/11/2013

Case: Stock Exchange
Test set:
− make it comprehensive
− make it in-depth and aggressive
− make it easy to assess and approve

Organization:
− get the right people involved (testing, automation, etc)
− use scarce resources efficiently (in particular the four Daltons)
− work with stake holders to let the process be transparent

Technical:
− use of the keyword method ("action words")
− use "test objectives" so auditors can see quickly what you're testing
− use great test design, don't mix apples and oranges

"Sign off lubrication":
− auditors signed off on the tests, not the test results
− "the test is complete", not "the system works well"

Results:
− deadline was met one day before final date
− the automated tests were the only ones used for acceptance
− no functional errors found afterwards
© 2013 LogiGear Corporation. All Rights Reserved

Question

What is wrong with the
following pictures?

© 2013 LogiGear Corporation. All Rights Reserved

13
4/11/2013

No Y2K Problems in Auckland Airport??

© 2013 LogiGear Corporation. All Rights Reserved

Anything wrong with this instruction ?

You should change your battery or switch to outlet
power immediately to keep from losing your work.

© 2013 LogiGear Corporation. All Rights Reserved

14
4/11/2013

Why Better Test Development?
Are your
suffering from
lame tests too?

Many tests are often mechanical now
−
−
−
−

blindly follows specs or reqs
which often suits ok, but lacks aggression
no combinations, no unexpected situations
"methodical" does not have to mean "mechanical"

For a higher “ambition level” you need
− understanding of the system under test, and the business under test
− analytical understanding of what could go wrong
− creativity, and the commitment to use it

Poor test development results in
− cumbersome automation due to lack of focus
− tedious retest cycles, loosing the agile advantage

© 2013 LogiGear Corporation. All Rights Reserved

Test Design
Effective test breakdown (into test modules)
− make sure every test module has a clear focus
− keep different kinds and levels of tests separate

Right level of actions
− as “high level” if possible, hiding as many details as
much as possible
− but not if the details are relevant for the test

It is my believe that successful automation is not a technical
challenge. It is most of all a test design challenge.

© 2013 LogiGear Corporation. All Rights Reserved

15
4/11/2013

Case: A Financial Project
Large project, with many consultants
− embraced the approach
− however, they were from our competitor

One of the first with action words
However, the project was confident about their
test development, no help needed
Result: many very hard to maintain tests, and
way too many action words
− crushing complexity
− almost the end of the action words method
− one memo saved the day
© 2013 LogiGear Corporation. All Rights Reserved

The Three “Holy Grails” of Test Design
Metaphor to depict three main steps in test design
Using "grail" to illustrate that there is no one perfect
solution, but that it matters to pay attention (to search)
About quality of tests, but most of all about scalability and
maintainability in BIG projects
Organization of tests into test modules
Right approach for each test module
Proper level of detail in the test specification
© 2013 LogiGear Corporation. All Rights Reserved

16
4/11/2013

Case for organizing tests in BIG projects
Can help keep the volume down
Isolate the complexities
Efficient and re-usable automation
Deal with changing requirements
For example: much of tested subject matter is not
system specific, but business specific
− a mortgage is a mortgage

© 2013 LogiGear Corporation. All Rights Reserved

What's the trick...

© 2013 LogiGear Corporation. All Rights Reserved

17
4/11/2013

What's the trick...
Have or acquire facilities to store and organize
you content
Edit your stuff
Decide where to put what
− assign and label the shelves

Put it there
If the organization is not sufficient anymore, add
to it or change it
© 2013 LogiGear Corporation. All Rights Reserved

Properties of a good Breakdown
Test modules are well differentiated and clear in
scope
Reflects the level of tests
Balanced in size and amount
Modules are mutually independent
Fit the priorities and planning of the project

© 2013 LogiGear Corporation. All Rights Reserved

18
4/11/2013

Breakdown Criteria
Straightforward Criteria
− Architecture of the system under test (client, server, protocol, sub
systems, components, modules, ...)
− Functionality (customers, finances, management information, ...)
− Kind of test (navigation flow, negative tests, response time, ...)
− Ambition level (smoke test, regression, aggressive, )

Additional Criteria
−
−
−
−
−

Stakeholders (like "Accounting", "Compliance", "HR", ...)
Complexity of the test (put complex tests in separate modules)
Technical aspects of execution (special hardware, multi-station, ...)
Overall project planning (availability of information, timelines, sprints, ...)
Risks involved (extra test modules for high risk areas)

© 2013 LogiGear Corporation. All Rights Reserved

Example breakdown
Tests of user interface
−
−
−

does function key F4 work
does listbox xyz the right values
is the tab order correct

Form Tests, do all the forms (dialogs, screens, pages) work:
−
−
−

can data be entered and is it stored well
is displayed data correct
split these from everything else

Function tests, do individual functions work
−

can I count the orders

Alternate paths in use cases
−

can I cancel a transaction

End-to-end tests
−
−

do all components of a system work well together in implementing the business processes
like enter sale order, then check inventory and accounting

Tests with specific automation needs
−

like multi station tests

Tests of non-UI functions
High ambition level tests (aggressive tests)
−

can I break the system under test

If in doubt: try high level first
© 2013 LogiGear Corporation. All Rights Reserved

19
4/11/2013

What is probably not a good design
Navigational and functional tests are mixed
− for example "over checking": a test of a premium calculation also
checks the existence of a window

You have to change all of them for every new release of
the system under test
All test modules have a similar design
Test modules are dependent on each other
You can’t start developing any test modules early in the
life cycle

© 2013 LogiGear Corporation. All Rights Reserved

Symptoms
Tediousness in the test and test automation
process
No sense of control
Complaining people
Unnecessary high test maintenance
− changes in the system under test impact many tests
− hard to understand which tests need to be modified

Difficulties in running any test
− teams start "debugging" tests

© 2013 LogiGear Corporation. All Rights Reserved

20
4/11/2013

Example of an application under test
Various item types
−
−
−
−
−
−

tests
actions
interface definitions
data sets
folders
...

Various operations
−
−
−
−

open
cut, copy, paste
check out
...

Various ways to initiate an operation
−
−
−
−
−
−
−
−

context menu, with or without accelerator key
main menu, with or without accelerator key
toolbar
short cut key
function key
drag and drop
double click
....

© 2013 LogiGear Corporation. All Rights Reserved

Defining some modules
Test modules for operations
−
−
−
−
−

primary and alternate paths
various values for fields like "comment" in check-in
paste in other projects
copy and paste various groups
not necessarily on each item type

Test modules for items
− address all item types at least once
− on each item type perform each operation
− not necessarily each variant of each operation

UI handling
− try for UI command if it starts the intended operation
− not necessarily on each item type or operation variant

Concurrency
−
−
−
−

try concurrency sensitive operations with multiple stations
in varying concurrency scenarios, with and without local "refreshes"
not necessarily each item type or operation variant
certainly not each UI command included
© 2013 LogiGear Corporation. All Rights Reserved

21
4/11/2013

Questions for Test Design
Does your organization make
something like a high level test
design?
If yes, how do you document it?

© 2013 LogiGear Corporation. All Rights Reserved

Case Study
Large IT provider
New version of one of their major web-sites
Test scope was user acceptance test (functional
acceptance)
− the users were the “business owners”

Development was off-shore

© 2013 LogiGear Corporation. All Rights Reserved

22
4/11/2013

Case Study
Test development was done separate from
automation
− time-line for test development: May – Oct
− time-line for automation (roughly): Jan – Feb

All tests were reviewed and approved by the
business owners
− acceptance was finished by the end of the test
development cycle

© 2013 LogiGear Corporation. All Rights Reserved

Example of a Test Development Plan
Nr
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Module
Portal Navigation, Audience
Portal Navigation, Search
Membership, registration
Portal Navigation, Category
Portal Navigation, Topic and Expert
Access Control
Portal Navigation, Task
Contact DSPP
Portal search
Membership, review and update
Program contact assignment
Company, registration
Catalog, view and query
Site map
Membership, affiliation
Learn about DSPP
Products and services
What's new
Company, life cycle
Specialized programs
Customer surveys
Software downloads
Newsletters
Internationalization and localization
Membership, life cycles
Collaboration, forums
Collaboration, blogs
Collaboration, mailing lists

Business Owner
Date to BO
Robyn Peterson
05 / 23
Ted Jones
05 / 27
Steve Shao
06 / 03
Ted Jones
06 / 08
Ted Jones
06 / 13
Mike Soderfeldt
06 / 17
Ted Jones
06 / 22
Ted Jones
06 / 27
Mike Soderfeldt
07 / 01
Steve Shao
07 / 05
Alan Lai
07 / 11
Steve Shao
07 / 14
Robyn Peterson
07 / 19
Ted Jones
07 / 25
Steve Shao
07 / 28
Ted Jones
08 / 01
Steve Shao, Robyn Peterson
08 / 08
Ted Jones
08 / 11
Steve Shao, Alan Lai
08 / 17
Ted Jones, Steve Shao
08 / 22
Ted Jones
08 / 29
Mike Soderfeldt
09 / 01
Ted Jones
09 / 06
Ted Jones
09 / 13
Steve Shao
09 / 19
Ted Jones
09 / 23
Mike Soderfeldt
09 / 28
Ted Jones
10 / 03
© 2013 LogiGear Corporation. All Rights Reserved

23
4/11/2013

Review Process with Stake Holders
START

Test Team sends draft
Module to Stake Holder

Stake Holder reviews:
- coverage
- correctness

changes needed?

no

Stake Holder returns
notice of approval

yes

Test Team receives and
processes notes

Stake Holder returns
notes:
- additions
- corrections

Test Team marks the
Module as "Final"

END

© 2013 LogiGear Corporation. All Rights Reserved

Case Study, Results
All tests were developed and reviewed on schedule
− many notes and questions during test development phase

The automation was 100% of the tests
− all actions were automated, thus automating all test modules

The test development took an estimated 18 person
months
− one on-shore resource, two off-shore resources

The automation took between one and two months
− focused on actions
− most time was spent in handling changes in the interface (layout of pages
etc)

© 2013 LogiGear Corporation. All Rights Reserved

24
4/11/2013

Case: The French Director
Mid size company
Struggling under high pressure
Testing of their main product, standard financial
software
Control and priority main issue
Unfamiliar business culture
Main instrument: module break down
© 2013 LogiGear Corporation. All Rights Reserved

Test Modules versus Test Cases
The test module is a bigger unit in the test design
− easier to identify
− a chapter rather than a paragraph
− easier to plan and manage, as a product (can be treated as part of product
backlog in scrum projects)

Better flow of execution
− each test case can set up for the next one
− keep test modules independent, test cases can be dependent

Test cases become creative output, rather than stifling input
− avoids having to define all test cases at once early in the process

Clear scope helps to identify cases, actions and checks
− using "test objectives" to further detail scope
− had a significant effect on maintainability

© 2013 LogiGear Corporation. All Rights Reserved

25
4/11/2013

"Thou Shall Not Debug Tests..."
Large and complex test projects can be hard to "get to
run"
If they are however, start with taking a good look again at
your test design...
Rule of thumb: don't debug tests. If tests don't run
smoothly, make sure:
− lower level tests have been successfully executed first -> UI flow in the AUT
is stable
− actions and interface definitions have been tested sufficiently with their own
test modules -> automation can be trusted
− are you test modules not too long and complex?

© 2013 LogiGear Corporation. All Rights Reserved

Grail 2: Approach per Test Module
Plan the test module:
− when to develop: is enough specification available
− when to execute: make sure the functionality at action level is welltested and working already

Process:
−
−
−
−

do an intake: understand what is needed and devise an approach
analyze of requirements
formulate "test objectives"
create "test cases"

Identify stakeholders and their involvement:
− users, subject matter experts
− developers
− auditors

Choose testing techniques if applicable:
− boundary analysis, decision tables, transition diagrams, soap opera
testing, ...
© 2013 LogiGear Corporation. All Rights Reserved

26
4/11/2013

Eye on the ball, Scope
Always know the scope of the test module
The scope should be unambiguous
The scope determines many things:
−
−
−
−

what the test objectives are
which test cases to expect
what level of actions to use
what the checks are about and which events should
generate a warning or error (if a “lower” functionality is
wrong)

© 2013 LogiGear Corporation. All Rights Reserved

What I have seen not work
"Over-Checking": having checks that don't fit the scope of
the test
Forcing data driven: making all tests data driven
(variables, data files) without clear reason
Combinatorial explosions: test all ... for all ... in all ...
All actions high level (or all actions low level)
Many tests for forms and dialogs, little tests for business
processes
Abundance of irrelevant comments, and lack of relevant
comments
© 2013 LogiGear Corporation. All Rights Reserved

27
4/11/2013

Detail out the scope with test objectives
...
TO-3.51 The exit date must be after the entry date
...

test objective

TO-3.51

enter employment
check error message

Bill Goodfellow 2002-10-02
2002-10-01
The exit date must be after the entry date.

name

entry date

exit date

© 2013 LogiGear Corporation. All Rights Reserved

Examples of Testing Techniques
Equivalence class partitioning
− any age between 18 and 65
Boundary condition analysis
− try 17, 18, 19 and 64, 65, 66
Error guessing
− try Cécile Schäfer to test sorting of
a name list
Exploratory
− "Exploratory testing is
simultaneous learning, test design,
and test execution", James Bach,
www.satisfice.com
Error seeding
− deliberately injecting faults in a test
version of the system, to see if the
tests catch them
− handle with care, don't let the bugs
get into the production version

Decision tables
− define possible situations and the
expected responses of the system
under test
State transition diagrams
− identify "states" of the system, and
have your tests go through each
transition between states at least
once
Jungle Testing
− focus on unexpected situations,
like hacking attacks
Soap Opera Testing
− describe typical situations and
scenarios in the style of episodes
of a soap opera, with fixed
characters
− high density of events,
exaggerated
− make sure the system under test
can still handle these
© 2013 LogiGear Corporation. All Rights Reserved

28
4/11/2013

"Jungle Testing"
Expect the unexpected
−
−
−
−

unexpected requests
unexpected situations (often data oriented)
deliberate attacks
how does a generic design respond to a specific unexpected event?

Difference in thinking
− coding bug: implementation is different from what was intended/specified
− jungle bug: system does not respond well to an unexpected situation

To address
−
−
−
−
−
−
−

study the matter (common hack attacks, ...)
make a risk analysis
make time to discuss about it (analysis, brainstorm)
involve people who can know
use "exploratory testing" (see James Bach's work on this)
use an agile approach for test development
applies equally to testing, requirements, design
• testing should focus on the specific attacks

− consider randomized testing, like "monkey" testing
© 2013 LogiGear Corporation. All Rights Reserved

Soap Opera Testing
Informal scenario technique to invite subject-matter
experiences into the tests, and efficiently address multiple
objectives
Using a recurring theme, with “episodes”
About “real life”
But condensed
And more extreme
Typically created with a high involvement of end-users
and/or subject-matter experts
© 2013 LogiGear Corporation. All Rights Reserved

29
4/11/2013

Lisa Crispin: Disorder Depot . . .
There are 20 preorders for George W. Bush action
figures in "Enterprise", the ERP system, awaiting the
receipt of the items in the warehouse.
Finally, the great day arrives, and Jane at the warehouse
receives 100 of the action figures as available inventory
against the purchase order. She updates the item record
in Enterprise to show it is no longer a preorder.
Some time passes, during which the Enterprise
background workflow to release preorders runs. The 20
orders are pick-released and sent down to the warehouse.
Source: Hans Buwalda, Soap Opera Testing (article), Better Software Magazine, February 2005
© 2013 LogiGear Corporation. All Rights Reserved

Lisa Crispin: Disorder Depot . . .
Then Joe loses control of his forklift and accidentally
drives it into the shelf containing the Bush action figures.
All appear to be shredded to bits. Jane, horrified,
removes all 100 items from available inventory with a
miscellaneous issue. Meanwhile, more orders for this very
popular item have come in to Enterprise.
Sorting through the rubble, Jane and Joe find that 14 of
the action figures have survived intact in their boxes.
Jane adds them back into available inventory with a
miscellaneous receipt.

© 2013 LogiGear Corporation. All Rights Reserved

30
4/11/2013

Lisa Crispin: Disorder Depot . . .
This scenario tests
• Preorder process
• PO receipt process
• Miscellaneous receipt and issue
• Backorder process
• Pick-release process
• Preorder release process
• Warehouse cancels

© 2013 LogiGear Corporation. All Rights Reserved

Vary your tests?
Automated tests have a tendency to be rigid, and
predictable
Real-world situations are not necessarily
predictable
Whenever possible try to vary:
− with select other data cases that still fit the goal of tests
− with randomized behavior of the test

© 2013 LogiGear Corporation. All Rights Reserved

31
4/11/2013

Generation and randomization techniques
Model-based
− use models of the system under test to create tests
− see: Harry Robinson, www.model-based-testing.org, and Hans Buwalda, Better
Software, March 2003

Data driven testing
− apply one test scenario to multiple data elements
− either coming from a file or produce by an automation

"Monkey testing"
−
−
−
−

use automation to generate random data or behavior
"smart monkeys" will follow typical user behavior, most helpful in efficiency
"dumb monkeys" are more purely random, may find more unexpected issues
long simulations can expose bugs traditional tests won't find

Extended Random Regression
−
−
−
−

have a large database of tests
randomly select and run them, for a very long time
this will expose bugs otherwise hidden
see Cem Kaner e.a.: "High Volume Test Automation", StarEast 2004

© 2013 LogiGear Corporation. All Rights Reserved

Data Driven Testing
Separate test logic from the data
Possible origins for the data:
−
−
−
−

earlier steps in the test
data table
randomizer, or other formula
external sources, like a database query

Use "variables" as placeholders in the test case,
instead of hard values
Data driven is powerful, but use modestly:
− value cannot be known at test time, or changes over time
− having many data variations is meaningful for the test
© 2013 LogiGear Corporation. All Rights Reserved

32
4/11/2013

Variables and expressions with keywords
TEST CASE

TC 02
car

last name

car

John
John

Doe
Doe

Chevvy Volt
Chevvy Volt

car

check quantity

>> volts

first name

rent car
rent car

available

Chevvy Volt

get quantity

Rent some more cars

expected

Chevvy Volt

# volts - 2

This test does not need an absolute number for the
available cars, just wants to see if a stock is updated
As a convention we denote an assignment with ">>"
The "#" indicates an expression
© 2013 LogiGear Corporation. All Rights Reserved

Data driven testing with keywords
TEST CASE

TC 03

Check stocks

use data set

/cars
car

get quantity

available

check quantity

>> quantity
last name

car

# first

# last

# car

car

rent car

# car
first name

expected

# car

DATA SET

cars

car

data set

first

last

value

Chevvy Volt
Ford Escape
Chrysler 300
Buick Verano
BMW 750
Toyota Corolla

John
Mary
Jane
Tom
Henry
Vivian

Doe
Kane
Collins
Anderson
Smyth
Major

40000
22500
29000
23000
87000
16000

# quantity - 1

repeat for data set

The test lines will be repeated for each row in the data set
The values represented by "car", "first" and "last" come
from the selected row of the data set
© 2013 LogiGear Corporation. All Rights Reserved

33
4/11/2013

Combinations
Input values
− determine equivalence classes of values for a variable or field
− for each class pick a value (or randomize)

Options, settings
Configurations
− operating systems, operating system versions and flavors
• Windows service packs, Linux distributions

−
−
−
−

browsers, browser versions
protocol stacks (IPv4, IPv6, USB, ...)
processors
DBMS's

Combinations of all of the above
Trying all combinations will spin out of control quickly
© 2013 LogiGear Corporation. All Rights Reserved

Pairwise versus exhaustive testing
Group values of variables in pairs (or tuples with more than 2)
Each pair (tuple) should occur in the test at least once
− maybe not in every run, but at least once before you assume "done"
− consider to go through combinations round-robin, for example pick a different
combination every time you run a build acceptance test
− in a NASA study:
•
•
•

67 percent of failures triggered by a single value
93 percent by two-way combinations, and
98 percent by three-way combinations

Example, configurations
− operating system: Windows XP,
Apple OS X, Red Hat Enterprise Linux
− browser: Internet Explorer, Firefox, Chrome
− processor: Intel, AMD
− database: MySQL, Sybase, Oracle
− 72 combinations possible, to test each pair: 10 tests

Source: PRACTICAL COMBINATORIAL TESTING, D. Richard Kuhn, Raghu N.
Kacker, Yu Lei, NIST Special Publication 800-142, October, 2010

Example of tools:
− ACTS from NIST, PICT from Microsoft, AllPairs from James Bach (Perl)
− for a longer list see: www.pairwise.org

These techniques and tool are supportive only. Often priorities
between platforms and values can drive more informed selection Rights Reserved
© 2013 LogiGear Corporation. All

34
4/11/2013

Grail 3: Specification Level, choosing actions
Scope of the test determines the specification level
As high level as appropriate, as little arguments as
possible
− Use default values for non-relevant arguments

Clear names (usually verb + noun usually works well)
− to standardize action names: standardize both the verbs and the nouns, so
"check customer" versus "verify client" (or vice versa)
− tests are not C++ code: avoid "technical habits", like mixed case and (worse)
underlines

Manage the Actions
Document the Actions
By-product of the test design
© 2013 LogiGear Corporation. All Rights Reserved

Case: American Bank
Project for a new teller system
Large, state of the art
Many system releases, many adjustments
Need for very high level of automation
Over 1 million test lines, in over 650 test modules
Initially little attention paid to "holy grails"
− UI and functional tests in the same modules
− virtually un-maintainable, came close to killing the project
− test design forced upon the team by a powerful stakeholder
who did not care much for methods...

Emergency re-organization of the test modules
− after system changes the tests would run again within a day
© 2013 LogiGear Corporation. All Rights Reserved

35
4/11/2013

Example of using actions
In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function
• the "#" means an expression, in this case a variable
• the ">>" assign to a variable for use later on in the test
key

key navigate
key navigate

F7
3
page tab

locate page tab

Scan Criteria
w indow

wait for controls loaded

search
text

check breadcrumb

general functions > search
w indow

scan direction

Backward

w indow

enter value

control

search

select

value

control

value

search

control

search

click

business date match # bus date

source

go

w indow

wait for controls loaded

search results
w indow

get

control

search results

sequence number

variable

>> seq num
© 2013 LogiGear Corporation. All Rights Reserved

Example of using actions
In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function
• the "#" means an expression, in this case a variable
• the ">>" assign to a variable for use later on in the test

variable

get sequence number

>> seq num

© 2013 LogiGear Corporation. All Rights Reserved

36
4/11/2013

Mid level actions
Most tests will have low level and high level actions
− low level: generic operations, know the interface, don't know the functionality
• examples: "selection menu item", "expand tree node", ...

− high level: business oriented operations, know the functionality, don't know
the interface
• examples: "enter purchase order", "check inventory of article"

For complex forms (dialog) with many input fields
consider using "mid level" actions
− an argument for each field
− for use in high level actions

enter customer
enter address fields

Examples of mid-level actions:
− "enter address fields"
− "check address fields"

...

enter

select

set

...

© 2013 LogiGear Corporation. All Rights Reserved

Mapping an interface entity (like a window)
INTERFACE ENTITY balance inquiry
interface entity setting title
Balance inquiry
ta name

interface element
interface element

ta class

label

last name
first name
client id

text
text
text

Last name:
First name (optional):
Client id (optional):

ta name

interface element
interface element
interface element

ta class

caption

View Balance
Close

ta name

interface element

view balance button
close
button
ta class

global pos

balance

label

label 5

An interface mapping will map windows and controls to names
When the interface of an application changes, you only have to
update this in one place
The interface mapping is a key step in your automation success,
allocate time to design it well
© 2013 LogiGear Corporation. All Rights Reserved

37
4/11/2013

Some Tips to Get Stable Automation
Make the system under test automation-friendly
Use "active" timing
Test your automation
Use automation to identify differences between
versions of the system under test
Keep an eye on the test design

© 2013 LogiGear Corporation. All Rights Reserved

Automation-friendly design: hidden properties
Look for properties a human user can't see, but a test tool can
This approach is a must-do for speedier and more stable automation
−
−
−
−

interface mapping is often bottleneck, and source of maintenance problems
with predefined identifying property values an interface map can be created without "spy" tools
not sensitive to changes in the system under test
not sensitive to languages and localizations

Examples:
−
−
−

"id" attribute for HTML elements
"name" field for Java controls
"AccessibleName" property in .Net controls (see below)

© 2013 LogiGear Corporation. All Rights Reserved

38
4/11/2013

Active Timing
Passive timing
− wait a set amount of time
− in large scale testing, try to avoid passive timing altogether:
• if wait too short, test will be interrupted
• if wait too long, time is wasted

Active timing
− wait for a measurable event
− usually the wait is up to a, generous, maximum time
− common example: wait for a window or control to appear (usually the test tool
will do this for you)

Even if not obvious, find something to wait for...
Involve developers if needed
− relatively easy in an agile team, but also in traditional projects, give this
priority

If using a waiting loop
− make sure to use a "sleep" function in each cycle that frees up the processor
(giving the AUT time to respond)
− wait for an end time, rather then a set amount of cycles
© 2013 LogiGear Corporation. All Rights Reserved

Active Timing, your situation
How much passive timing do you have in your
scripts?
If you're not sure, find out...
... and let me know

"First action I took upon my return was to evaluate the percentage of
passive time in our code and found passive time 68% versus active
time 32%. Needless to say our automation test cases were very
expensive time operations and now I know why..."
Raed Atawneh, 2012 (extract)

© 2013 LogiGear Corporation. All Rights Reserved

39
4/11/2013

Things to wait for...
Wait for a last control or elements to load
− developers can help knowing which one that is

Non-UI criteria
− API function
− existence of a file

Criteria added in development specifically for this purpose, like:
− "disabling" big slow controls (like lists or trees) until they're done loading
− API functions or UI window or control properties

Use a "delta" approach:
− every wait cycle, test if there was a change; if no change, assume that the
loading time is over:
− examples of changes:
• the controls on a window
• count of items in a list
• size a file (like a log file)

© 2013 LogiGear Corporation. All Rights Reserved

Alternatives to UI automation ("non-GUI")
A GUI (Graphical User Interface) is only one example of an interface
for interaction with a system under test
Examples
−
−
−
−
−
−
−
−
−

HTTP and XML based interfaces, like REST
application programming interfaces (API’s)
embedded software
protocols
files, batches
databases
command line interfaces (CLI’s)
multi-media
mobile devices

In many cases non-GUI automation is used since there simply is not
GUI, but it can also often speed things up:
−
−

tends to be more straightforward technically, little effort needed to build up or maintain
once it works, it tends to work much faster and more stably than GUI automation

In BIG testing projects routinely:
−
−

identify which non-GUI alternatives are available
as part of test planning: identify which tests qualify for non-GUI automation

© 2013 LogiGear Corporation. All Rights Reserved

40
4/11/2013

Technical Complexity
Technology is another dimension that can make a project
"complex"
Examples:
− graphics, charts, 3D, ...
− hard to access systems, like embedded software, iOS, Flash,
dedicated hardware
− difficult protocols, like SS7, transactions servers

Approach: isolate the technical problems
−
−
−
−

embed in functions and actions
let experts look at them
tackle early in a project, since impact is large
once resolved, no longer center stage

© 2013 LogiGear Corporation. All Rights Reserved

The importance of innovation
Large and complex testing projects pose many challenges
Initial discussion of approach is a key requisite
−
−
−
−
−
−

thinking before doing
tackling technologies
agreeing on methods and practices
who does what, who needs to be involved
high level test design
debate the problems, not just the solutions

However, also plan for continuous improvements
−
−
−
−

this is at the heart of agile thinking, and it applies very much to big testing
never stop thinking, "there is always one more trick"
share the tricks, other teams may like them too
improvements can apply to test design, to automation techniques, or even to
how to organize the work

© 2013 LogiGear Corporation. All Rights Reserved

41
4/11/2013

Tools that can help manage BIG projects
Application Lifecycle Management (ALM)
−
−
−
−

abundant now, mainly on the wings of agile
very good for control, team cooperation, and traceability
often relate to IDE's (like Microsoft TFS and Visual Studio)
examples: Rally, Jira, TFS

Test Managers
−
−
−

as separate tools on their way out
morphing into or replaced by ALM options
examples: HP Quality Center, Microsoft Test Manager

Test development and automation tools
−
−

develop and/or automate tests
• these are not the same, automation tools are not always so good for test development
examples are HP Quick Test Pro, Borland Silk, Selenium, FitNesse, Microsoft Coded UI, and LogiGear's
TestArchitect and TestArchitect for Visual Studio (our own products)

Build tools
−
−
−
−

succeed the traditional "make" tools
in particular "continuous build" tools combine "make" functionality with source control systems to rebuild
components that have changed, either continuously or on set times, like nightly
can very well also run related tests (unit and functional), and act on the results (stop build, report, etc)
examples: Hudson, Jenkins, TFS

Bug trackers
−
−
−

not only register issues, but also facilitate their follow up, with workflow features
often also part of other tools, and tend to get absorbed now by the ALMs
Examples: BugZilla, Mantis, Trac

© 2013 LogiGear Corporation. All Rights Reserved

Tooling and Traceability
ALM, IDE,
Project Mgr,
Req Mgr

Test Development Tool
Automation Tool

Execution Manager
Continous Build Tool

Issue Tracker
ALM

Execution Result

Bug, issue

Test Module
Reference item
(ALM item, req,
code module, ...)

Test Objective

Test Case

Testing
Trace back

© 2013 LogiGear Corporation. All Rights Reserved

42
4/11/2013

Test Execution
Have an explicit approach for when and how to execute
which tests
Having a good high level test design will help to organize
this
Execution can be selective or integral
− unit tests are typically executed selectively, possibly automatically
based on code changes in a system like SVN or TFS
− for functional tests, decisions are needed:
• selective execution will be quicker and more efficient
• integral execution may catch more issues ("bonus bugs")
• generally extensive functional test execution will be related to releases, rather
than code check ins

− the ability to run "big testing" efficiently may determine how much can
be done

© 2013 LogiGear Corporation. All Rights Reserved

Environments, configurations
Many factors can influence details of automation
−
−
−
−

language, localization
hardware
version of the system under test
system components, like OS or browser

Test design can reflect these
− certain test modules are more general
− others are specific, for example for a language

But for tests that do not care about the differences, the
automation just needs to "deal" with them
− shield them from the tests

© 2013 LogiGear Corporation. All Rights Reserved

43
4/11/2013

"Variations"
Capture variations of the system under test in the actions and interface
definitions, rather than in the tests (unless relevant there).
Can be a feature in a test playback tool, or something you do with a global
variable or setting.

. . .
Actions, Interface Definitions
"Master Switch"

Variation

Variation

Variation

© 2013 LogiGear Corporation. All Rights Reserved

Possible set up of variations
linked variation

keyworded variation

Specify for example in a dialog when you start an execution:

© 2013 LogiGear Corporation. All Rights Reserved

44
4/11/2013

Test Environments
Physical
•
•
•
•

hardware
infrastructure
location
...

• costs money
• can be scarce

Software
•
•
•
•

programs
data models
protocols
...

• configurations

Data
• initial data
• parameters / tables
• ...

• availability
• manageability
© 2013 LogiGear Corporation. All Rights Reserved

Dealing with data
Constructed data is easier to manage
− can use automation to generate it, and to enter it in the environment
− result of test analysis and design, reflecting "interesting" situations
− however, less "surprises": real life situations which were not foreseen

Real-world data is challenging to organize
− make it a project, or task, in itself
− make absolutely sure to deal with privacy, security and legal aspects
appropriately
• study this, ask advice
• apply appropriate "scrubbing"

Consider using automation to select data for a test
− set criteria ("need a male older than 50, married, living in Denver"),
query for matching cases, and select one randomly (if possible a
different one each run)
− this approach will introduce variation and unexpectedness, making
automated tests stronger and more interesting
© 2013 LogiGear Corporation. All Rights Reserved

45
4/11/2013

Unattended testing...
When a test cannot pass, it can be:
− a difference between expected and recorded values or behavior, as a result
of a check designed by the tester: this is a fail
− the automation encounters a problem, like a window or control doesn't show,
that is not part of a check: this is an error

An error can disrupt the test flow, and you may want to catch and
handle it properly:
− skip smaller or larger parts of the ongoing test
− bring the system back in a known state (typically: close any open windows,
go to the main screen)
− make sure the report clearly indicates these kind of problems, to avoid false
positives
− example "on error action" that executes a predefined action that will do
recovery

However, better is to avoid these situations
− lots of efforts needed for unattended testing should raise questions about test
design or quality of the automation ("thou shall not debug tests")
© 2013 LogiGear Corporation. All Rights Reserved

"Known bug" problem
Not uncommon in large scale systems
− typically related to a version of the system under test

A known bug may:
− generate fails you want to ignore, also in statistics
− throw off automation

If many known bug situation occur, take another look at
your high level test design
One possible workaround, a "known bug" action:
− other alternatives: conditionally ignore steps or single check points
version

known bug
...
end known bug

1.1

© 2013 LogiGear Corporation. All Rights Reserved

46
4/11/2013

Virtualization
Virtual machines rather than physical machines
− allow "guest" systems to operate on a "host" system
− host can be Windows, Linux, etc, but also a specialized "hypervisor"
− the hypervisor can be "hosted" or "bare metal"

Main providers:
−
−
−
−

VMWare: ESX and ESXi
Microsoft: Hyper-V
Oracle/Sun: Virtual Box
Citrix: Xen (open source)

Hardware support gets common now
− processor, chipset, i/o
− Like Intel's i7/Xeon

For most testing purposes you need virtual clients, not virtual servers
− most offerings in the market currently target virtual servers, particularly data centers

Virtual clients will become more mainstream with the coming of VM's as part
of regular operating systems
− Windows 8: Hyper-V
− Linux: KVM
© 2013 LogiGear Corporation. All Rights Reserved

Virtualization, a testers dream...
In particular for functional testing
Much easier to define and create needed configurations
− you basically just need storage
− managing this is your next challenge

One stored configuration can be re-used over and over again
The VM can always start "fresh", in particular with
− fresh base data (either server or client)
− specified state, for example to repeat a particular problematic automation
situation

Can take "snap shots" of situations, for analysis of problems
Can use automation itself to select and start/stop suitable VM's
− for example using actions for this
− or letting an overnight or continuous build take care of this

© 2013 LogiGear Corporation. All Rights Reserved

47
4/11/2013

Virtualization, bad dream?
Performance, response times, capacities
Virtual machine latency can add timing problems
− see next slide
− can be derailing in big test runs

Management of images
− images can be large, and difficult to store and move around
• there can be many, with numbers growing combinatorial style
• configuration in the VM can have an impact, like fixed/growing virtual disks

− distinguish between managed configurations and sandboxes
− define ownership, organize it
− IT may be the one giving out (running) VM's, restricting your flexibility

Managing running tests in virtual machines can take additional efforts
on top of managing the VM's themselves
− with the luxury of having VM's the number of executing machines can
increase rapidly
− one approach: let longer running tests report their progress to a central
monitoring service (various tools have features for this)
© 2013 LogiGear Corporation. All Rights Reserved

Virtualization: "time is relative"
Consider this waiting time loop, typical for a test script:
− endTime = currentTime + maxWait
− while not endTime, wait in 100 millisecond intervals

When the physical machine overloads VM's can get slow or have
drop outs, and endTime may pass not due to AUT latency
− GetLocalTime will suffer from the latency
− GetTickCount is probably better, but known for being unreliable on VM's

Therefore tests that run smooth on physical machines, may not
consistently do so on VM's. The timing problems are not easy to
predict
Possible approaches:
− in general: be generous with maximum wait times if you can
− don't put too many virtual machines on a physical box
− consider a compensation algorithm, for example using both tick count and clock time

© 2013 LogiGear Corporation. All Rights Reserved

48
4/11/2013

Virtual machines, capacity

We started regression with 140 VMs.
Very slow performance of
Citrix VM clients.

Key to pricing is number of VM's that can run in parallel
on a physical machine
An automated test execution will typically keep a VM
more busy than human use
Factors in determining VM/PM ratio:
−
−
−
−

memory, for guest OS, AUT, test tooling
storage devices (physical devices, not disk images)
processors, processor cores
specific hardware support (becoming more common)
• processor, chipset, I/O

© 2013 LogiGear Corporation. All Rights Reserved

Building up virtualization
Pay attention to pricing:
− beefed up hardware can increase VM's/box ratio, but at a price
− software can be expensive depending on features, that you may not
need

In a large organization, virtual machines are probably available
− make sure to allocate timely (which can be long before you get there
with your sprints)
− keep in mind the capacity requirements

Logical and physical management
− which images, the wealth of possible images can quickly become
hard to see forest through the trees
− physical management of infrastructure is beyond this tutorial

Minimum requirement: snapshots/images
− freeware versions don't always carry this feature
− allow to set up: OS, environment, AUT, tooling, but also: data, states

© 2013 LogiGear Corporation. All Rights Reserved

49
4/11/2013

Infrastructure
For large scale test execution this needs attention
− physical infrastructure, but also how to use it

Also consider managing infrastructure and test
execution as a separate task
− in or out of the team
− avoid slowing down development (of system, test and/or
automation)

© 2013 LogiGear Corporation. All Rights Reserved

Remote execution, servers
Allowing execution separately from the machines the testers and
automation engineers are working on increases scalability
Large scale text execution, in particular with VM's, like to have:
−
−

lots of processing power, lots of cores
lots of memory

Test execution tends to care less about:
−
−

storage
networking

Test execution facilities tend to be a bottle neck very quickly in big
testing projects
−

the teams can use whatever they can get

First step up: give team members a second machine
Second step up: use servers, users coordinate their use of them
Third step up: major infrastructures with organized allocation
© 2013 LogiGear Corporation. All Rights Reserved

50
4/11/2013

Tower Servers

Smaller shops (smaller companies, departments)
Affordable, simple, first step up from clients execution
Not very scalable when the projects get larger
© 2013 LogiGear Corporation. All Rights Reserved

Rack Servers

Well scalable
Pricing not unlike tower servers
Tend to need more mature IT expertise
© 2013 LogiGear Corporation. All Rights Reserved

51
4/11/2013

Server Blades

Big league infrastructure, high density, very scalable
Tends to be pricey, use when space and energy matters
Usually out of sight for you and your team
© 2013 LogiGear Corporation. All Rights Reserved

Cloud
Cloud can be target of testing
− normal tests, plus cloud specific tests
•

functional, load, response times

− from multiple locations
− moving production through data centers

Cloud can be host of test execution
− considerations can be economical or organizational
− providers offer imaging facilities, similar to virtual machines
− make sure machines are rented and returned efficiently

Public cloud providers like EC2 offer API's, so your automation can
automatically allocate and release them
− be careful, software bugs can have costing consequences
− for example, consider having a second automation process to double-check cloud
machines have been released after a set time

Note: public cloud is not taking of as fast as expected, cloud services,
and private clouds, taking of much faster
© 2013 LogiGear Corporation. All Rights Reserved
(Xinhua Photo)

52
4/11/2013

Cloud Providers

Source: Jack of All Clouds, January 2011
http://www.jackofallclouds.com/2011/01/state-of-the-cloud-january-201/
© 2013 LogiGear Corporation. All Rights Reserved

Cloud growth

source: IDC forecast, 2010

Growth of public clouds not as big as expected
Cost benefits not necessarily convincing
− low startup cost, but long ongoing cost

See also: news.cnet.com/8301-13556_3-20063361-61.html
© 2013 LogiGear Corporation. All Rights Reserved

53
4/11/2013

Cloud, example pricing, hourly rates
Linux

Small
Large
Extra Large

Windows

0.085
0.34
0.68

0.12
0.48
0.96

1.7 GB, 1 core (32 bits)
7.5 GB, 4 cores
15 GB, 8 cores

0.50
1.00
2.00

0.62
1.24
2.48

17.1 GB, 6.5 core
34.2 GB, 13 cores
68.4 GB, 26 cores

0.17
0.68

0.29
1.16

1.7 GB, 5 core (32 bits)
7 GB, 20 cores

High memory

Extra Large
Double Extra Large
Quadruple Extra Large
High CPU

Medium
Extra Large

Source: Amazon EC2 (my interpretation, actual prices may vary)
© 2013 LogiGear Corporation. All Rights Reserved

Cloud, example economy
small

Windows
buy (estimate)
hours to break even
months (24 / 7)

large

extra

$0.12
$300
2,500
3.4

$0.48
$650
1,354
1.8

$0.96
$900
938
1.3

Not counting possible use of VM's within the buy option
Also not counting: additional cost of ownership elements for owning or
cloud (like IT management, contract and usage management)
Impressions:
− cloud could fit well for bursty testing needs, which is often the case
− for full continuous, or very frequent, testing: consider buying
− hybrid models may fit many big-testing situations: own a base capacity, rent
more during peak use periods
© 2013 LogiGear Corporation. All Rights Reserved

54
4/11/2013

Data centers can go down

However, disruption could have been minimized by using multiple data centers
© 2013 LogiGear Corporation. All Rights Reserved

Data centers can go down

This time, it did involve multiple data centers . . .
© 2013 LogiGear Corporation. All Rights Reserved

55
4/11/2013

Data centers can go down

Service providers can occasionally go down too
© 2013 LogiGear Corporation. All Rights Reserved

Cloud, usage for special testing needs
Multi-region testing
− Amazon for example has several regions
•
•
•
•
•

US East, Northern Virginia
US West, Oregon, Northern California
EU, Ireland
Asia Pacific, Singapore, Tokyo
South America, Sao Paulo

− be careful that data transfers between regions costs money
(0.01/GB)

Load generation
− example: "JMeter In The Cloud"
•
•
•
•

based on the JMeter load test tool
uses Amazon AMI's for the slave machines
allows to distribute the AMI's in the different regions of Amazon
see more here:

aws.amazon.com/amis/jmeter-in-the-cloud-a-cloud-based-load-testing-environment

© 2013 LogiGear Corporation. All Rights Reserved

56
4/11/2013

Questions for Infrastructure
What kind of infrastructure does
your organization use for
testing?
What is the role of
virtualization, now or in the
future?
Are you using a private or a
public cloud for testing?

© 2013 LogiGear Corporation. All Rights Reserved

Example of a cloud system under test

source: Windows Azure reference platform
© 2013 LogiGear Corporation. All Rights Reserved

57
4/11/2013

Approaches
Automation does not have to be black box
− for very big systems, a separate black box automation effort may not
be efficient
− and building and keeping lab situations might be cumbersome
− some simple hooks can greatly help already
− remember... this is about automation, not test design.

Make testability part of requirements and architecture
− a key question should not just be "how do I design this", but "how do I
test this" (test design, automation)
− some cloud/web systems are changed frequently, and tested "live"
• "Testing in Production (TiP)"

− allow redirection of some or all traffic through another version of a
component or layer

Example: reverse proxy's enabling A/B testing
see also: Ken Johnston's chapter in the book of Dorothy Graham and Mark Fewster,
and his keynote at StarWest 2012

© 2013 LogiGear Corporation. All Rights Reserved

A/B testing with a reverse proxy
A

A

Users

Reverse
Proxy

A

B

Servers

B

Watch your test design, easy to drown in technical solutions only
B could be a real-life user or also a keyword driven test machine
A/B testing means part of traffic is routed through a different
server or component (see if it works, and/or how users react)
A similar strategy could be done at any component level

A

current

B

new

© 2013 LogiGear Corporation. All Rights Reserved

58
4/11/2013

Organization
Much of the success is gained or lost in how you organize the
process
−
−
−
−

part of the teams
who does test design
who does automation
what to outsource, what to keep in-house

Write a plan of approach for the test development and automation
−
−
−
−
−
−

scope, assumptions, risks, planning
methods, best practices
tools, technologies, architecture
stake holders, including roles and processes for input and approvals
team
...
Test design is a skill . . .

Assemble the right resources
− testers, lead testers
− automation engineer(s)
− managers, ambassadors, ...

Automation is a skill . . .
Management is a skill . . .

. . . and those skills are
different . . .

© 2013 LogiGear Corporation. All Rights Reserved

Industrial Organization
Large scale testing can move from a "design" to a
"production" focus
− mostly applies to test execution, but also seen for test development
− this not black and white, both paradigms can occur in the same projects
− this is often more easy to outsource than development

A production organization is different a development
organization
−
−
−
−
−

this is not unique for software
different professional culture
emphasis more on delivery and scale, "thinking big"
discipline rather than creativity, "get stuff done"
activities are like planning, control, logistics, information

© 2013 LogiGear Corporation. All Rights Reserved

59
4/11/2013

Task in "production" (test execution)
Keeping the tests running
Allocating resources
Respond to hick-ups
Analyze and address automation issues
Address fails or other testing outcomes
− including dealing with "known bugs"
− part of a bigger team

© 2013 LogiGear Corporation. All Rights Reserved

Stake Holders
EXTERNAL

INTERNAL

Customers

Management

Vendors
Quality Assurance

Test Development

Government
Agencies
Publicity

Technology/
Infrastructure

Test Automation

End User
Departments
Marketing/
Sales

System
Development
After Sales/
Help Desk

Production

© 2013 LogiGear Corporation. All Rights Reserved

60
4/11/2013

Team roles, examples
Test development
Automation
Planning and managing the test runs
Managing environments
Managing infrastructure
Dealing with stakeholders
Analysis of results, and follow up
Reporting
© 2013 LogiGear Corporation. All Rights Reserved

Test Development and Automation in sprints
Agile life cycle

product
owner

team

Product
Backlog

Sprint

Test Module
Definition
(optional)

prod owner
& team

Products

Test Module Development

Test re-use

Interface Definition

Automation re-use

Action Automation
Test Execution

Test development
User stories
Documentation
Domain understanding

Acceptance Criteria
PO Questions
Situations
Relations

Main Level Test Modules
Interaction Test Modules
Cross over Test Modules
© 2013 LogiGear Corporation. All Rights Reserved

61
4/11/2013

Test automation in sprints
Try keep the main test modules at a similar level as the
user stories and acceptance criteria
Aim for "sprint + zero", meaning: try to get test
development and automation "done" in the same sprint,
not the next one
− next one means work clutters up, part of team is not working on the
same sprint, work is done double (manually and automated), ...

Make sure you can do the interface mapping by hand
(using developer provided identifications)
− can do earlier, before UI is finalized, and
− recording of actions (not tests) will go better

Also plan for additional test modules:
− low-level testing of the interaction with the system under test (like
UI's)
− crossing over to other parts of the system under test
© 2013 LogiGear Corporation. All Rights Reserved

Fitting in sprints
Agree on the approach:
− questions like does "done" include tests developed and automated?
− do we see testing and automation as distinguishable tasks and
skillsets
− is testability a requirement for the software

Create good starting conditions for a sprint:
− automation technology available (like hooks, calling functions, etc)
− how to deal with data and environments
− understanding of subject matter, testing, automation, etc

Make testing and automation part of the evaluations
Address tests and automation also in hardening sprints
Just like for development, use discussions with the team
and product owners to deepen understanding:
− also to help identify negative, alternate and unexpected situations
© 2013 LogiGear Corporation. All Rights Reserved

62
4/11/2013

Testing as a profession
"Do thorough acceptance testing, but not only by the
customer"
− source: "Agile Software Testing in a Large-Scale Project", Israeli Air
Force

Focus on tests, not development:
− what can be consequences of situations and events
− relieve developers

Knowledge and experience with testing techniques and
principles
The challenge for the tester in the new era is to become a
more credible professional tester,
− not a pseudo programmer
− part of the team

Forcing a nontechnical tester to become a programmer
may lose a good tester and gain a poor programmer
© 2013 LogiGear Corporation. All Rights Reserved

Automation is a profession too
Overlaps with regular system development, but not same
Less concerned with complex code structures or
algorithms
More concerned with navigating through other software
efficiently, dealing with control classes, obtaining
information, timing, etc
− if you would compare developers to "creators", automation engineers might
be likened to "adventurers"...

The automation engineering role can also be a consultant:
− for test developers: help express tests efficiently
− for system developers: how to make a system more automation friendly
− important player in innovation in the automated testing

© 2013 LogiGear Corporation. All Rights Reserved

63
4/11/2013

Questions for Organization
How is your testing currently
organized (who is doing what)?
−
−
−
−
−

test design
test development
automation
execution
assessment of release readiness

Do you use agile? If yes, is
there a role for a test
professional? And for an
automation professional?

© 2013 LogiGear Corporation. All Rights Reserved

Reporting
Aim at needs:
−
−
−

avoid lengthy automated reports, have bottom line numbers
reports for stake holders
reporting for the team

Reporting for a big testing project is about:
−
−
−

test and automation progress
production (running the tests)
results (aimed at system under test)

Teams need (relevant) details
−
−
−

what happened, reproducibility, ...
either the tests, the automation, or the system under test
overall situations, with an ability to "drill down" to problem areas

Management needs:
−
−
−
−

status, expectations, issues (realistic! bad news matter, you get punished for not telling)
bottom lines, plan versus reality confrontation
dates, efforts, used resources, costs, run times, ...
never allow planned numbers or dates to be "updated"

Also for reporting, test organization is a key driver
© 2013 LogiGear Corporation. All Rights Reserved

64
4/11/2013

War rooms
Helpful if response times are critical, and a need for cooperation,
towards the same goal
− similar grounds as for agile scrum rooms

Set up at critical times, like before important deadlines, or during
critical releases
Can temporarily bring together multiple parties, that normally are not
co-workers
− like competitor vendors

Pay attention to physical conditions
− machines, monitors, white boards, meeting places, headsets, ...
− food, drinks, ...

The test execution cycle should match the needs of the war room
approach
−
−
−
−

fast turnarounds
effortless
completeness
selective or integral
See also: "Your Game is Live, Now What?", Jane Fraser, Electronic Arts
© 2013 LogiGear Corporation. All Rights Reserved

Globalization....

© 2013 LogiGear Corporation. All Rights Reserved

65
4/11/2013

Main Challenges
Other countries

Distances

Time differences

© 2013 LogiGear Corporation. All Rights Reserved

Globalization
Three Challenges:
− another countries, other cultures
− geographic distances
− time differences

Seven "Patterns":
−
−
−
−
−
−
−

"Solution"
"Push Back"
"Time Pressure"
"Surprises"
"Ownership"
"Mythical Man Month"
"Cooperation"

© 2013 LogiGear Corporation. All Rights Reserved

66
4/11/2013

Challenge: Other Country

© 2013 LogiGear Corporation. All Rights Reserved

Other Country
Differences in culture
−

more on the next slide...

Different languages, and accents
Differences in education
−
−
−

style, orientation and contents
position of critical thinking, factual knowledge, practice, theory,...
US, British, French, Asian, ...

Differences in circumstances
−
−
−

demographics
economy, infrastructure
politics

Apprehension on-shore and off-shore about job security doesn't help in
projects
−
−

management responsibility: understand your strategic intentions, and their consequences, and clarify
them
be realistic in cost and benefit expectations

© 2013 LogiGear Corporation. All Rights Reserved

67
4/11/2013

More on Culture...
Regional culture. There are numerous factors:
−

−

very difficult to make general statements
• many anecdotes, stories and perceptions, some are very helpful, some have limited general
value
• not sure on impact of regional culture (see also [Al-Ani])
numerous factors, like history, religion, political system
• e.g. valuing of: critical thinking, theory, bottom-line, relations, status, work-ethic, bad news,
saying 'no'
• entertaining guests, eating habits, alcohol, meat, humor, etc
• position of leaders, position of women managers
• mistakes can be benign and funny, but also damaging, visibly or hidden, in particular perceived
disrespect hurts

Organizational culture
−
−
−

can be different from country to country, sector to sector, company to company, group to group
I feel this to be at least as strong than regional culture (see for example [Al-Ani])
you can have at least some control over this

Professional cultures
−

for example engineers, QA, managers, ...

Some ideas to help:
−
−

get to know each other (it helps, see for example [Gotel])
study the matter, and make adaptations
© 2013 LogiGear Corporation. All Rights Reserved

© 2013 LogiGear Corporation. All Rights Reserved

68
4/11/2013

© 2013 LogiGear Corporation. All Rights Reserved

© 2013 LogiGear Corporation. All Rights Reserved

69
4/11/2013

© 2013 LogiGear Corporation. All Rights Reserved

Challenge: Distance

© 2013 LogiGear Corporation. All Rights Reserved

70
4/11/2013

Distance
Continuous logistical challenges
Lots of costs, and disruptions, for traveling
Distance creates distrust and conflict
− could be "normal" behavior, inherent to humans

Complex coordination can create misunderstandings
− on technical topics
− on actions, priorities, and intentions

© 2013 LogiGear Corporation. All Rights Reserved

Challenge: Time difference

© 2013 LogiGear Corporation. All Rights Reserved

71
4/11/2013

Challenge: Time difference
Additional complication for communication and
coordination
Places a major burden on both on-shore and off-shore
staff
− having to work evenings and/or early mornings
− potential for exhaustion, lack of relaxation, mistakes, irritation

Can easily lead to loss of time at critical moments
Some solutions:
−
−
−
−

manage this actively
constantly seek to optimize task and responsibility allocation
build the on-shore and off-shore organizations to match
seek ways to save meeting time, like optimal information handling

© 2013 LogiGear Corporation. All Rights Reserved

Effect of time difference
Report from the team to the US management . . .
Performance comparison TestArchitect 5 and 6
Test Module:
“Segment Y, Default Settings”

TestArchitect 5
TestArchitect 6

Windows

~ 4:16 m
~ 11:00 m

Linux

~ 4:28 m
~ 8:00 m

© 2013 LogiGear Corporation. All Rights Reserved

72
4/11/2013

Patterns
Experiences seem to follow patterns
− at least our own experiences do
− variations are numerous, but seem to follow similar lines
− following are examples, not limitative

It can help to recognize patterns quickly, and act upon
them
Resolutions have side-effects, can introduce new issues
− for example strengthening local management means less direct
contact with the project members doing the work

Just about every pattern occurs in every direction
− from your perspective regarding "them"
− their perspective on you, or each other
− sometimes equaling, sometimes mirroring
© 2013 LogiGear Corporation. All Rights Reserved

Pattern: "The Solution"
Typical sequence of events:
− the team finds a problem in running a test
− the team discusses it and comes up with a "solution"
− the solution: (1) creates issues, and (2) hides the real
problem

Better way:
− define as an issue
− discuss with project manager and customer

© 2013 LogiGear Corporation. All Rights Reserved

73
4/11/2013

Pattern: "Push Back"
US side, or customer, gives bad direction
Team doesn't like it, but feels obliged to follow orders
The result is disappointing
Team is blamed
− and will speak up even less next time

Better way:
− discuss with the principal/customer at multiple levels
• strategic about direction, operational day-to-day

− empower and encourage the team to speak up
− write plans of approach, and reports

© 2013 LogiGear Corporation. All Rights Reserved

Pattern: "Time Pressure"
Deadline must be met
− no matter what
− use over-time
− "failure is not an option"

Deadlines are sometimes real, sometimes not
−
−
−
−

become a routine on the US side
easy to pressure over the email
very difficult for a non-empowered team to push back
risk: inflation of urgency

Better way:
−
−
−
−

good planning
proper weighing of deadlines and priorities
frequent reporting
local management
© 2013 LogiGear Corporation. All Rights Reserved

74
4/11/2013

Pattern: "Surprises"
Good news travels better than bad news...
− should be the other way around
− the "cover up": "let's fix, no need to tell...."
− over time: needing bigger cover ups to conceal
smaller ones
− not unique for off-shoring, but more difficult to
detect and deal with

Once a surprise happens:
− you will feel frustrated, and betrayed
− fix the problems, point out the consequences of
hiding, avoid screaming and flaming

Better ways:
−
−
−
−
−

agree: NO SURPRISES!!
emphasize again and again
train against this
continuously manage, point out
the magic word: transparency

SUPRISES

© 2013 LogiGear Corporation. All Rights Reserved

Pattern: "Ownership"
Shared responsibility is no responsibility
Effort-based versus result-based
On-shore players feel the off-shore team has a result responsibility
Off-shore team members feel an effort-based responsibility ("work
hard")
Better way:
− clear responsibilities and expectations
− on-shore ownership for quality control of system under test
• and therefore the tests

− off-shore ownership of producing good tests and good automation
− empower according to ownership

© 2013 LogiGear Corporation. All Rights Reserved

75
4/11/2013

Pattern: "Mythical Man Month"
Fred Brooks classic book, "Mythical man month":
− "Assigning more programmers to a project running behind schedule
will make it even later"
− "The bearing of a child takes nine months, no matter how many
women are assigned"

In test automation, there must be clear ownership of:
− test design (not just cranking out test cases)
− automation, this is different skill and interest

Assign at least the following roles:
− project lead, owns quality and schedule
− test lead: owns test design, coaches and coordinates the other testers
− automation: make the actions work (assuming ABT, not the test
cases)

Define distinct career paths in: testing, automation,
management
© 2013 LogiGear Corporation. All Rights Reserved

Pattern: "Cooperation"
Communication is tedious, takes a long time
Questions, questions, questions, ...
− reverse: questions don't get answered

For at least one side in private time, extra annoying
Misunderstandings, confusion, actions not followed up
− double check apparent "crazy things" with the team before jumping to conclusions, and
actions (assume the other side is not "nuts" or "dumb"...)

Please understand: distance fosters conflicts
− we're born that way, be ready for it

Better ways:
− prioritize training, coaching, preparation and planning. Saves a lot of questions...
− write stuff down, use briefs, minutes
− define workflows and information flows
•

buckets, reporting, select and use good tools

− specialize meetings
•
•

table things for in-depth meetings
ask to meet internally first

− be quick, no more than 30 mins
© 2013 LogiGear Corporation. All Rights Reserved

76
4/11/2013

Cooperation

© 2013 LogiGear Corporation. All Rights Reserved

Training, some ideas
Many areas, big pay-offs:
−
−
−
−
−
−
−
−

system under test
subject matter under test, domain knowledge
methods, best practices
technologies, tools, ...
processes
soft skills, like creativity, critical thinking, management, ...
language
cross-cultural

Have exams
−
−
−

think about the consequences of passing and failing
teams pay more attention when they know they will get tested
you will know whether you were understood

Also have coaching and train-the-trainers
−
−
−
−

more experienced people help newbie's
also runs a risk: bad habits can creep in and procreate
"Tribal knowledge", learning by osmosis, water cooler conversations, encourage it
consider "special interest groups (SIG's)"

Rule of thumb for off-shore teams: hire for technical knowledge, train for business
knowledge
The on-shore staff needs training and coaching too
© 2013 LogiGear Corporation. All Rights Reserved

77
4/11/2013

Additional ideas
Go there, be with the team, experience yourself how "your side" is doing
−

I go about twice per year

Manage ownership, is it you or them
−

the distinction between efforts and results

Provide clear direction, constant attention and coaching
Supervise, supervise, supervise
−

but don't micromanage if the other side has ownership

Ask to create example products (like ABT test modules and actions), review
them carefully, and use as direction for subsequent work
Leadership style: participative styles seem most common (as opposed to
consensus or authoritative, see also [Al-Ani])
Organize informal/fun events, provide a good environment
−
−

solidify the group, improve retention
include visiting US staff, this tends to do a lot of good ("priceless")

Manage expectations
−
−

stuff takes time and energy
differences can be addressed, but not 100%

cake...
© 2013 LogiGear Corporation. All Rights Reserved

Outsourcing and Agile
If done well, can provide relieve to a lot of the patterns
Several models possible
Model 1: Full team outsourcing
−

development, testing and automation

Model 2: "2nd unit"
−

off-shore team works under control of sprint team members

Model 3: Part of integrated team:
−
−
−

needs online tool like Jira or Rally
you must have shared meetings
advantage: more project time

Large scale test development and automation might be easier to
outsource than development

© 2013 LogiGear Corporation. All Rights Reserved

78
4/11/2013

Summary
Not all "big project" challenges are the same
Think before you do. Best results come from planning
well, and combining effective concepts, tricks and tools
Consider tests and automation as products
Team work is a key for short term and long term success
There are many options for infrastructure, but keep an
eye on economy and planning
Off-shoring can help scale up, but needs attention to do it
right, in particular communication
Repeat of initial invitation
Focus today was on overview and concepts, not always on details. Please see
me in person for any discussion you would like on your situation that I didn't
cover. We're also exhibiting here, probably easiest to reach me there.
© 2013 LogiGear Corporation. All Rights Reserved

Homework . . .
1.

Testing Computer Software, Cem Kaner, Hung Nguyen, Jack Falk, Wiley

2.

Lessons Learned in Software Testing, Cem Kaner, James Bach, Bret Pettichord, Wiley

3.

Experiences of Test Automation, Dorothy Graham, Mark Fewster, Addison Wesley, 2012

4.

Automating Software Testing, Dorothy Graham, Mark Fewster, Addison Wesley

5.

"Build a Successful Global Training Program", Michael Hackett, www.logigear.com

6.

Action Based Testing (overview article), Hans Buwalda, Better Software, March 2011

7.

Action Figures (on model-based testing), Hans Buwalda, Better Software, March 2003

8.

Integrated Test Design & Automation, Hans Buwalda, Dennis Janssen and Iris Pinkster, Addison Wesley

9.

Soap Opera Testing (article), Hans Buwalda, Better Software Magazine, February 2005

10.

Testing with Action Words, Abandoning Record and Playback, Hans Buwalda, Eurostar 1996

11.

QA All Stars, Building Your Dream Team, Hans Buwalda, Better Software, September 2006

12.

The 5% Solutions, Hans Buwalda, Software Test & Performance Magazine, September 2006

13.

Happy About Global Software Test Automation, Hung Nguyen, Michael Hackett, e.a., Happy About

14.

Testing Applications on the Web, Hung Nguyen, Robert Johnson, Michael Hackett, Wiley

15.

Practical Combinatorial Testing, Richard Kuhn, Raghu Kacker, Yu Lei, NIST, October, 2010

16.

Agile Software Testing in a Large-Scale Project, David Talby, Arie Keren, Orit Hazzan, Yael Dubinsky, IEEE
Software, July/August 2006

17.

JMeter in the Cloud, Jörg Kalsbach, http://aws.amazon.com/amis/2924

18.

Using Monkey Test Tools, Noel Nyman, STQE issue January/February 2000

19.

High Volume Test Automation, Cem Kaner, Walter P. Bond, Pat McGee, StarEast 2004

20.

Descriptive Analysis of Fear and Distrust in Early Phases of GSD Projects, Arttu Piri, Tuomas Niinimäki, Casper
Lassenius, 2009 Fourth IEEE International Conference on Global Software Engineering [Piri]

21.

Quality Indicators on Global Software Development Projects: Does 'Getting to Know You' Really Matter? Olly Gotel, Vidya Kulkarni,
Moniphal Say, Christelle Scharff, Thanwadee Sunetnanta, 2009 Fourth IEEE International Conference on Global Software
Engineering [Gotel]
© 2013 LogiGear Corporation. All Rights Reserved

79
4/11/2013

Thanks...
Please fill out the evaluation form, from the back
of the book
Let me know any questions or concerns:
We're at the expo, questions welcome
− I will be there myself too quite a bit
TESTING FOR SALE

email:

hans @ logigear.com

articles:

www.happytester.com

company:

www.logigear.com

TestArchitect: www.testarchitect.com

Supersize your tests
for less...

we're at the expo
© 2013 LogiGear Corporation. All Rights Reserved

80

Weitere ähnliche Inhalte

Was ist angesagt?

Universal test solutions customer testimonial 10192013-v2.2
Universal test solutions customer testimonial 10192013-v2.2Universal test solutions customer testimonial 10192013-v2.2
Universal test solutions customer testimonial 10192013-v2.2Universal Technology Solutions
 
Test Metrics in Agile: A Powerful Tool to Demonstrate Value
Test Metrics in Agile: A Powerful Tool to Demonstrate ValueTest Metrics in Agile: A Powerful Tool to Demonstrate Value
Test Metrics in Agile: A Powerful Tool to Demonstrate ValueTechWell
 
Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical ...
Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical ...Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical ...
Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical ...TEST Huddle
 
Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010
Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010
Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010TEST Huddle
 
Introducing Keyword-Driven Test Automation
Introducing Keyword-Driven Test AutomationIntroducing Keyword-Driven Test Automation
Introducing Keyword-Driven Test AutomationTechWell
 
Introducing Keyword-driven Test Automation
Introducing Keyword-driven Test AutomationIntroducing Keyword-driven Test Automation
Introducing Keyword-driven Test AutomationTechWell
 
Instill a DevOps Testing Culture in Your Team and Organization
Instill a DevOps Testing Culture in Your Team and Organization Instill a DevOps Testing Culture in Your Team and Organization
Instill a DevOps Testing Culture in Your Team and Organization TechWell
 
Avatars of Test Driven Development (TDD)
Avatars of Test Driven Development (TDD)Avatars of Test Driven Development (TDD)
Avatars of Test Driven Development (TDD)Naresh Jain
 
Selenium DeTox for Achieving the Right Testing Pyramid
Selenium DeTox for Achieving the Right Testing PyramidSelenium DeTox for Achieving the Right Testing Pyramid
Selenium DeTox for Achieving the Right Testing PyramidNaresh Jain
 
Seven Keys to Navigating Your Agile Testing Transition
Seven Keys to Navigating Your Agile Testing TransitionSeven Keys to Navigating Your Agile Testing Transition
Seven Keys to Navigating Your Agile Testing TransitionTechWell
 

Was ist angesagt? (10)

Universal test solutions customer testimonial 10192013-v2.2
Universal test solutions customer testimonial 10192013-v2.2Universal test solutions customer testimonial 10192013-v2.2
Universal test solutions customer testimonial 10192013-v2.2
 
Test Metrics in Agile: A Powerful Tool to Demonstrate Value
Test Metrics in Agile: A Powerful Tool to Demonstrate ValueTest Metrics in Agile: A Powerful Tool to Demonstrate Value
Test Metrics in Agile: A Powerful Tool to Demonstrate Value
 
Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical ...
Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical ...Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical ...
Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical ...
 
Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010
Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010
Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010
 
Introducing Keyword-Driven Test Automation
Introducing Keyword-Driven Test AutomationIntroducing Keyword-Driven Test Automation
Introducing Keyword-Driven Test Automation
 
Introducing Keyword-driven Test Automation
Introducing Keyword-driven Test AutomationIntroducing Keyword-driven Test Automation
Introducing Keyword-driven Test Automation
 
Instill a DevOps Testing Culture in Your Team and Organization
Instill a DevOps Testing Culture in Your Team and Organization Instill a DevOps Testing Culture in Your Team and Organization
Instill a DevOps Testing Culture in Your Team and Organization
 
Avatars of Test Driven Development (TDD)
Avatars of Test Driven Development (TDD)Avatars of Test Driven Development (TDD)
Avatars of Test Driven Development (TDD)
 
Selenium DeTox for Achieving the Right Testing Pyramid
Selenium DeTox for Achieving the Right Testing PyramidSelenium DeTox for Achieving the Right Testing Pyramid
Selenium DeTox for Achieving the Right Testing Pyramid
 
Seven Keys to Navigating Your Agile Testing Transition
Seven Keys to Navigating Your Agile Testing TransitionSeven Keys to Navigating Your Agile Testing Transition
Seven Keys to Navigating Your Agile Testing Transition
 

Andere mochten auch

Staying Ahead in the Cybersecurity Game: What Matters Now
Staying Ahead in the Cybersecurity Game: What Matters NowStaying Ahead in the Cybersecurity Game: What Matters Now
Staying Ahead in the Cybersecurity Game: What Matters NowCapgemini
 
Индивидуальная программа карьерного развития
Индивидуальная программа карьерного развития Индивидуальная программа карьерного развития
Индивидуальная программа карьерного развития LAZOVOY
 
Avaliação para o investimento social privado - metodologias
Avaliação para o investimento social privado - metodologiasAvaliação para o investimento social privado - metodologias
Avaliação para o investimento social privado - metodologiasBruno Rabelo
 
íNdia Cavernas De Ajanta
íNdia   Cavernas De AjantaíNdia   Cavernas De Ajanta
íNdia Cavernas De Ajantacab3032
 
Manual técnico del software
Manual técnico del softwareManual técnico del software
Manual técnico del softwareYenny Aldana
 
Educacion y personas_con_discapacidad_presente_futuro
Educacion y personas_con_discapacidad_presente_futuroEducacion y personas_con_discapacidad_presente_futuro
Educacion y personas_con_discapacidad_presente_futuroMarta Montoro
 
Bridgeprojectsource online-120723110339-phpapp02
Bridgeprojectsource online-120723110339-phpapp02Bridgeprojectsource online-120723110339-phpapp02
Bridgeprojectsource online-120723110339-phpapp02Ferit Fazliu
 
Trabajo de la tierra victoria
Trabajo de la tierra   victoriaTrabajo de la tierra   victoria
Trabajo de la tierra victoriacomputacion123
 
Checklistojithoz 120521125515-phpapp01
Checklistojithoz 120521125515-phpapp01Checklistojithoz 120521125515-phpapp01
Checklistojithoz 120521125515-phpapp01Yadira Azpilcueta
 
Padrões de aptidão do enfermeiro forense
Padrões de aptidão do enfermeiro forensePadrões de aptidão do enfermeiro forense
Padrões de aptidão do enfermeiro forenseAlbino Gomes
 
Mobile technology and information literacy instruction: the McGill Library ex...
Mobile technology and information literacy instruction: the McGill Library ex...Mobile technology and information literacy instruction: the McGill Library ex...
Mobile technology and information literacy instruction: the McGill Library ex...Maria Savova
 
Donostweets 4 de octubre - Estrategia Digital
Donostweets 4 de octubre - Estrategia DigitalDonostweets 4 de octubre - Estrategia Digital
Donostweets 4 de octubre - Estrategia DigitalVirginia Aguirre
 

Andere mochten auch (20)

Guía Aula Virtual - Estudiante
Guía Aula Virtual - EstudianteGuía Aula Virtual - Estudiante
Guía Aula Virtual - Estudiante
 
Estructura teatro del devenir fca2015 (1)
Estructura teatro del devenir fca2015 (1)Estructura teatro del devenir fca2015 (1)
Estructura teatro del devenir fca2015 (1)
 
Staying Ahead in the Cybersecurity Game: What Matters Now
Staying Ahead in the Cybersecurity Game: What Matters NowStaying Ahead in the Cybersecurity Game: What Matters Now
Staying Ahead in the Cybersecurity Game: What Matters Now
 
Индивидуальная программа карьерного развития
Индивидуальная программа карьерного развития Индивидуальная программа карьерного развития
Индивидуальная программа карьерного развития
 
Avaliação para o investimento social privado - metodologias
Avaliação para o investimento social privado - metodologiasAvaliação para o investimento social privado - metodologias
Avaliação para o investimento social privado - metodologias
 
íNdia Cavernas De Ajanta
íNdia   Cavernas De AjantaíNdia   Cavernas De Ajanta
íNdia Cavernas De Ajanta
 
Manual técnico del software
Manual técnico del softwareManual técnico del software
Manual técnico del software
 
Plan anual 2013 1
Plan anual 2013 1Plan anual 2013 1
Plan anual 2013 1
 
Romeinen 1
Romeinen 1Romeinen 1
Romeinen 1
 
Educacion y personas_con_discapacidad_presente_futuro
Educacion y personas_con_discapacidad_presente_futuroEducacion y personas_con_discapacidad_presente_futuro
Educacion y personas_con_discapacidad_presente_futuro
 
Bridgeprojectsource online-120723110339-phpapp02
Bridgeprojectsource online-120723110339-phpapp02Bridgeprojectsource online-120723110339-phpapp02
Bridgeprojectsource online-120723110339-phpapp02
 
Trabajo de la tierra victoria
Trabajo de la tierra   victoriaTrabajo de la tierra   victoria
Trabajo de la tierra victoria
 
Fiestas TRES CANTOS
Fiestas TRES CANTOSFiestas TRES CANTOS
Fiestas TRES CANTOS
 
Presentacion
PresentacionPresentacion
Presentacion
 
Checklistojithoz 120521125515-phpapp01
Checklistojithoz 120521125515-phpapp01Checklistojithoz 120521125515-phpapp01
Checklistojithoz 120521125515-phpapp01
 
Padrões de aptidão do enfermeiro forense
Padrões de aptidão do enfermeiro forensePadrões de aptidão do enfermeiro forense
Padrões de aptidão do enfermeiro forense
 
Mobile technology and information literacy instruction: the McGill Library ex...
Mobile technology and information literacy instruction: the McGill Library ex...Mobile technology and information literacy instruction: the McGill Library ex...
Mobile technology and information literacy instruction: the McGill Library ex...
 
Síntese dos processos (formação & avaliação)
Síntese dos processos (formação & avaliação)Síntese dos processos (formação & avaliação)
Síntese dos processos (formação & avaliação)
 
Donostweets 4 de octubre - Estrategia Digital
Donostweets 4 de octubre - Estrategia DigitalDonostweets 4 de octubre - Estrategia Digital
Donostweets 4 de octubre - Estrategia Digital
 
Vhs, kaposi
Vhs, kaposiVhs, kaposi
Vhs, kaposi
 

Ähnlich wie The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More

The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More
The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and MoreThe Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More
The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and MoreTechWell
 
The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More
The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and MoreThe Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More
The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and MoreTechWell
 
'BIG Testing' with Hans Buwalda
'BIG Testing' with Hans Buwalda 'BIG Testing' with Hans Buwalda
'BIG Testing' with Hans Buwalda TEST Huddle
 
Introducing Keyword-driven Test Automation
Introducing Keyword-driven Test AutomationIntroducing Keyword-driven Test Automation
Introducing Keyword-driven Test AutomationTechWell
 
Better Test Designs to Drive Test Automation Excellence
Better Test Designs to Drive Test Automation ExcellenceBetter Test Designs to Drive Test Automation Excellence
Better Test Designs to Drive Test Automation ExcellenceTechWell
 
Introducing Keyword-Driven Test Automation
Introducing Keyword-Driven Test AutomationIntroducing Keyword-Driven Test Automation
Introducing Keyword-Driven Test AutomationTechWell
 
Introducing Keyword-driven Test Automation
Introducing Keyword-driven Test AutomationIntroducing Keyword-driven Test Automation
Introducing Keyword-driven Test AutomationTechWell
 
Model-Based Testing with Keywords
Model-Based Testing with KeywordsModel-Based Testing with Keywords
Model-Based Testing with KeywordsTechWell
 
The Survey Says: Testers Spend Their Time Doing...
The Survey Says: Testers Spend Their Time Doing...The Survey Says: Testers Spend Their Time Doing...
The Survey Says: Testers Spend Their Time Doing...TechWell
 
Management Issues in Test Automation
Management Issues in Test AutomationManagement Issues in Test Automation
Management Issues in Test AutomationTechWell
 
Universal test solutions customer testimonial 10192013-v2.3
Universal test solutions customer testimonial 10192013-v2.3Universal test solutions customer testimonial 10192013-v2.3
Universal test solutions customer testimonial 10192013-v2.3Universal Technology Solutions
 
Top 5 Pitfalls of Test Automation and How To Avoid Them
Top 5 Pitfalls of Test Automation and How To Avoid ThemTop 5 Pitfalls of Test Automation and How To Avoid Them
Top 5 Pitfalls of Test Automation and How To Avoid ThemSundar Sritharan
 
Software Houses Use Agile Development
Software Houses Use Agile DevelopmentSoftware Houses Use Agile Development
Software Houses Use Agile DevelopmentKayla Miller
 
Improving ROI with Scriptless Test Automation
Improving ROI with Scriptless Test AutomationImproving ROI with Scriptless Test Automation
Improving ROI with Scriptless Test AutomationMindfire LLC
 
The Automation Firehose: Be Strategic & Tactical With Your Mobile & Web Testing
The Automation Firehose: Be Strategic & Tactical With Your Mobile & Web TestingThe Automation Firehose: Be Strategic & Tactical With Your Mobile & Web Testing
The Automation Firehose: Be Strategic & Tactical With Your Mobile & Web TestingPerfecto by Perforce
 
Use Automation to Assist—Not Replace—Manual Testing
Use Automation to Assist—Not Replace—Manual TestingUse Automation to Assist—Not Replace—Manual Testing
Use Automation to Assist—Not Replace—Manual TestingTechWell
 
Augmenting Coded UI
Augmenting Coded UIAugmenting Coded UI
Augmenting Coded UItravisk
 
Management Issues in Test Automation
Management Issues in Test AutomationManagement Issues in Test Automation
Management Issues in Test AutomationTechWell
 
The Tester’s Role: Balancing Technical Acumen and User Advocacy
The Tester’s Role: Balancing Technical Acumen and User AdvocacyThe Tester’s Role: Balancing Technical Acumen and User Advocacy
The Tester’s Role: Balancing Technical Acumen and User AdvocacyTechWell
 
Software testing 2012 - A Year in Review
Software testing 2012 - A Year in ReviewSoftware testing 2012 - A Year in Review
Software testing 2012 - A Year in ReviewJohan Hoberg
 

Ähnlich wie The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More (20)

The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More
The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and MoreThe Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More
The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More
 
The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More
The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and MoreThe Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More
The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More
 
'BIG Testing' with Hans Buwalda
'BIG Testing' with Hans Buwalda 'BIG Testing' with Hans Buwalda
'BIG Testing' with Hans Buwalda
 
Introducing Keyword-driven Test Automation
Introducing Keyword-driven Test AutomationIntroducing Keyword-driven Test Automation
Introducing Keyword-driven Test Automation
 
Better Test Designs to Drive Test Automation Excellence
Better Test Designs to Drive Test Automation ExcellenceBetter Test Designs to Drive Test Automation Excellence
Better Test Designs to Drive Test Automation Excellence
 
Introducing Keyword-Driven Test Automation
Introducing Keyword-Driven Test AutomationIntroducing Keyword-Driven Test Automation
Introducing Keyword-Driven Test Automation
 
Introducing Keyword-driven Test Automation
Introducing Keyword-driven Test AutomationIntroducing Keyword-driven Test Automation
Introducing Keyword-driven Test Automation
 
Model-Based Testing with Keywords
Model-Based Testing with KeywordsModel-Based Testing with Keywords
Model-Based Testing with Keywords
 
The Survey Says: Testers Spend Their Time Doing...
The Survey Says: Testers Spend Their Time Doing...The Survey Says: Testers Spend Their Time Doing...
The Survey Says: Testers Spend Their Time Doing...
 
Management Issues in Test Automation
Management Issues in Test AutomationManagement Issues in Test Automation
Management Issues in Test Automation
 
Universal test solutions customer testimonial 10192013-v2.3
Universal test solutions customer testimonial 10192013-v2.3Universal test solutions customer testimonial 10192013-v2.3
Universal test solutions customer testimonial 10192013-v2.3
 
Top 5 Pitfalls of Test Automation and How To Avoid Them
Top 5 Pitfalls of Test Automation and How To Avoid ThemTop 5 Pitfalls of Test Automation and How To Avoid Them
Top 5 Pitfalls of Test Automation and How To Avoid Them
 
Software Houses Use Agile Development
Software Houses Use Agile DevelopmentSoftware Houses Use Agile Development
Software Houses Use Agile Development
 
Improving ROI with Scriptless Test Automation
Improving ROI with Scriptless Test AutomationImproving ROI with Scriptless Test Automation
Improving ROI with Scriptless Test Automation
 
The Automation Firehose: Be Strategic & Tactical With Your Mobile & Web Testing
The Automation Firehose: Be Strategic & Tactical With Your Mobile & Web TestingThe Automation Firehose: Be Strategic & Tactical With Your Mobile & Web Testing
The Automation Firehose: Be Strategic & Tactical With Your Mobile & Web Testing
 
Use Automation to Assist—Not Replace—Manual Testing
Use Automation to Assist—Not Replace—Manual TestingUse Automation to Assist—Not Replace—Manual Testing
Use Automation to Assist—Not Replace—Manual Testing
 
Augmenting Coded UI
Augmenting Coded UIAugmenting Coded UI
Augmenting Coded UI
 
Management Issues in Test Automation
Management Issues in Test AutomationManagement Issues in Test Automation
Management Issues in Test Automation
 
The Tester’s Role: Balancing Technical Acumen and User Advocacy
The Tester’s Role: Balancing Technical Acumen and User AdvocacyThe Tester’s Role: Balancing Technical Acumen and User Advocacy
The Tester’s Role: Balancing Technical Acumen and User Advocacy
 
Software testing 2012 - A Year in Review
Software testing 2012 - A Year in ReviewSoftware testing 2012 - A Year in Review
Software testing 2012 - A Year in Review
 

Mehr von TechWell

Failing and Recovering
Failing and RecoveringFailing and Recovering
Failing and RecoveringTechWell
 
Test Design for Fully Automated Build Architecture
Test Design for Fully Automated Build ArchitectureTest Design for Fully Automated Build Architecture
Test Design for Fully Automated Build ArchitectureTechWell
 
System-Level Test Automation: Ensuring a Good Start
System-Level Test Automation: Ensuring a Good StartSystem-Level Test Automation: Ensuring a Good Start
System-Level Test Automation: Ensuring a Good StartTechWell
 
Build Your Mobile App Quality and Test Strategy
Build Your Mobile App Quality and Test StrategyBuild Your Mobile App Quality and Test Strategy
Build Your Mobile App Quality and Test StrategyTechWell
 
Testing Transformation: The Art and Science for Success
Testing Transformation: The Art and Science for SuccessTesting Transformation: The Art and Science for Success
Testing Transformation: The Art and Science for SuccessTechWell
 
Implement BDD with Cucumber and SpecFlow
Implement BDD with Cucumber and SpecFlowImplement BDD with Cucumber and SpecFlow
Implement BDD with Cucumber and SpecFlowTechWell
 
Develop WebDriver Automated Tests—and Keep Your Sanity
Develop WebDriver Automated Tests—and Keep Your SanityDevelop WebDriver Automated Tests—and Keep Your Sanity
Develop WebDriver Automated Tests—and Keep Your SanityTechWell
 
Eliminate Cloud Waste with a Holistic DevOps Strategy
Eliminate Cloud Waste with a Holistic DevOps StrategyEliminate Cloud Waste with a Holistic DevOps Strategy
Eliminate Cloud Waste with a Holistic DevOps StrategyTechWell
 
Transform Test Organizations for the New World of DevOps
Transform Test Organizations for the New World of DevOpsTransform Test Organizations for the New World of DevOps
Transform Test Organizations for the New World of DevOpsTechWell
 
The Fourth Constraint in Project Delivery—Leadership
The Fourth Constraint in Project Delivery—LeadershipThe Fourth Constraint in Project Delivery—Leadership
The Fourth Constraint in Project Delivery—LeadershipTechWell
 
Resolve the Contradiction of Specialists within Agile Teams
Resolve the Contradiction of Specialists within Agile TeamsResolve the Contradiction of Specialists within Agile Teams
Resolve the Contradiction of Specialists within Agile TeamsTechWell
 
Pin the Tail on the Metric: A Field-Tested Agile Game
Pin the Tail on the Metric: A Field-Tested Agile GamePin the Tail on the Metric: A Field-Tested Agile Game
Pin the Tail on the Metric: A Field-Tested Agile GameTechWell
 
Agile Performance Holarchy (APH)—A Model for Scaling Agile Teams
Agile Performance Holarchy (APH)—A Model for Scaling Agile TeamsAgile Performance Holarchy (APH)—A Model for Scaling Agile Teams
Agile Performance Holarchy (APH)—A Model for Scaling Agile TeamsTechWell
 
A Business-First Approach to DevOps Implementation
A Business-First Approach to DevOps ImplementationA Business-First Approach to DevOps Implementation
A Business-First Approach to DevOps ImplementationTechWell
 
Databases in a Continuous Integration/Delivery Process
Databases in a Continuous Integration/Delivery ProcessDatabases in a Continuous Integration/Delivery Process
Databases in a Continuous Integration/Delivery ProcessTechWell
 
Mobile Testing: What—and What Not—to Automate
Mobile Testing: What—and What Not—to AutomateMobile Testing: What—and What Not—to Automate
Mobile Testing: What—and What Not—to AutomateTechWell
 
Cultural Intelligence: A Key Skill for Success
Cultural Intelligence: A Key Skill for SuccessCultural Intelligence: A Key Skill for Success
Cultural Intelligence: A Key Skill for SuccessTechWell
 
Turn the Lights On: A Power Utility Company's Agile Transformation
Turn the Lights On: A Power Utility Company's Agile TransformationTurn the Lights On: A Power Utility Company's Agile Transformation
Turn the Lights On: A Power Utility Company's Agile TransformationTechWell
 
Scale: The Most Hyped Term in Agile Development Today
Scale: The Most Hyped Term in Agile Development TodayScale: The Most Hyped Term in Agile Development Today
Scale: The Most Hyped Term in Agile Development TodayTechWell
 

Mehr von TechWell (20)

Failing and Recovering
Failing and RecoveringFailing and Recovering
Failing and Recovering
 
Test Design for Fully Automated Build Architecture
Test Design for Fully Automated Build ArchitectureTest Design for Fully Automated Build Architecture
Test Design for Fully Automated Build Architecture
 
System-Level Test Automation: Ensuring a Good Start
System-Level Test Automation: Ensuring a Good StartSystem-Level Test Automation: Ensuring a Good Start
System-Level Test Automation: Ensuring a Good Start
 
Build Your Mobile App Quality and Test Strategy
Build Your Mobile App Quality and Test StrategyBuild Your Mobile App Quality and Test Strategy
Build Your Mobile App Quality and Test Strategy
 
Testing Transformation: The Art and Science for Success
Testing Transformation: The Art and Science for SuccessTesting Transformation: The Art and Science for Success
Testing Transformation: The Art and Science for Success
 
Implement BDD with Cucumber and SpecFlow
Implement BDD with Cucumber and SpecFlowImplement BDD with Cucumber and SpecFlow
Implement BDD with Cucumber and SpecFlow
 
Develop WebDriver Automated Tests—and Keep Your Sanity
Develop WebDriver Automated Tests—and Keep Your SanityDevelop WebDriver Automated Tests—and Keep Your Sanity
Develop WebDriver Automated Tests—and Keep Your Sanity
 
Ma 15
Ma 15Ma 15
Ma 15
 
Eliminate Cloud Waste with a Holistic DevOps Strategy
Eliminate Cloud Waste with a Holistic DevOps StrategyEliminate Cloud Waste with a Holistic DevOps Strategy
Eliminate Cloud Waste with a Holistic DevOps Strategy
 
Transform Test Organizations for the New World of DevOps
Transform Test Organizations for the New World of DevOpsTransform Test Organizations for the New World of DevOps
Transform Test Organizations for the New World of DevOps
 
The Fourth Constraint in Project Delivery—Leadership
The Fourth Constraint in Project Delivery—LeadershipThe Fourth Constraint in Project Delivery—Leadership
The Fourth Constraint in Project Delivery—Leadership
 
Resolve the Contradiction of Specialists within Agile Teams
Resolve the Contradiction of Specialists within Agile TeamsResolve the Contradiction of Specialists within Agile Teams
Resolve the Contradiction of Specialists within Agile Teams
 
Pin the Tail on the Metric: A Field-Tested Agile Game
Pin the Tail on the Metric: A Field-Tested Agile GamePin the Tail on the Metric: A Field-Tested Agile Game
Pin the Tail on the Metric: A Field-Tested Agile Game
 
Agile Performance Holarchy (APH)—A Model for Scaling Agile Teams
Agile Performance Holarchy (APH)—A Model for Scaling Agile TeamsAgile Performance Holarchy (APH)—A Model for Scaling Agile Teams
Agile Performance Holarchy (APH)—A Model for Scaling Agile Teams
 
A Business-First Approach to DevOps Implementation
A Business-First Approach to DevOps ImplementationA Business-First Approach to DevOps Implementation
A Business-First Approach to DevOps Implementation
 
Databases in a Continuous Integration/Delivery Process
Databases in a Continuous Integration/Delivery ProcessDatabases in a Continuous Integration/Delivery Process
Databases in a Continuous Integration/Delivery Process
 
Mobile Testing: What—and What Not—to Automate
Mobile Testing: What—and What Not—to AutomateMobile Testing: What—and What Not—to Automate
Mobile Testing: What—and What Not—to Automate
 
Cultural Intelligence: A Key Skill for Success
Cultural Intelligence: A Key Skill for SuccessCultural Intelligence: A Key Skill for Success
Cultural Intelligence: A Key Skill for Success
 
Turn the Lights On: A Power Utility Company's Agile Transformation
Turn the Lights On: A Power Utility Company's Agile TransformationTurn the Lights On: A Power Utility Company's Agile Transformation
Turn the Lights On: A Power Utility Company's Agile Transformation
 
Scale: The Most Hyped Term in Agile Development Today
Scale: The Most Hyped Term in Agile Development TodayScale: The Most Hyped Term in Agile Development Today
Scale: The Most Hyped Term in Agile Development Today
 

Kürzlich hochgeladen

Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 

Kürzlich hochgeladen (20)

Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 

The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More

  • 1. MB Full-day Tutorial 4/29/13 8:30AM The Challenges of BIG Testing: Automation, Virtualization, Outsourcing, and More Presented by: Hans Buwalda LogiGear Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  • 2. Hans Buwalda An internationally recognized expert in testing, Hans Buwalda is a pioneer of keyword-driven test automation, an approach now widely adopted throughout the testing industry. Originally from the Netherlands, Hans is the CTO of California-based LogiGear, directing the development of the successful Action Based Testing™methodology for keyword-driven test automation and its supporting TestArchitect™ toolset. Prior to joining LogiGear, Hans served as project director at CMG (now Logica). Hans speaks frequently at international conferences on concepts such as Soap Opera Testing, Three Holy Grails of Test Development, Testing in the Cold, and Jungle Testing. Hans is coauthor of Integrated Test Design and Automation: Using the TestFrame Method.
  • 3. 4/11/2013 STAREAST 2013, Tutorial MB Orlando, Monday April 29 The Challenges of BIG Testing Automation, Virtualization, Outsourcing, and More Hans Buwalda LogiGear © 2013 LogiGear Corporation. All Rights Reserved Introduction − industries − roles in testing © 2013 LogiGear Corporation. All Rights Reserved 1
  • 4. 4/11/2013 About LogiGear www.logigear.com www.testarchitect.com Software testing company, around since 1994 Testing and test automation expertise, services and tooling − consultancy, training − test development and automation services − "test integrated" development services Aims to be thought leader, in particular for large and complex test projects Products: − TestArchitect™, TestArchitect for Visual Studio™ − integrating test development with test management and automation − based on modularized keyword-driven testing © 2013 LogiGear Corporation. All Rights Reserved About Hans www.happytester.com hans @ logigear.com Dutch guy, living and working in California since 2001, as CTO of LogiGear Background in math, computer science, management Original career in management consultancy, since 1994 focusing on testing and test automation − keywords, agile testing, big testing, . . . © 2013 LogiGear Corporation. All Rights Reserved 2
  • 5. 4/11/2013 Topics for today Automation Designing and organizing tests Executing tests Team, organization and process Off-shoring, globalization © 2013 LogiGear Corporation. All Rights Reserved What is "BIG" Big efforts in development, automation, execution and/or follow up It takes a long and/or capacity to run tests (lot of tests, lot of versions, lot of configurations, ...) Scalability, short term and long term Complexity, functional, technical Number and diversity of players and stakeholders − pigs, chicken, elephants, ankle biters, ... Various definitions of "big" possible... and relevant... − "10 machines" or "10 acres" − "1000 tests" or "1000 weeks of testing" Big today means: big for you − "non trivial", you need to think about it "Windows 8 has undergone more than 1,240,000,000 hours of testing" Steven Sinofsky, Microsoft, 2012 © 2013 LogiGear Corporation. All Rights Reserved 3
  • 6. 4/11/2013 Existential Questions Why test? Why not test? Why automate tests? Why not automate tests? © 2013 LogiGear Corporation. All Rights Reserved Why test? People expect us to do Somebody wants us to Increases certainty and control − Showing absence of problems Finds faults, saving time, money, damage − Showing presence of problems © 2013 LogiGear Corporation. All Rights Reserved 4
  • 7. 4/11/2013 Why not test? It costs time and money You might find problems . . . We forgot to plan for it We need the resources for development It is difficult It's hard to manage © 2013 LogiGear Corporation. All Rights Reserved Why Automate Tests? It is more fun Can save time and money − potentially improving time-to-market Can capture key application knowledge in a reusable way Consolidates a structured way of working − when established as integral part of system development process Can speeds up development life cycles Execution typically is more reliable − a robot is not subjective © 2013 LogiGear Corporation. All Rights Reserved 5
  • 8. 4/11/2013 The Power of Robot Perception FINISHED FILES ARE THE RE SULT OF YEARS OF SCIENTI FIC STUDY COMBINED WITH THE EXPERIENCE OF YEARS... © 2013 LogiGear Corporation. All Rights Reserved Why not Automate? Can rule out the human elements − promotes "mechanical" testing − might not find "unexpected" problems More sensitive to good practices − pitfalls are plentiful maintenance can crush automation... Creates more software to manage Needs/uses technical expertise in the test team Tends to dominate the testing process − at the cost of good test development © 2013 LogiGear Corporation. All Rights Reserved 6
  • 9. 4/11/2013 The Power of Human Perception Olny srmat poelpe can raed tihs. I cdnuolt blveiee taht I cluod aulaclty uesdnatnrd waht I was rdanieg. The phaonmneal pweor of the hmuan mnid, aoccdrnig to a rscheearch at Cmabrigde Uinervtisy, it deosn't mttaer in waht oredr the ltteers in a wrod are, the olny iprmoatnt tihng is taht the frist and lsat ltteer be in the rghit pclae. The rset can be a taotl mses and you can sitll raed it wouthit a porbelm. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef, but the wrod as a wlohe. © 2013 LogiGear Corporation. All Rights Reserved About tests in big projects Regular tests may be activities, complex tests are products. In fact any test that you want to run more than once is a product Every test that is written down with sufficient detail should be automated Automation − No longer an option in most situations − Also a key prerequisite of most agile approaches How tests are written and automated can make or break large scale testing © 2013 LogiGear Corporation. All Rights Reserved 7
  • 10. 4/11/2013 Keywords, essential for scalability Distinguish tasks for test development and for automation The test developer creates tests using "actions" (my term) Each action consists of a keyword and arguments The automation task focuses on automating the actions Each action is automated only once fragment from a test with actions number actions, each with a keyword and arguments name new product P-9009 Sledge Hammer 5 number quantity add quantity add quantity add quantity P-9009 P-9009 P-9009 20 3 6 number check quantity P-9009 quantity 34 quantity read from top to bottom "34" is the expected value here © 2013 LogiGear Corporation. All Rights Reserved Potential benefits of keywords More tests, better tests − more breadth − more depth Fast, results can be quickly available − the design directly drives the automation Separates the tests from the technical scripting language − easier to involve business subject matter experts − the action format allows for easy readability Less efforts for automation − "script free" in most cases Automation more stable and maintainable − limited and manageable impact of changes in the system under test Develop tests more early in the life cycle − deal with execution details later ... © 2013 LogiGear Corporation. All Rights Reserved 8
  • 11. 4/11/2013 Risks of keywords Often seen as silver bullet, complications are underestimated − often treated as a technical "trick" − testers can get squeezed and marginalized • developers and users dictating tests • automation engineers dictating actions − or testers get the automation responsibility, thus becoming pseudo programmers The method needs understanding and experience to be successful − pitfalls are many, and can have a negative effect on the outcome Lack of method and structure can risk manageability − maintainability not as good as hoped − results can be disappointing, approach will be blamed © 2013 LogiGear Corporation. All Rights Reserved Case: International Financial Project One of the largest projects to date with action words Over 10 000 windows, meant for use in 85 countries Long development cycle (400 pp, 4 years and counting) Maintenance very hard Testing major bottleneck Much investment in automation techniques were needed to become successful Also a lot of attention for team and work environment helped the success Team of 35 test developers, 2 automation engineers © 2013 LogiGear Corporation. All Rights Reserved 9
  • 12. 4/11/2013 Keywords need a method By themselves keywords don't provide much scalability − they can even backfire and make automation more cumbersome − a method can help tell you which keywords to use when, and how to organize the process Today we'll look at Action Based Testing (ABT) − addresses test management, test development and automation − large focus on test design as the main driver for automation success Central deliveries in ABT are the "Test Modules" − developed in spreadsheets − each test module contains "test objectives" and "test cases" − each test module is a separate (mini) project, each test module can involve different stake holders © 2013 LogiGear Corporation. All Rights Reserved Example of an ABT test module Consists of an (1) initial part, (2) test cases and (3) a final part Focus is on readability, and a clear scope Navigation details are avoided, unless they're meant to be tested TEST MODULE Car Rental Payments user start system john TEST CASE TC 01 Rent some cars first name last name car John John Doe Doe Ford Escape Chevvy Volt last name amount Doe 140.4 rent car rent car check payment FINAL close application © 2013 LogiGear Corporation. All Rights Reserved 10
  • 13. 4/11/2013 Example of a "low level" test module In "low level" tests interaction details are not hidden, since they are the target of the test The right level of abstraction depends on the scope of the test, and is an outcome of your test design process TEST MODULE Screen Flow user start system john TEST CASE TC 01 "New Order" button first name control main new order click window check window exists new order FINAL close application © 2013 LogiGear Corporation. All Rights Reserved Re-use actions to make new actions In the below example we use another sheet, but if you code actions, you could do something similar Often low level tests are re-used into these action definitions ACTION DEFINITION check payment user check control value main last name # last name control main view balance window click Jones window enter default value last name amount window argument argument control expected main balance # amount © 2013 LogiGear Corporation. All Rights Reserved 11
  • 14. 4/11/2013 Action Based Testing Test Development Plan Break down Test Module 1 Test Module 2 Test Module N Test Objectives Test Objectives Test Objectives Test Cases Test Cases ... Test Cases Automate Actions ACTION AUTOMATION © 2013 LogiGear Corporation. All Rights Reserved Case: Stock Exchange Transition from floor-based to screen-based trade The Four Daltons (French comic book characters) Created on basis of an existing standard package − result: very little specifications Consisting of four major, different, systems that need to work in realtime Failures and bugs are not an option: − core of the financial system of the country, 100K revenue per second − traders not necessarily following rules In-depth knowledge limited to four people − nicknamed "The Four Daltons", after characters in a French comic book series about the wild west − none of the four Daltons was involved in testing, testing was in a vacuum Three months to go... − − − − test development (and scripted automation) had failed test department not cooperating well with developers and domain experts internal and external auditors had raised the alarm and... the Dutch Crown Prince was scheduled to put the system into use!! © 2013 LogiGear Corporation. All Rights Reserved 12
  • 15. 4/11/2013 Case: Stock Exchange Test set: − make it comprehensive − make it in-depth and aggressive − make it easy to assess and approve Organization: − get the right people involved (testing, automation, etc) − use scarce resources efficiently (in particular the four Daltons) − work with stake holders to let the process be transparent Technical: − use of the keyword method ("action words") − use "test objectives" so auditors can see quickly what you're testing − use great test design, don't mix apples and oranges "Sign off lubrication": − auditors signed off on the tests, not the test results − "the test is complete", not "the system works well" Results: − deadline was met one day before final date − the automated tests were the only ones used for acceptance − no functional errors found afterwards © 2013 LogiGear Corporation. All Rights Reserved Question What is wrong with the following pictures? © 2013 LogiGear Corporation. All Rights Reserved 13
  • 16. 4/11/2013 No Y2K Problems in Auckland Airport?? © 2013 LogiGear Corporation. All Rights Reserved Anything wrong with this instruction ? You should change your battery or switch to outlet power immediately to keep from losing your work. © 2013 LogiGear Corporation. All Rights Reserved 14
  • 17. 4/11/2013 Why Better Test Development? Are your suffering from lame tests too? Many tests are often mechanical now − − − − blindly follows specs or reqs which often suits ok, but lacks aggression no combinations, no unexpected situations "methodical" does not have to mean "mechanical" For a higher “ambition level” you need − understanding of the system under test, and the business under test − analytical understanding of what could go wrong − creativity, and the commitment to use it Poor test development results in − cumbersome automation due to lack of focus − tedious retest cycles, loosing the agile advantage © 2013 LogiGear Corporation. All Rights Reserved Test Design Effective test breakdown (into test modules) − make sure every test module has a clear focus − keep different kinds and levels of tests separate Right level of actions − as “high level” if possible, hiding as many details as much as possible − but not if the details are relevant for the test It is my believe that successful automation is not a technical challenge. It is most of all a test design challenge. © 2013 LogiGear Corporation. All Rights Reserved 15
  • 18. 4/11/2013 Case: A Financial Project Large project, with many consultants − embraced the approach − however, they were from our competitor One of the first with action words However, the project was confident about their test development, no help needed Result: many very hard to maintain tests, and way too many action words − crushing complexity − almost the end of the action words method − one memo saved the day © 2013 LogiGear Corporation. All Rights Reserved The Three “Holy Grails” of Test Design Metaphor to depict three main steps in test design Using "grail" to illustrate that there is no one perfect solution, but that it matters to pay attention (to search) About quality of tests, but most of all about scalability and maintainability in BIG projects Organization of tests into test modules Right approach for each test module Proper level of detail in the test specification © 2013 LogiGear Corporation. All Rights Reserved 16
  • 19. 4/11/2013 Case for organizing tests in BIG projects Can help keep the volume down Isolate the complexities Efficient and re-usable automation Deal with changing requirements For example: much of tested subject matter is not system specific, but business specific − a mortgage is a mortgage © 2013 LogiGear Corporation. All Rights Reserved What's the trick... © 2013 LogiGear Corporation. All Rights Reserved 17
  • 20. 4/11/2013 What's the trick... Have or acquire facilities to store and organize you content Edit your stuff Decide where to put what − assign and label the shelves Put it there If the organization is not sufficient anymore, add to it or change it © 2013 LogiGear Corporation. All Rights Reserved Properties of a good Breakdown Test modules are well differentiated and clear in scope Reflects the level of tests Balanced in size and amount Modules are mutually independent Fit the priorities and planning of the project © 2013 LogiGear Corporation. All Rights Reserved 18
  • 21. 4/11/2013 Breakdown Criteria Straightforward Criteria − Architecture of the system under test (client, server, protocol, sub systems, components, modules, ...) − Functionality (customers, finances, management information, ...) − Kind of test (navigation flow, negative tests, response time, ...) − Ambition level (smoke test, regression, aggressive, ) Additional Criteria − − − − − Stakeholders (like "Accounting", "Compliance", "HR", ...) Complexity of the test (put complex tests in separate modules) Technical aspects of execution (special hardware, multi-station, ...) Overall project planning (availability of information, timelines, sprints, ...) Risks involved (extra test modules for high risk areas) © 2013 LogiGear Corporation. All Rights Reserved Example breakdown Tests of user interface − − − does function key F4 work does listbox xyz the right values is the tab order correct Form Tests, do all the forms (dialogs, screens, pages) work: − − − can data be entered and is it stored well is displayed data correct split these from everything else Function tests, do individual functions work − can I count the orders Alternate paths in use cases − can I cancel a transaction End-to-end tests − − do all components of a system work well together in implementing the business processes like enter sale order, then check inventory and accounting Tests with specific automation needs − like multi station tests Tests of non-UI functions High ambition level tests (aggressive tests) − can I break the system under test If in doubt: try high level first © 2013 LogiGear Corporation. All Rights Reserved 19
  • 22. 4/11/2013 What is probably not a good design Navigational and functional tests are mixed − for example "over checking": a test of a premium calculation also checks the existence of a window You have to change all of them for every new release of the system under test All test modules have a similar design Test modules are dependent on each other You can’t start developing any test modules early in the life cycle © 2013 LogiGear Corporation. All Rights Reserved Symptoms Tediousness in the test and test automation process No sense of control Complaining people Unnecessary high test maintenance − changes in the system under test impact many tests − hard to understand which tests need to be modified Difficulties in running any test − teams start "debugging" tests © 2013 LogiGear Corporation. All Rights Reserved 20
  • 23. 4/11/2013 Example of an application under test Various item types − − − − − − tests actions interface definitions data sets folders ... Various operations − − − − open cut, copy, paste check out ... Various ways to initiate an operation − − − − − − − − context menu, with or without accelerator key main menu, with or without accelerator key toolbar short cut key function key drag and drop double click .... © 2013 LogiGear Corporation. All Rights Reserved Defining some modules Test modules for operations − − − − − primary and alternate paths various values for fields like "comment" in check-in paste in other projects copy and paste various groups not necessarily on each item type Test modules for items − address all item types at least once − on each item type perform each operation − not necessarily each variant of each operation UI handling − try for UI command if it starts the intended operation − not necessarily on each item type or operation variant Concurrency − − − − try concurrency sensitive operations with multiple stations in varying concurrency scenarios, with and without local "refreshes" not necessarily each item type or operation variant certainly not each UI command included © 2013 LogiGear Corporation. All Rights Reserved 21
  • 24. 4/11/2013 Questions for Test Design Does your organization make something like a high level test design? If yes, how do you document it? © 2013 LogiGear Corporation. All Rights Reserved Case Study Large IT provider New version of one of their major web-sites Test scope was user acceptance test (functional acceptance) − the users were the “business owners” Development was off-shore © 2013 LogiGear Corporation. All Rights Reserved 22
  • 25. 4/11/2013 Case Study Test development was done separate from automation − time-line for test development: May – Oct − time-line for automation (roughly): Jan – Feb All tests were reviewed and approved by the business owners − acceptance was finished by the end of the test development cycle © 2013 LogiGear Corporation. All Rights Reserved Example of a Test Development Plan Nr 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Module Portal Navigation, Audience Portal Navigation, Search Membership, registration Portal Navigation, Category Portal Navigation, Topic and Expert Access Control Portal Navigation, Task Contact DSPP Portal search Membership, review and update Program contact assignment Company, registration Catalog, view and query Site map Membership, affiliation Learn about DSPP Products and services What's new Company, life cycle Specialized programs Customer surveys Software downloads Newsletters Internationalization and localization Membership, life cycles Collaboration, forums Collaboration, blogs Collaboration, mailing lists Business Owner Date to BO Robyn Peterson 05 / 23 Ted Jones 05 / 27 Steve Shao 06 / 03 Ted Jones 06 / 08 Ted Jones 06 / 13 Mike Soderfeldt 06 / 17 Ted Jones 06 / 22 Ted Jones 06 / 27 Mike Soderfeldt 07 / 01 Steve Shao 07 / 05 Alan Lai 07 / 11 Steve Shao 07 / 14 Robyn Peterson 07 / 19 Ted Jones 07 / 25 Steve Shao 07 / 28 Ted Jones 08 / 01 Steve Shao, Robyn Peterson 08 / 08 Ted Jones 08 / 11 Steve Shao, Alan Lai 08 / 17 Ted Jones, Steve Shao 08 / 22 Ted Jones 08 / 29 Mike Soderfeldt 09 / 01 Ted Jones 09 / 06 Ted Jones 09 / 13 Steve Shao 09 / 19 Ted Jones 09 / 23 Mike Soderfeldt 09 / 28 Ted Jones 10 / 03 © 2013 LogiGear Corporation. All Rights Reserved 23
  • 26. 4/11/2013 Review Process with Stake Holders START Test Team sends draft Module to Stake Holder Stake Holder reviews: - coverage - correctness changes needed? no Stake Holder returns notice of approval yes Test Team receives and processes notes Stake Holder returns notes: - additions - corrections Test Team marks the Module as "Final" END © 2013 LogiGear Corporation. All Rights Reserved Case Study, Results All tests were developed and reviewed on schedule − many notes and questions during test development phase The automation was 100% of the tests − all actions were automated, thus automating all test modules The test development took an estimated 18 person months − one on-shore resource, two off-shore resources The automation took between one and two months − focused on actions − most time was spent in handling changes in the interface (layout of pages etc) © 2013 LogiGear Corporation. All Rights Reserved 24
  • 27. 4/11/2013 Case: The French Director Mid size company Struggling under high pressure Testing of their main product, standard financial software Control and priority main issue Unfamiliar business culture Main instrument: module break down © 2013 LogiGear Corporation. All Rights Reserved Test Modules versus Test Cases The test module is a bigger unit in the test design − easier to identify − a chapter rather than a paragraph − easier to plan and manage, as a product (can be treated as part of product backlog in scrum projects) Better flow of execution − each test case can set up for the next one − keep test modules independent, test cases can be dependent Test cases become creative output, rather than stifling input − avoids having to define all test cases at once early in the process Clear scope helps to identify cases, actions and checks − using "test objectives" to further detail scope − had a significant effect on maintainability © 2013 LogiGear Corporation. All Rights Reserved 25
  • 28. 4/11/2013 "Thou Shall Not Debug Tests..." Large and complex test projects can be hard to "get to run" If they are however, start with taking a good look again at your test design... Rule of thumb: don't debug tests. If tests don't run smoothly, make sure: − lower level tests have been successfully executed first -> UI flow in the AUT is stable − actions and interface definitions have been tested sufficiently with their own test modules -> automation can be trusted − are you test modules not too long and complex? © 2013 LogiGear Corporation. All Rights Reserved Grail 2: Approach per Test Module Plan the test module: − when to develop: is enough specification available − when to execute: make sure the functionality at action level is welltested and working already Process: − − − − do an intake: understand what is needed and devise an approach analyze of requirements formulate "test objectives" create "test cases" Identify stakeholders and their involvement: − users, subject matter experts − developers − auditors Choose testing techniques if applicable: − boundary analysis, decision tables, transition diagrams, soap opera testing, ... © 2013 LogiGear Corporation. All Rights Reserved 26
  • 29. 4/11/2013 Eye on the ball, Scope Always know the scope of the test module The scope should be unambiguous The scope determines many things: − − − − what the test objectives are which test cases to expect what level of actions to use what the checks are about and which events should generate a warning or error (if a “lower” functionality is wrong) © 2013 LogiGear Corporation. All Rights Reserved What I have seen not work "Over-Checking": having checks that don't fit the scope of the test Forcing data driven: making all tests data driven (variables, data files) without clear reason Combinatorial explosions: test all ... for all ... in all ... All actions high level (or all actions low level) Many tests for forms and dialogs, little tests for business processes Abundance of irrelevant comments, and lack of relevant comments © 2013 LogiGear Corporation. All Rights Reserved 27
  • 30. 4/11/2013 Detail out the scope with test objectives ... TO-3.51 The exit date must be after the entry date ... test objective TO-3.51 enter employment check error message Bill Goodfellow 2002-10-02 2002-10-01 The exit date must be after the entry date. name entry date exit date © 2013 LogiGear Corporation. All Rights Reserved Examples of Testing Techniques Equivalence class partitioning − any age between 18 and 65 Boundary condition analysis − try 17, 18, 19 and 64, 65, 66 Error guessing − try Cécile Schäfer to test sorting of a name list Exploratory − "Exploratory testing is simultaneous learning, test design, and test execution", James Bach, www.satisfice.com Error seeding − deliberately injecting faults in a test version of the system, to see if the tests catch them − handle with care, don't let the bugs get into the production version Decision tables − define possible situations and the expected responses of the system under test State transition diagrams − identify "states" of the system, and have your tests go through each transition between states at least once Jungle Testing − focus on unexpected situations, like hacking attacks Soap Opera Testing − describe typical situations and scenarios in the style of episodes of a soap opera, with fixed characters − high density of events, exaggerated − make sure the system under test can still handle these © 2013 LogiGear Corporation. All Rights Reserved 28
  • 31. 4/11/2013 "Jungle Testing" Expect the unexpected − − − − unexpected requests unexpected situations (often data oriented) deliberate attacks how does a generic design respond to a specific unexpected event? Difference in thinking − coding bug: implementation is different from what was intended/specified − jungle bug: system does not respond well to an unexpected situation To address − − − − − − − study the matter (common hack attacks, ...) make a risk analysis make time to discuss about it (analysis, brainstorm) involve people who can know use "exploratory testing" (see James Bach's work on this) use an agile approach for test development applies equally to testing, requirements, design • testing should focus on the specific attacks − consider randomized testing, like "monkey" testing © 2013 LogiGear Corporation. All Rights Reserved Soap Opera Testing Informal scenario technique to invite subject-matter experiences into the tests, and efficiently address multiple objectives Using a recurring theme, with “episodes” About “real life” But condensed And more extreme Typically created with a high involvement of end-users and/or subject-matter experts © 2013 LogiGear Corporation. All Rights Reserved 29
  • 32. 4/11/2013 Lisa Crispin: Disorder Depot . . . There are 20 preorders for George W. Bush action figures in "Enterprise", the ERP system, awaiting the receipt of the items in the warehouse. Finally, the great day arrives, and Jane at the warehouse receives 100 of the action figures as available inventory against the purchase order. She updates the item record in Enterprise to show it is no longer a preorder. Some time passes, during which the Enterprise background workflow to release preorders runs. The 20 orders are pick-released and sent down to the warehouse. Source: Hans Buwalda, Soap Opera Testing (article), Better Software Magazine, February 2005 © 2013 LogiGear Corporation. All Rights Reserved Lisa Crispin: Disorder Depot . . . Then Joe loses control of his forklift and accidentally drives it into the shelf containing the Bush action figures. All appear to be shredded to bits. Jane, horrified, removes all 100 items from available inventory with a miscellaneous issue. Meanwhile, more orders for this very popular item have come in to Enterprise. Sorting through the rubble, Jane and Joe find that 14 of the action figures have survived intact in their boxes. Jane adds them back into available inventory with a miscellaneous receipt. © 2013 LogiGear Corporation. All Rights Reserved 30
  • 33. 4/11/2013 Lisa Crispin: Disorder Depot . . . This scenario tests • Preorder process • PO receipt process • Miscellaneous receipt and issue • Backorder process • Pick-release process • Preorder release process • Warehouse cancels © 2013 LogiGear Corporation. All Rights Reserved Vary your tests? Automated tests have a tendency to be rigid, and predictable Real-world situations are not necessarily predictable Whenever possible try to vary: − with select other data cases that still fit the goal of tests − with randomized behavior of the test © 2013 LogiGear Corporation. All Rights Reserved 31
  • 34. 4/11/2013 Generation and randomization techniques Model-based − use models of the system under test to create tests − see: Harry Robinson, www.model-based-testing.org, and Hans Buwalda, Better Software, March 2003 Data driven testing − apply one test scenario to multiple data elements − either coming from a file or produce by an automation "Monkey testing" − − − − use automation to generate random data or behavior "smart monkeys" will follow typical user behavior, most helpful in efficiency "dumb monkeys" are more purely random, may find more unexpected issues long simulations can expose bugs traditional tests won't find Extended Random Regression − − − − have a large database of tests randomly select and run them, for a very long time this will expose bugs otherwise hidden see Cem Kaner e.a.: "High Volume Test Automation", StarEast 2004 © 2013 LogiGear Corporation. All Rights Reserved Data Driven Testing Separate test logic from the data Possible origins for the data: − − − − earlier steps in the test data table randomizer, or other formula external sources, like a database query Use "variables" as placeholders in the test case, instead of hard values Data driven is powerful, but use modestly: − value cannot be known at test time, or changes over time − having many data variations is meaningful for the test © 2013 LogiGear Corporation. All Rights Reserved 32
  • 35. 4/11/2013 Variables and expressions with keywords TEST CASE TC 02 car last name car John John Doe Doe Chevvy Volt Chevvy Volt car check quantity >> volts first name rent car rent car available Chevvy Volt get quantity Rent some more cars expected Chevvy Volt # volts - 2 This test does not need an absolute number for the available cars, just wants to see if a stock is updated As a convention we denote an assignment with ">>" The "#" indicates an expression © 2013 LogiGear Corporation. All Rights Reserved Data driven testing with keywords TEST CASE TC 03 Check stocks use data set /cars car get quantity available check quantity >> quantity last name car # first # last # car car rent car # car first name expected # car DATA SET cars car data set first last value Chevvy Volt Ford Escape Chrysler 300 Buick Verano BMW 750 Toyota Corolla John Mary Jane Tom Henry Vivian Doe Kane Collins Anderson Smyth Major 40000 22500 29000 23000 87000 16000 # quantity - 1 repeat for data set The test lines will be repeated for each row in the data set The values represented by "car", "first" and "last" come from the selected row of the data set © 2013 LogiGear Corporation. All Rights Reserved 33
  • 36. 4/11/2013 Combinations Input values − determine equivalence classes of values for a variable or field − for each class pick a value (or randomize) Options, settings Configurations − operating systems, operating system versions and flavors • Windows service packs, Linux distributions − − − − browsers, browser versions protocol stacks (IPv4, IPv6, USB, ...) processors DBMS's Combinations of all of the above Trying all combinations will spin out of control quickly © 2013 LogiGear Corporation. All Rights Reserved Pairwise versus exhaustive testing Group values of variables in pairs (or tuples with more than 2) Each pair (tuple) should occur in the test at least once − maybe not in every run, but at least once before you assume "done" − consider to go through combinations round-robin, for example pick a different combination every time you run a build acceptance test − in a NASA study: • • • 67 percent of failures triggered by a single value 93 percent by two-way combinations, and 98 percent by three-way combinations Example, configurations − operating system: Windows XP, Apple OS X, Red Hat Enterprise Linux − browser: Internet Explorer, Firefox, Chrome − processor: Intel, AMD − database: MySQL, Sybase, Oracle − 72 combinations possible, to test each pair: 10 tests Source: PRACTICAL COMBINATORIAL TESTING, D. Richard Kuhn, Raghu N. Kacker, Yu Lei, NIST Special Publication 800-142, October, 2010 Example of tools: − ACTS from NIST, PICT from Microsoft, AllPairs from James Bach (Perl) − for a longer list see: www.pairwise.org These techniques and tool are supportive only. Often priorities between platforms and values can drive more informed selection Rights Reserved © 2013 LogiGear Corporation. All 34
  • 37. 4/11/2013 Grail 3: Specification Level, choosing actions Scope of the test determines the specification level As high level as appropriate, as little arguments as possible − Use default values for non-relevant arguments Clear names (usually verb + noun usually works well) − to standardize action names: standardize both the verbs and the nouns, so "check customer" versus "verify client" (or vice versa) − tests are not C++ code: avoid "technical habits", like mixed case and (worse) underlines Manage the Actions Document the Actions By-product of the test design © 2013 LogiGear Corporation. All Rights Reserved Case: American Bank Project for a new teller system Large, state of the art Many system releases, many adjustments Need for very high level of automation Over 1 million test lines, in over 650 test modules Initially little attention paid to "holy grails" − UI and functional tests in the same modules − virtually un-maintainable, came close to killing the project − test design forced upon the team by a powerful stakeholder who did not care much for methods... Emergency re-organization of the test modules − after system changes the tests would run again within a day © 2013 LogiGear Corporation. All Rights Reserved 35
  • 38. 4/11/2013 Example of using actions In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function • the "#" means an expression, in this case a variable • the ">>" assign to a variable for use later on in the test key key navigate key navigate F7 3 page tab locate page tab Scan Criteria w indow wait for controls loaded search text check breadcrumb general functions > search w indow scan direction Backward w indow enter value control search select value control value search control search click business date match # bus date source go w indow wait for controls loaded search results w indow get control search results sequence number variable >> seq num © 2013 LogiGear Corporation. All Rights Reserved Example of using actions In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function • the "#" means an expression, in this case a variable • the ">>" assign to a variable for use later on in the test variable get sequence number >> seq num © 2013 LogiGear Corporation. All Rights Reserved 36
  • 39. 4/11/2013 Mid level actions Most tests will have low level and high level actions − low level: generic operations, know the interface, don't know the functionality • examples: "selection menu item", "expand tree node", ... − high level: business oriented operations, know the functionality, don't know the interface • examples: "enter purchase order", "check inventory of article" For complex forms (dialog) with many input fields consider using "mid level" actions − an argument for each field − for use in high level actions enter customer enter address fields Examples of mid-level actions: − "enter address fields" − "check address fields" ... enter select set ... © 2013 LogiGear Corporation. All Rights Reserved Mapping an interface entity (like a window) INTERFACE ENTITY balance inquiry interface entity setting title Balance inquiry ta name interface element interface element ta class label last name first name client id text text text Last name: First name (optional): Client id (optional): ta name interface element interface element interface element ta class caption View Balance Close ta name interface element view balance button close button ta class global pos balance label label 5 An interface mapping will map windows and controls to names When the interface of an application changes, you only have to update this in one place The interface mapping is a key step in your automation success, allocate time to design it well © 2013 LogiGear Corporation. All Rights Reserved 37
  • 40. 4/11/2013 Some Tips to Get Stable Automation Make the system under test automation-friendly Use "active" timing Test your automation Use automation to identify differences between versions of the system under test Keep an eye on the test design © 2013 LogiGear Corporation. All Rights Reserved Automation-friendly design: hidden properties Look for properties a human user can't see, but a test tool can This approach is a must-do for speedier and more stable automation − − − − interface mapping is often bottleneck, and source of maintenance problems with predefined identifying property values an interface map can be created without "spy" tools not sensitive to changes in the system under test not sensitive to languages and localizations Examples: − − − "id" attribute for HTML elements "name" field for Java controls "AccessibleName" property in .Net controls (see below) © 2013 LogiGear Corporation. All Rights Reserved 38
  • 41. 4/11/2013 Active Timing Passive timing − wait a set amount of time − in large scale testing, try to avoid passive timing altogether: • if wait too short, test will be interrupted • if wait too long, time is wasted Active timing − wait for a measurable event − usually the wait is up to a, generous, maximum time − common example: wait for a window or control to appear (usually the test tool will do this for you) Even if not obvious, find something to wait for... Involve developers if needed − relatively easy in an agile team, but also in traditional projects, give this priority If using a waiting loop − make sure to use a "sleep" function in each cycle that frees up the processor (giving the AUT time to respond) − wait for an end time, rather then a set amount of cycles © 2013 LogiGear Corporation. All Rights Reserved Active Timing, your situation How much passive timing do you have in your scripts? If you're not sure, find out... ... and let me know "First action I took upon my return was to evaluate the percentage of passive time in our code and found passive time 68% versus active time 32%. Needless to say our automation test cases were very expensive time operations and now I know why..." Raed Atawneh, 2012 (extract) © 2013 LogiGear Corporation. All Rights Reserved 39
  • 42. 4/11/2013 Things to wait for... Wait for a last control or elements to load − developers can help knowing which one that is Non-UI criteria − API function − existence of a file Criteria added in development specifically for this purpose, like: − "disabling" big slow controls (like lists or trees) until they're done loading − API functions or UI window or control properties Use a "delta" approach: − every wait cycle, test if there was a change; if no change, assume that the loading time is over: − examples of changes: • the controls on a window • count of items in a list • size a file (like a log file) © 2013 LogiGear Corporation. All Rights Reserved Alternatives to UI automation ("non-GUI") A GUI (Graphical User Interface) is only one example of an interface for interaction with a system under test Examples − − − − − − − − − HTTP and XML based interfaces, like REST application programming interfaces (API’s) embedded software protocols files, batches databases command line interfaces (CLI’s) multi-media mobile devices In many cases non-GUI automation is used since there simply is not GUI, but it can also often speed things up: − − tends to be more straightforward technically, little effort needed to build up or maintain once it works, it tends to work much faster and more stably than GUI automation In BIG testing projects routinely: − − identify which non-GUI alternatives are available as part of test planning: identify which tests qualify for non-GUI automation © 2013 LogiGear Corporation. All Rights Reserved 40
  • 43. 4/11/2013 Technical Complexity Technology is another dimension that can make a project "complex" Examples: − graphics, charts, 3D, ... − hard to access systems, like embedded software, iOS, Flash, dedicated hardware − difficult protocols, like SS7, transactions servers Approach: isolate the technical problems − − − − embed in functions and actions let experts look at them tackle early in a project, since impact is large once resolved, no longer center stage © 2013 LogiGear Corporation. All Rights Reserved The importance of innovation Large and complex testing projects pose many challenges Initial discussion of approach is a key requisite − − − − − − thinking before doing tackling technologies agreeing on methods and practices who does what, who needs to be involved high level test design debate the problems, not just the solutions However, also plan for continuous improvements − − − − this is at the heart of agile thinking, and it applies very much to big testing never stop thinking, "there is always one more trick" share the tricks, other teams may like them too improvements can apply to test design, to automation techniques, or even to how to organize the work © 2013 LogiGear Corporation. All Rights Reserved 41
  • 44. 4/11/2013 Tools that can help manage BIG projects Application Lifecycle Management (ALM) − − − − abundant now, mainly on the wings of agile very good for control, team cooperation, and traceability often relate to IDE's (like Microsoft TFS and Visual Studio) examples: Rally, Jira, TFS Test Managers − − − as separate tools on their way out morphing into or replaced by ALM options examples: HP Quality Center, Microsoft Test Manager Test development and automation tools − − develop and/or automate tests • these are not the same, automation tools are not always so good for test development examples are HP Quick Test Pro, Borland Silk, Selenium, FitNesse, Microsoft Coded UI, and LogiGear's TestArchitect and TestArchitect for Visual Studio (our own products) Build tools − − − − succeed the traditional "make" tools in particular "continuous build" tools combine "make" functionality with source control systems to rebuild components that have changed, either continuously or on set times, like nightly can very well also run related tests (unit and functional), and act on the results (stop build, report, etc) examples: Hudson, Jenkins, TFS Bug trackers − − − not only register issues, but also facilitate their follow up, with workflow features often also part of other tools, and tend to get absorbed now by the ALMs Examples: BugZilla, Mantis, Trac © 2013 LogiGear Corporation. All Rights Reserved Tooling and Traceability ALM, IDE, Project Mgr, Req Mgr Test Development Tool Automation Tool Execution Manager Continous Build Tool Issue Tracker ALM Execution Result Bug, issue Test Module Reference item (ALM item, req, code module, ...) Test Objective Test Case Testing Trace back © 2013 LogiGear Corporation. All Rights Reserved 42
  • 45. 4/11/2013 Test Execution Have an explicit approach for when and how to execute which tests Having a good high level test design will help to organize this Execution can be selective or integral − unit tests are typically executed selectively, possibly automatically based on code changes in a system like SVN or TFS − for functional tests, decisions are needed: • selective execution will be quicker and more efficient • integral execution may catch more issues ("bonus bugs") • generally extensive functional test execution will be related to releases, rather than code check ins − the ability to run "big testing" efficiently may determine how much can be done © 2013 LogiGear Corporation. All Rights Reserved Environments, configurations Many factors can influence details of automation − − − − language, localization hardware version of the system under test system components, like OS or browser Test design can reflect these − certain test modules are more general − others are specific, for example for a language But for tests that do not care about the differences, the automation just needs to "deal" with them − shield them from the tests © 2013 LogiGear Corporation. All Rights Reserved 43
  • 46. 4/11/2013 "Variations" Capture variations of the system under test in the actions and interface definitions, rather than in the tests (unless relevant there). Can be a feature in a test playback tool, or something you do with a global variable or setting. . . . Actions, Interface Definitions "Master Switch" Variation Variation Variation © 2013 LogiGear Corporation. All Rights Reserved Possible set up of variations linked variation keyworded variation Specify for example in a dialog when you start an execution: © 2013 LogiGear Corporation. All Rights Reserved 44
  • 47. 4/11/2013 Test Environments Physical • • • • hardware infrastructure location ... • costs money • can be scarce Software • • • • programs data models protocols ... • configurations Data • initial data • parameters / tables • ... • availability • manageability © 2013 LogiGear Corporation. All Rights Reserved Dealing with data Constructed data is easier to manage − can use automation to generate it, and to enter it in the environment − result of test analysis and design, reflecting "interesting" situations − however, less "surprises": real life situations which were not foreseen Real-world data is challenging to organize − make it a project, or task, in itself − make absolutely sure to deal with privacy, security and legal aspects appropriately • study this, ask advice • apply appropriate "scrubbing" Consider using automation to select data for a test − set criteria ("need a male older than 50, married, living in Denver"), query for matching cases, and select one randomly (if possible a different one each run) − this approach will introduce variation and unexpectedness, making automated tests stronger and more interesting © 2013 LogiGear Corporation. All Rights Reserved 45
  • 48. 4/11/2013 Unattended testing... When a test cannot pass, it can be: − a difference between expected and recorded values or behavior, as a result of a check designed by the tester: this is a fail − the automation encounters a problem, like a window or control doesn't show, that is not part of a check: this is an error An error can disrupt the test flow, and you may want to catch and handle it properly: − skip smaller or larger parts of the ongoing test − bring the system back in a known state (typically: close any open windows, go to the main screen) − make sure the report clearly indicates these kind of problems, to avoid false positives − example "on error action" that executes a predefined action that will do recovery However, better is to avoid these situations − lots of efforts needed for unattended testing should raise questions about test design or quality of the automation ("thou shall not debug tests") © 2013 LogiGear Corporation. All Rights Reserved "Known bug" problem Not uncommon in large scale systems − typically related to a version of the system under test A known bug may: − generate fails you want to ignore, also in statistics − throw off automation If many known bug situation occur, take another look at your high level test design One possible workaround, a "known bug" action: − other alternatives: conditionally ignore steps or single check points version known bug ... end known bug 1.1 © 2013 LogiGear Corporation. All Rights Reserved 46
  • 49. 4/11/2013 Virtualization Virtual machines rather than physical machines − allow "guest" systems to operate on a "host" system − host can be Windows, Linux, etc, but also a specialized "hypervisor" − the hypervisor can be "hosted" or "bare metal" Main providers: − − − − VMWare: ESX and ESXi Microsoft: Hyper-V Oracle/Sun: Virtual Box Citrix: Xen (open source) Hardware support gets common now − processor, chipset, i/o − Like Intel's i7/Xeon For most testing purposes you need virtual clients, not virtual servers − most offerings in the market currently target virtual servers, particularly data centers Virtual clients will become more mainstream with the coming of VM's as part of regular operating systems − Windows 8: Hyper-V − Linux: KVM © 2013 LogiGear Corporation. All Rights Reserved Virtualization, a testers dream... In particular for functional testing Much easier to define and create needed configurations − you basically just need storage − managing this is your next challenge One stored configuration can be re-used over and over again The VM can always start "fresh", in particular with − fresh base data (either server or client) − specified state, for example to repeat a particular problematic automation situation Can take "snap shots" of situations, for analysis of problems Can use automation itself to select and start/stop suitable VM's − for example using actions for this − or letting an overnight or continuous build take care of this © 2013 LogiGear Corporation. All Rights Reserved 47
  • 50. 4/11/2013 Virtualization, bad dream? Performance, response times, capacities Virtual machine latency can add timing problems − see next slide − can be derailing in big test runs Management of images − images can be large, and difficult to store and move around • there can be many, with numbers growing combinatorial style • configuration in the VM can have an impact, like fixed/growing virtual disks − distinguish between managed configurations and sandboxes − define ownership, organize it − IT may be the one giving out (running) VM's, restricting your flexibility Managing running tests in virtual machines can take additional efforts on top of managing the VM's themselves − with the luxury of having VM's the number of executing machines can increase rapidly − one approach: let longer running tests report their progress to a central monitoring service (various tools have features for this) © 2013 LogiGear Corporation. All Rights Reserved Virtualization: "time is relative" Consider this waiting time loop, typical for a test script: − endTime = currentTime + maxWait − while not endTime, wait in 100 millisecond intervals When the physical machine overloads VM's can get slow or have drop outs, and endTime may pass not due to AUT latency − GetLocalTime will suffer from the latency − GetTickCount is probably better, but known for being unreliable on VM's Therefore tests that run smooth on physical machines, may not consistently do so on VM's. The timing problems are not easy to predict Possible approaches: − in general: be generous with maximum wait times if you can − don't put too many virtual machines on a physical box − consider a compensation algorithm, for example using both tick count and clock time © 2013 LogiGear Corporation. All Rights Reserved 48
  • 51. 4/11/2013 Virtual machines, capacity We started regression with 140 VMs. Very slow performance of Citrix VM clients. Key to pricing is number of VM's that can run in parallel on a physical machine An automated test execution will typically keep a VM more busy than human use Factors in determining VM/PM ratio: − − − − memory, for guest OS, AUT, test tooling storage devices (physical devices, not disk images) processors, processor cores specific hardware support (becoming more common) • processor, chipset, I/O © 2013 LogiGear Corporation. All Rights Reserved Building up virtualization Pay attention to pricing: − beefed up hardware can increase VM's/box ratio, but at a price − software can be expensive depending on features, that you may not need In a large organization, virtual machines are probably available − make sure to allocate timely (which can be long before you get there with your sprints) − keep in mind the capacity requirements Logical and physical management − which images, the wealth of possible images can quickly become hard to see forest through the trees − physical management of infrastructure is beyond this tutorial Minimum requirement: snapshots/images − freeware versions don't always carry this feature − allow to set up: OS, environment, AUT, tooling, but also: data, states © 2013 LogiGear Corporation. All Rights Reserved 49
  • 52. 4/11/2013 Infrastructure For large scale test execution this needs attention − physical infrastructure, but also how to use it Also consider managing infrastructure and test execution as a separate task − in or out of the team − avoid slowing down development (of system, test and/or automation) © 2013 LogiGear Corporation. All Rights Reserved Remote execution, servers Allowing execution separately from the machines the testers and automation engineers are working on increases scalability Large scale text execution, in particular with VM's, like to have: − − lots of processing power, lots of cores lots of memory Test execution tends to care less about: − − storage networking Test execution facilities tend to be a bottle neck very quickly in big testing projects − the teams can use whatever they can get First step up: give team members a second machine Second step up: use servers, users coordinate their use of them Third step up: major infrastructures with organized allocation © 2013 LogiGear Corporation. All Rights Reserved 50
  • 53. 4/11/2013 Tower Servers Smaller shops (smaller companies, departments) Affordable, simple, first step up from clients execution Not very scalable when the projects get larger © 2013 LogiGear Corporation. All Rights Reserved Rack Servers Well scalable Pricing not unlike tower servers Tend to need more mature IT expertise © 2013 LogiGear Corporation. All Rights Reserved 51
  • 54. 4/11/2013 Server Blades Big league infrastructure, high density, very scalable Tends to be pricey, use when space and energy matters Usually out of sight for you and your team © 2013 LogiGear Corporation. All Rights Reserved Cloud Cloud can be target of testing − normal tests, plus cloud specific tests • functional, load, response times − from multiple locations − moving production through data centers Cloud can be host of test execution − considerations can be economical or organizational − providers offer imaging facilities, similar to virtual machines − make sure machines are rented and returned efficiently Public cloud providers like EC2 offer API's, so your automation can automatically allocate and release them − be careful, software bugs can have costing consequences − for example, consider having a second automation process to double-check cloud machines have been released after a set time Note: public cloud is not taking of as fast as expected, cloud services, and private clouds, taking of much faster © 2013 LogiGear Corporation. All Rights Reserved (Xinhua Photo) 52
  • 55. 4/11/2013 Cloud Providers Source: Jack of All Clouds, January 2011 http://www.jackofallclouds.com/2011/01/state-of-the-cloud-january-201/ © 2013 LogiGear Corporation. All Rights Reserved Cloud growth source: IDC forecast, 2010 Growth of public clouds not as big as expected Cost benefits not necessarily convincing − low startup cost, but long ongoing cost See also: news.cnet.com/8301-13556_3-20063361-61.html © 2013 LogiGear Corporation. All Rights Reserved 53
  • 56. 4/11/2013 Cloud, example pricing, hourly rates Linux Small Large Extra Large Windows 0.085 0.34 0.68 0.12 0.48 0.96 1.7 GB, 1 core (32 bits) 7.5 GB, 4 cores 15 GB, 8 cores 0.50 1.00 2.00 0.62 1.24 2.48 17.1 GB, 6.5 core 34.2 GB, 13 cores 68.4 GB, 26 cores 0.17 0.68 0.29 1.16 1.7 GB, 5 core (32 bits) 7 GB, 20 cores High memory Extra Large Double Extra Large Quadruple Extra Large High CPU Medium Extra Large Source: Amazon EC2 (my interpretation, actual prices may vary) © 2013 LogiGear Corporation. All Rights Reserved Cloud, example economy small Windows buy (estimate) hours to break even months (24 / 7) large extra $0.12 $300 2,500 3.4 $0.48 $650 1,354 1.8 $0.96 $900 938 1.3 Not counting possible use of VM's within the buy option Also not counting: additional cost of ownership elements for owning or cloud (like IT management, contract and usage management) Impressions: − cloud could fit well for bursty testing needs, which is often the case − for full continuous, or very frequent, testing: consider buying − hybrid models may fit many big-testing situations: own a base capacity, rent more during peak use periods © 2013 LogiGear Corporation. All Rights Reserved 54
  • 57. 4/11/2013 Data centers can go down However, disruption could have been minimized by using multiple data centers © 2013 LogiGear Corporation. All Rights Reserved Data centers can go down This time, it did involve multiple data centers . . . © 2013 LogiGear Corporation. All Rights Reserved 55
  • 58. 4/11/2013 Data centers can go down Service providers can occasionally go down too © 2013 LogiGear Corporation. All Rights Reserved Cloud, usage for special testing needs Multi-region testing − Amazon for example has several regions • • • • • US East, Northern Virginia US West, Oregon, Northern California EU, Ireland Asia Pacific, Singapore, Tokyo South America, Sao Paulo − be careful that data transfers between regions costs money (0.01/GB) Load generation − example: "JMeter In The Cloud" • • • • based on the JMeter load test tool uses Amazon AMI's for the slave machines allows to distribute the AMI's in the different regions of Amazon see more here: aws.amazon.com/amis/jmeter-in-the-cloud-a-cloud-based-load-testing-environment © 2013 LogiGear Corporation. All Rights Reserved 56
  • 59. 4/11/2013 Questions for Infrastructure What kind of infrastructure does your organization use for testing? What is the role of virtualization, now or in the future? Are you using a private or a public cloud for testing? © 2013 LogiGear Corporation. All Rights Reserved Example of a cloud system under test source: Windows Azure reference platform © 2013 LogiGear Corporation. All Rights Reserved 57
  • 60. 4/11/2013 Approaches Automation does not have to be black box − for very big systems, a separate black box automation effort may not be efficient − and building and keeping lab situations might be cumbersome − some simple hooks can greatly help already − remember... this is about automation, not test design. Make testability part of requirements and architecture − a key question should not just be "how do I design this", but "how do I test this" (test design, automation) − some cloud/web systems are changed frequently, and tested "live" • "Testing in Production (TiP)" − allow redirection of some or all traffic through another version of a component or layer Example: reverse proxy's enabling A/B testing see also: Ken Johnston's chapter in the book of Dorothy Graham and Mark Fewster, and his keynote at StarWest 2012 © 2013 LogiGear Corporation. All Rights Reserved A/B testing with a reverse proxy A A Users Reverse Proxy A B Servers B Watch your test design, easy to drown in technical solutions only B could be a real-life user or also a keyword driven test machine A/B testing means part of traffic is routed through a different server or component (see if it works, and/or how users react) A similar strategy could be done at any component level A current B new © 2013 LogiGear Corporation. All Rights Reserved 58
  • 61. 4/11/2013 Organization Much of the success is gained or lost in how you organize the process − − − − part of the teams who does test design who does automation what to outsource, what to keep in-house Write a plan of approach for the test development and automation − − − − − − scope, assumptions, risks, planning methods, best practices tools, technologies, architecture stake holders, including roles and processes for input and approvals team ... Test design is a skill . . . Assemble the right resources − testers, lead testers − automation engineer(s) − managers, ambassadors, ... Automation is a skill . . . Management is a skill . . . . . . and those skills are different . . . © 2013 LogiGear Corporation. All Rights Reserved Industrial Organization Large scale testing can move from a "design" to a "production" focus − mostly applies to test execution, but also seen for test development − this not black and white, both paradigms can occur in the same projects − this is often more easy to outsource than development A production organization is different a development organization − − − − − this is not unique for software different professional culture emphasis more on delivery and scale, "thinking big" discipline rather than creativity, "get stuff done" activities are like planning, control, logistics, information © 2013 LogiGear Corporation. All Rights Reserved 59
  • 62. 4/11/2013 Task in "production" (test execution) Keeping the tests running Allocating resources Respond to hick-ups Analyze and address automation issues Address fails or other testing outcomes − including dealing with "known bugs" − part of a bigger team © 2013 LogiGear Corporation. All Rights Reserved Stake Holders EXTERNAL INTERNAL Customers Management Vendors Quality Assurance Test Development Government Agencies Publicity Technology/ Infrastructure Test Automation End User Departments Marketing/ Sales System Development After Sales/ Help Desk Production © 2013 LogiGear Corporation. All Rights Reserved 60
  • 63. 4/11/2013 Team roles, examples Test development Automation Planning and managing the test runs Managing environments Managing infrastructure Dealing with stakeholders Analysis of results, and follow up Reporting © 2013 LogiGear Corporation. All Rights Reserved Test Development and Automation in sprints Agile life cycle product owner team Product Backlog Sprint Test Module Definition (optional) prod owner & team Products Test Module Development Test re-use Interface Definition Automation re-use Action Automation Test Execution Test development User stories Documentation Domain understanding Acceptance Criteria PO Questions Situations Relations Main Level Test Modules Interaction Test Modules Cross over Test Modules © 2013 LogiGear Corporation. All Rights Reserved 61
  • 64. 4/11/2013 Test automation in sprints Try keep the main test modules at a similar level as the user stories and acceptance criteria Aim for "sprint + zero", meaning: try to get test development and automation "done" in the same sprint, not the next one − next one means work clutters up, part of team is not working on the same sprint, work is done double (manually and automated), ... Make sure you can do the interface mapping by hand (using developer provided identifications) − can do earlier, before UI is finalized, and − recording of actions (not tests) will go better Also plan for additional test modules: − low-level testing of the interaction with the system under test (like UI's) − crossing over to other parts of the system under test © 2013 LogiGear Corporation. All Rights Reserved Fitting in sprints Agree on the approach: − questions like does "done" include tests developed and automated? − do we see testing and automation as distinguishable tasks and skillsets − is testability a requirement for the software Create good starting conditions for a sprint: − automation technology available (like hooks, calling functions, etc) − how to deal with data and environments − understanding of subject matter, testing, automation, etc Make testing and automation part of the evaluations Address tests and automation also in hardening sprints Just like for development, use discussions with the team and product owners to deepen understanding: − also to help identify negative, alternate and unexpected situations © 2013 LogiGear Corporation. All Rights Reserved 62
  • 65. 4/11/2013 Testing as a profession "Do thorough acceptance testing, but not only by the customer" − source: "Agile Software Testing in a Large-Scale Project", Israeli Air Force Focus on tests, not development: − what can be consequences of situations and events − relieve developers Knowledge and experience with testing techniques and principles The challenge for the tester in the new era is to become a more credible professional tester, − not a pseudo programmer − part of the team Forcing a nontechnical tester to become a programmer may lose a good tester and gain a poor programmer © 2013 LogiGear Corporation. All Rights Reserved Automation is a profession too Overlaps with regular system development, but not same Less concerned with complex code structures or algorithms More concerned with navigating through other software efficiently, dealing with control classes, obtaining information, timing, etc − if you would compare developers to "creators", automation engineers might be likened to "adventurers"... The automation engineering role can also be a consultant: − for test developers: help express tests efficiently − for system developers: how to make a system more automation friendly − important player in innovation in the automated testing © 2013 LogiGear Corporation. All Rights Reserved 63
  • 66. 4/11/2013 Questions for Organization How is your testing currently organized (who is doing what)? − − − − − test design test development automation execution assessment of release readiness Do you use agile? If yes, is there a role for a test professional? And for an automation professional? © 2013 LogiGear Corporation. All Rights Reserved Reporting Aim at needs: − − − avoid lengthy automated reports, have bottom line numbers reports for stake holders reporting for the team Reporting for a big testing project is about: − − − test and automation progress production (running the tests) results (aimed at system under test) Teams need (relevant) details − − − what happened, reproducibility, ... either the tests, the automation, or the system under test overall situations, with an ability to "drill down" to problem areas Management needs: − − − − status, expectations, issues (realistic! bad news matter, you get punished for not telling) bottom lines, plan versus reality confrontation dates, efforts, used resources, costs, run times, ... never allow planned numbers or dates to be "updated" Also for reporting, test organization is a key driver © 2013 LogiGear Corporation. All Rights Reserved 64
  • 67. 4/11/2013 War rooms Helpful if response times are critical, and a need for cooperation, towards the same goal − similar grounds as for agile scrum rooms Set up at critical times, like before important deadlines, or during critical releases Can temporarily bring together multiple parties, that normally are not co-workers − like competitor vendors Pay attention to physical conditions − machines, monitors, white boards, meeting places, headsets, ... − food, drinks, ... The test execution cycle should match the needs of the war room approach − − − − fast turnarounds effortless completeness selective or integral See also: "Your Game is Live, Now What?", Jane Fraser, Electronic Arts © 2013 LogiGear Corporation. All Rights Reserved Globalization.... © 2013 LogiGear Corporation. All Rights Reserved 65
  • 68. 4/11/2013 Main Challenges Other countries Distances Time differences © 2013 LogiGear Corporation. All Rights Reserved Globalization Three Challenges: − another countries, other cultures − geographic distances − time differences Seven "Patterns": − − − − − − − "Solution" "Push Back" "Time Pressure" "Surprises" "Ownership" "Mythical Man Month" "Cooperation" © 2013 LogiGear Corporation. All Rights Reserved 66
  • 69. 4/11/2013 Challenge: Other Country © 2013 LogiGear Corporation. All Rights Reserved Other Country Differences in culture − more on the next slide... Different languages, and accents Differences in education − − − style, orientation and contents position of critical thinking, factual knowledge, practice, theory,... US, British, French, Asian, ... Differences in circumstances − − − demographics economy, infrastructure politics Apprehension on-shore and off-shore about job security doesn't help in projects − − management responsibility: understand your strategic intentions, and their consequences, and clarify them be realistic in cost and benefit expectations © 2013 LogiGear Corporation. All Rights Reserved 67
  • 70. 4/11/2013 More on Culture... Regional culture. There are numerous factors: − − very difficult to make general statements • many anecdotes, stories and perceptions, some are very helpful, some have limited general value • not sure on impact of regional culture (see also [Al-Ani]) numerous factors, like history, religion, political system • e.g. valuing of: critical thinking, theory, bottom-line, relations, status, work-ethic, bad news, saying 'no' • entertaining guests, eating habits, alcohol, meat, humor, etc • position of leaders, position of women managers • mistakes can be benign and funny, but also damaging, visibly or hidden, in particular perceived disrespect hurts Organizational culture − − − can be different from country to country, sector to sector, company to company, group to group I feel this to be at least as strong than regional culture (see for example [Al-Ani]) you can have at least some control over this Professional cultures − for example engineers, QA, managers, ... Some ideas to help: − − get to know each other (it helps, see for example [Gotel]) study the matter, and make adaptations © 2013 LogiGear Corporation. All Rights Reserved © 2013 LogiGear Corporation. All Rights Reserved 68
  • 71. 4/11/2013 © 2013 LogiGear Corporation. All Rights Reserved © 2013 LogiGear Corporation. All Rights Reserved 69
  • 72. 4/11/2013 © 2013 LogiGear Corporation. All Rights Reserved Challenge: Distance © 2013 LogiGear Corporation. All Rights Reserved 70
  • 73. 4/11/2013 Distance Continuous logistical challenges Lots of costs, and disruptions, for traveling Distance creates distrust and conflict − could be "normal" behavior, inherent to humans Complex coordination can create misunderstandings − on technical topics − on actions, priorities, and intentions © 2013 LogiGear Corporation. All Rights Reserved Challenge: Time difference © 2013 LogiGear Corporation. All Rights Reserved 71
  • 74. 4/11/2013 Challenge: Time difference Additional complication for communication and coordination Places a major burden on both on-shore and off-shore staff − having to work evenings and/or early mornings − potential for exhaustion, lack of relaxation, mistakes, irritation Can easily lead to loss of time at critical moments Some solutions: − − − − manage this actively constantly seek to optimize task and responsibility allocation build the on-shore and off-shore organizations to match seek ways to save meeting time, like optimal information handling © 2013 LogiGear Corporation. All Rights Reserved Effect of time difference Report from the team to the US management . . . Performance comparison TestArchitect 5 and 6 Test Module: “Segment Y, Default Settings” TestArchitect 5 TestArchitect 6 Windows ~ 4:16 m ~ 11:00 m Linux ~ 4:28 m ~ 8:00 m © 2013 LogiGear Corporation. All Rights Reserved 72
  • 75. 4/11/2013 Patterns Experiences seem to follow patterns − at least our own experiences do − variations are numerous, but seem to follow similar lines − following are examples, not limitative It can help to recognize patterns quickly, and act upon them Resolutions have side-effects, can introduce new issues − for example strengthening local management means less direct contact with the project members doing the work Just about every pattern occurs in every direction − from your perspective regarding "them" − their perspective on you, or each other − sometimes equaling, sometimes mirroring © 2013 LogiGear Corporation. All Rights Reserved Pattern: "The Solution" Typical sequence of events: − the team finds a problem in running a test − the team discusses it and comes up with a "solution" − the solution: (1) creates issues, and (2) hides the real problem Better way: − define as an issue − discuss with project manager and customer © 2013 LogiGear Corporation. All Rights Reserved 73
  • 76. 4/11/2013 Pattern: "Push Back" US side, or customer, gives bad direction Team doesn't like it, but feels obliged to follow orders The result is disappointing Team is blamed − and will speak up even less next time Better way: − discuss with the principal/customer at multiple levels • strategic about direction, operational day-to-day − empower and encourage the team to speak up − write plans of approach, and reports © 2013 LogiGear Corporation. All Rights Reserved Pattern: "Time Pressure" Deadline must be met − no matter what − use over-time − "failure is not an option" Deadlines are sometimes real, sometimes not − − − − become a routine on the US side easy to pressure over the email very difficult for a non-empowered team to push back risk: inflation of urgency Better way: − − − − good planning proper weighing of deadlines and priorities frequent reporting local management © 2013 LogiGear Corporation. All Rights Reserved 74
  • 77. 4/11/2013 Pattern: "Surprises" Good news travels better than bad news... − should be the other way around − the "cover up": "let's fix, no need to tell...." − over time: needing bigger cover ups to conceal smaller ones − not unique for off-shoring, but more difficult to detect and deal with Once a surprise happens: − you will feel frustrated, and betrayed − fix the problems, point out the consequences of hiding, avoid screaming and flaming Better ways: − − − − − agree: NO SURPRISES!! emphasize again and again train against this continuously manage, point out the magic word: transparency SUPRISES © 2013 LogiGear Corporation. All Rights Reserved Pattern: "Ownership" Shared responsibility is no responsibility Effort-based versus result-based On-shore players feel the off-shore team has a result responsibility Off-shore team members feel an effort-based responsibility ("work hard") Better way: − clear responsibilities and expectations − on-shore ownership for quality control of system under test • and therefore the tests − off-shore ownership of producing good tests and good automation − empower according to ownership © 2013 LogiGear Corporation. All Rights Reserved 75
  • 78. 4/11/2013 Pattern: "Mythical Man Month" Fred Brooks classic book, "Mythical man month": − "Assigning more programmers to a project running behind schedule will make it even later" − "The bearing of a child takes nine months, no matter how many women are assigned" In test automation, there must be clear ownership of: − test design (not just cranking out test cases) − automation, this is different skill and interest Assign at least the following roles: − project lead, owns quality and schedule − test lead: owns test design, coaches and coordinates the other testers − automation: make the actions work (assuming ABT, not the test cases) Define distinct career paths in: testing, automation, management © 2013 LogiGear Corporation. All Rights Reserved Pattern: "Cooperation" Communication is tedious, takes a long time Questions, questions, questions, ... − reverse: questions don't get answered For at least one side in private time, extra annoying Misunderstandings, confusion, actions not followed up − double check apparent "crazy things" with the team before jumping to conclusions, and actions (assume the other side is not "nuts" or "dumb"...) Please understand: distance fosters conflicts − we're born that way, be ready for it Better ways: − prioritize training, coaching, preparation and planning. Saves a lot of questions... − write stuff down, use briefs, minutes − define workflows and information flows • buckets, reporting, select and use good tools − specialize meetings • • table things for in-depth meetings ask to meet internally first − be quick, no more than 30 mins © 2013 LogiGear Corporation. All Rights Reserved 76
  • 79. 4/11/2013 Cooperation © 2013 LogiGear Corporation. All Rights Reserved Training, some ideas Many areas, big pay-offs: − − − − − − − − system under test subject matter under test, domain knowledge methods, best practices technologies, tools, ... processes soft skills, like creativity, critical thinking, management, ... language cross-cultural Have exams − − − think about the consequences of passing and failing teams pay more attention when they know they will get tested you will know whether you were understood Also have coaching and train-the-trainers − − − − more experienced people help newbie's also runs a risk: bad habits can creep in and procreate "Tribal knowledge", learning by osmosis, water cooler conversations, encourage it consider "special interest groups (SIG's)" Rule of thumb for off-shore teams: hire for technical knowledge, train for business knowledge The on-shore staff needs training and coaching too © 2013 LogiGear Corporation. All Rights Reserved 77
  • 80. 4/11/2013 Additional ideas Go there, be with the team, experience yourself how "your side" is doing − I go about twice per year Manage ownership, is it you or them − the distinction between efforts and results Provide clear direction, constant attention and coaching Supervise, supervise, supervise − but don't micromanage if the other side has ownership Ask to create example products (like ABT test modules and actions), review them carefully, and use as direction for subsequent work Leadership style: participative styles seem most common (as opposed to consensus or authoritative, see also [Al-Ani]) Organize informal/fun events, provide a good environment − − solidify the group, improve retention include visiting US staff, this tends to do a lot of good ("priceless") Manage expectations − − stuff takes time and energy differences can be addressed, but not 100% cake... © 2013 LogiGear Corporation. All Rights Reserved Outsourcing and Agile If done well, can provide relieve to a lot of the patterns Several models possible Model 1: Full team outsourcing − development, testing and automation Model 2: "2nd unit" − off-shore team works under control of sprint team members Model 3: Part of integrated team: − − − needs online tool like Jira or Rally you must have shared meetings advantage: more project time Large scale test development and automation might be easier to outsource than development © 2013 LogiGear Corporation. All Rights Reserved 78
  • 81. 4/11/2013 Summary Not all "big project" challenges are the same Think before you do. Best results come from planning well, and combining effective concepts, tricks and tools Consider tests and automation as products Team work is a key for short term and long term success There are many options for infrastructure, but keep an eye on economy and planning Off-shoring can help scale up, but needs attention to do it right, in particular communication Repeat of initial invitation Focus today was on overview and concepts, not always on details. Please see me in person for any discussion you would like on your situation that I didn't cover. We're also exhibiting here, probably easiest to reach me there. © 2013 LogiGear Corporation. All Rights Reserved Homework . . . 1. Testing Computer Software, Cem Kaner, Hung Nguyen, Jack Falk, Wiley 2. Lessons Learned in Software Testing, Cem Kaner, James Bach, Bret Pettichord, Wiley 3. Experiences of Test Automation, Dorothy Graham, Mark Fewster, Addison Wesley, 2012 4. Automating Software Testing, Dorothy Graham, Mark Fewster, Addison Wesley 5. "Build a Successful Global Training Program", Michael Hackett, www.logigear.com 6. Action Based Testing (overview article), Hans Buwalda, Better Software, March 2011 7. Action Figures (on model-based testing), Hans Buwalda, Better Software, March 2003 8. Integrated Test Design & Automation, Hans Buwalda, Dennis Janssen and Iris Pinkster, Addison Wesley 9. Soap Opera Testing (article), Hans Buwalda, Better Software Magazine, February 2005 10. Testing with Action Words, Abandoning Record and Playback, Hans Buwalda, Eurostar 1996 11. QA All Stars, Building Your Dream Team, Hans Buwalda, Better Software, September 2006 12. The 5% Solutions, Hans Buwalda, Software Test & Performance Magazine, September 2006 13. Happy About Global Software Test Automation, Hung Nguyen, Michael Hackett, e.a., Happy About 14. Testing Applications on the Web, Hung Nguyen, Robert Johnson, Michael Hackett, Wiley 15. Practical Combinatorial Testing, Richard Kuhn, Raghu Kacker, Yu Lei, NIST, October, 2010 16. Agile Software Testing in a Large-Scale Project, David Talby, Arie Keren, Orit Hazzan, Yael Dubinsky, IEEE Software, July/August 2006 17. JMeter in the Cloud, Jörg Kalsbach, http://aws.amazon.com/amis/2924 18. Using Monkey Test Tools, Noel Nyman, STQE issue January/February 2000 19. High Volume Test Automation, Cem Kaner, Walter P. Bond, Pat McGee, StarEast 2004 20. Descriptive Analysis of Fear and Distrust in Early Phases of GSD Projects, Arttu Piri, Tuomas Niinimäki, Casper Lassenius, 2009 Fourth IEEE International Conference on Global Software Engineering [Piri] 21. Quality Indicators on Global Software Development Projects: Does 'Getting to Know You' Really Matter? Olly Gotel, Vidya Kulkarni, Moniphal Say, Christelle Scharff, Thanwadee Sunetnanta, 2009 Fourth IEEE International Conference on Global Software Engineering [Gotel] © 2013 LogiGear Corporation. All Rights Reserved 79
  • 82. 4/11/2013 Thanks... Please fill out the evaluation form, from the back of the book Let me know any questions or concerns: We're at the expo, questions welcome − I will be there myself too quite a bit TESTING FOR SALE email: hans @ logigear.com articles: www.happytester.com company: www.logigear.com TestArchitect: www.testarchitect.com Supersize your tests for less... we're at the expo © 2013 LogiGear Corporation. All Rights Reserved 80