HBT - A Revolutionary Approach to Testing Software

Hypothesis Based Testing (HBT)
A revolutionary approach to testing software

T Ashok
ash@stagsoftware.com

in.linkedin.com/in/AshokSTAG
ash_thiru

Webinar: 22 June 2011, 1430-1530 IST

How can we seek inspirations from nature
to deliver “Clean Software”?

In the next hour, let us look at the act of
testing differently and see how we can
think better and test scientifically?

© 2011. STAG Software Private Limited. All rights reserved.

Quality is...

“... at least meeting expectations”

Meeting expectations implies
“satisfying properties”

i.e. Expect certain properties to be satisﬁed.

Properties are impeded by “potential defects”


Properties of matter

Physical & Chemical properties of
matter allow us to:
... classify
... understand behaviors, interactions
... enable checking purity

Think!
How can we use a similar train of
thought to identify “properties of
cleanliness” and then “types of
defects”?

http://www.science.wellsresearch.org/wp-content/uploads/2010/02/CBScienceHSPhysicalScienceSt2.jpg


“Properties of the system”
Cleanliness criteria End user expectations

“affected by”

Issues in speciﬁcations,
Potential Defect Types (PDT) structure, environment
and behavior


Expectations
“Properties of the system”
Needs Cleanliness criteria
Features

Environment
“impedes”
Behavior

Structure
Potential Defect Types (PDT)
Material

Expectations delivered
by Needs (Requirements)
via Features
that display Behavior
constructed from Materials
in accordance to a Structure
in a given Environment

Fractional Distillation

A technique to separate mixtures
that have components of different
boiling points

Think!
There are a variety of defect
types that may be present in the
system.

To optimally uncover the
From : http://withfriendship.com defects, can we separate these
types of defects on the basis of
certain properties and
optimally uncover the defects?


Fetal growth - Delivering systems (Babies too!)

Visualizing health :
- Size, Shape, Parts
- Types of issues not present

Think!
How can we measure the
‘intrinsic quality’ at a stage?

Quality Growth
At each stage certain types of
Source :http://www.environment.ucla.edu/media/images/Fetal_dev5.jpg
defects detected & removed.
Quality grows.

Can we chart this as “cleanliness
index”?


Sherlock Holmes

Focus on intellectual power to solve mysteries.

Strong application of deductive logic.

Think!
Can we hypothesize as to what can go
wrong and prove/disprove?

Hypothesize potential defect types and
prove presence of instances. If they are
not present, can we reason their
absence?


Test methodologies focus on activities that are driven by a process which are
powered by tools, yet successful outcomes still depend a lot on experience.

Typically methodologies are at organizational level.

On the other hand HBT
... is a personal scientiﬁc methodology enabled by STEMTM ,
a defect detection technology to deliver “Clean Software”


Scientific approach to uncovering defects

Cleanliness criteria What is the end user expectation of “Good Quality”?

Potential Defect Types What types of issues can result in poor quality?

Evaluation Stage When should I uncover them?

Test Types How do I uncover them?

Test Techniques What techniques to generate test cases?

Scenarios/Cases What are the test cases? Are they enough?

Scripts How do I execute them?

Metrics & Management How good is it? How am I doing?


HBT: Quality Levels, Aspects & Issues
Aspect Issues

Usage
L9 End user value User ﬂows, experience

Environment, Data, Logic
L8 Clean Deployment Compatibility, migration
Structure, Logic, Usage
L7 Attributes met Performance, security, volume, load...
Environment
L6 Environment cleanliness Resource leaks, Compatibility...

Logic, Environment
L5 Robustness Error handling
Data, Logic
L4 Behavior correctness Functionality

L3 Structural integrity Structure
Internal structural issues
Structure, Usage
L2 Input interface cleanliness UI issues

Data
L1 Input cleanliness
Input data handling


HBT - A Quick Introduction

SIX stages of powered EIGHT disciplines
DOING by
of THINKING
D8 D1
S6 S1
Analysis & Business value
Assess & Understand
management understanding
ANALYSE EXPECTATIONS
D7 D2
Execution & Defect
D8 D1
reporting STEM Core hypothesis
Tooling D7 D2
S5 STEM Understand 32 core
SUPPORT S2
D6 D3 CONTEXT concepts
D5 D4 Strategy &
Visibility
planning D3
D6
Devise Formulate
HYPOTHESIS Tooling Test design
PROOF

S4 S3 D5 D4

HBT powered STEM
Personal test methodology by Defect detection technology


D1 Business value understanding D2 Defect hypothesis D3 Test strategy & planning
Landscaping
EFF model Orthogonality principle
Viewpoints
(Error-Fault-Failure) Tooling needs assessment
Reductionist principle
Defect centricity principle Defect centered AB
Interaction matrix
Negative thinking Quality growth principle
Operational profiling
Orthogonality principle Techniques landscape
Attribute analysis
Defect typing Process landscape
GQM

D4 Test design D5 Tooling
Reductionist principle Automation complexity
Input granularity principle 32 core assessment
Box model Minimal babysitting
Behavior-Stimuli approach concepts principle
Techniques landscape Separation of concerns
Complexity assessment Tooling needs analysis
Operational profiling

D6 Visibility D7 Execution & Reporting D8 Analysis & Management

GQM
Contextual awareness Gating principle
Quality quantification
Defect rating principle Cycle scoping
model


How is HBT different?
from other methodologies...

Goal
al
p ic

Powered by experience
Ty

drives

T
Activities

HB
......................................

detection technology
Powered by defect
......................................
...................................... Activities
...................................... ......................................

(STEM)
...................................... ......................................
...................................... ......................................
......................................
......................................
......................................
hopefully
results in
Goal


Hypothesis Based Testing (HBT)

S1
Cleanliness criteria Potential defect types S3
S2

Staged & purposeful
Expectations detection

S4

Complete test cases

S6 Goal directed measures Sensible automation S5


Cleanliness criteria Potential defect types
Clear Baseline
S1, S2
Staged & purposeful
detection

Expectations
Set a clear goal for quality
Complete test cases

Example: Clean Water implies
Goal directed
measures
Sensible automation 1.Colorless
2.No suspended particles
3.No bacteria
4.Odorless

What information(properties) can be used
to identify this?

...Marketplace,Customers, End users
...Requirement(ﬂows), Usage, Deployment
... Features, Attributes
...Stage of development, Interactions
... Environment, Architecture
... Behavior, Structure


A goal focused approach
S3 to cleanliness
Staged & purposeful
detection Identify potential defect types that
Expectations can impede cleanliness
Complete test cases
Example:
Data validation
Timeouts
Goal directed Sensible automation Resource leakage
measures
Calculation
Storage
Presentation
Transactional ...

Scientiﬁc approach to hypothesizing defects is about looking at

FIVE Aspects - Data, Logic, Structure, Environment & Usage
from
THREE Views - Error injection, Fault proneness & Failure

Use STEM core concepts
> Negative thinking (Aspect)
> EFF Model (View)

“A Holmes-ian way of looking
at properties of elements”

Cleanliness criteria Potential defect types Levels, Types & Techniques -
STRATEGY
Staged & purposeful
detection
S4 NINE levels to Cleanliness
Expectations
L9 End user value
Complete test cases

L8 Clean Deployment

Goal directed L7 Attributes met
Sensible automation
measures
L6 Environment cleanliness

Quality Levels L3 L5 Robustness
PDT7
PDT6
L4 Behaviour correctness
L2 PDT5
Test Techniques (T1-T4)
PDT4
L1 PDT3 TT5 T4 L3 Structural integrity
PDT2
PDT:
PDT1 Potential Defect Types TT4 L2 Input interface cleanliness
T3
TT3 L1 Input cleanliness
Test Types PDT7
TT5
PDT6 TT2
T2
TT4 PDT5
PDT4 TT1 T1
TT3
PDT3
TT2 PDT2 TT:
TT1 PDT1 Test Types

“Fractional distillation of bug mixture”

Countable test cases &
Fault coverage
Staged & purposeful
detection

Expectations Use STEM Core concepts
> Box model
Complete test cases
> Behaviour Stimuli approach
S4
> Techniques landscape
Goal directed > Coverage evaluation
Sensible automation
measures
to
- Model behaviour
- Create behaviour scenarios
- Create stimuli (test cases)
Test Scenarios/Cases
R1 PDT1 Irrespective of who designs, #scenarios/
TS1 TC1,2,3
R2 PDT2 cases shall be same - COUNTABLE
TT
R3 TS2 TC4,5,6,7 PDT3

Requirements & Fault traceability

That test cases for a given
requirement shall have the ability
to detect speciﬁc types of defects Guarantee test adequacy.
FAULT COVERAGE Guarantee implies
that the means to the end is rational & provable

HBT Test Case Architecture

Organized by Quality levels
sub-ordered by items (features/modules..),
Level segregated by type,
ranked by importance/priority,
Item sub-divided into conformance(+) and robustness(-),
classiﬁed by early (smoke)/late-stage evaluation,
Type tagged by evaluation frequency,
linked by optimal execution order,
classiﬁed by execution mode (manual/automated)
Priority

Focus

Stage
A well architected set of test cases is like a
effective bait that can ‘attract‘ defects in the
Frequency
system.
Order

It is equally important to ensure that they
Mode
are well organized to enable execution
optimisation and have the right set of
information to ensure easy automation.


Cleanliness criteria Potential defect types Focused scenarios + Good
Automation Architecture
Staged & purposeful
detection

Expectations Level based testcriteria
Cleanliness scenarios yield
Complete test cases shorter scripts that are
more ﬂexible for change and
S5 easily maintainable.
Goal directed Sensible automation
measures

L9 End Expectations
user value

L8 Clean Deployment

L7 Attributes met

L6 Environment cleanliness

L5 Robustness

L4 Behavior correctness

L3 Structural integrity

L2 Input interface cleanliness

L1 Input cleanliness


“Cleanliness Index” -
Improved visibility
Staged & purposeful
detection L4
Expectations PDT10 TT8

Complete test cases PDT9 TT7

S6 L3 PDT9 TT6

Cleanliness
PDT8 TT5
Goal directed Sensible automation
measures PDT7
TT4
L2 PDT6
PDT5
TT3
L1 PDT4

PDT3
TT2
PDT2

PDT1 TT1

Stage
Quality report
CC1 CC2 CC3 CC4
R1 Met
R2
Not met
R3
Partially met
R4
R5

“Growth of a baby”

HBT - A Case Study
Two Teams, one using HBT & The other conventional approach

Test case details Test case details (HBT)

Module HBT Normal Increase Module Total Positive Negative

M1 100 28 257% M1 100 59 41

M2 85 52 63% M2 85 68 17
M3 95 66 44% M3 95 67 28
M4 132 72 83% M4 132 112 20
M5 127 28 354%
M5 127 85 42
M6 855 116 637%
M6 855 749 106
TOTAL 1394 362 285%
TOTAL 1394 1140 254
Nearly 3x increase in #test cases increasing
probability of higher defect yield 2x improvement in negative cases
increasing probability of better defect yield

Defect details Effort details (person-hours)
HBT Normal Increase Stage HBT Normal

#Defects 32 16 100% Test analysis &
30 20
Design
20 (Major), 12(Minor)
Out of these 32 defects, few were Front loading of effort resulted
residual defects, one being critical to in lowering support cost
corrupt the entire data.


HBT Results

50%-1000% reduction in post-release defects

Re-architecting test assets increases test coverage by 250%

30% defect leakage reduction from early stage

Smart automation - 3x reduction in time

Deskilling - Less experienced staff do better, faster ramp up, lower cost

‘Holes’ found & ﬁxed at requirement stage

Test assessment accelerates integration, de-risks deployment


Summarizing...
S2
Cleanliness criteria Potential defect types S3
S1

Staged & purposeful
detection

Expectations S4

Complete test cases

Goal directed
S6 Sensible automation S6
measures

Clear baseline

A goal focused approach to cleanliness

Levels, Types & Techniques - STRATEGY

Countable test cases & Fault coverage

Focused scenarios +
Good automation architecture

“Cleanliness Index” - Improved visibility


Thank you!

“...the value addition for us came from a blend of
HBT/STEM and the dedication/hard work from senior
management to excel in their services....”
Praveen Kanipakam, President
SHARP Software Development India

“...the methodology and team was the potent
combination that jumpstarted our QA."
Tapesh Kumar Agarwal, CTO & Co-Founder
Meddiff Technologies
Connect with us...
@stagsoft
“..the focus on scientiﬁc way of doing testing with blog.stagsoftware.com
their methodology HBT was reﬂected in the
effectiveness of test artifacts created by their team.”
Raj Khare, Chairman & Founder
SureWaves

“..the quality of test design and completeness of
testing as a result of their methodology is
evident in every release.”
Iype Isaac, CTO
Bankbazaar.com

STEMTM is the trademark of STAG Software Private Limited
© 2011. STAG Software Private Limited. All rights reserved. www.stagsoftware.com

HBT - A Revolutionary Approach to Testing Software

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (18)

Ähnlich wie HBT - A Revolutionary Approach to Testing Software

Ähnlich wie HBT - A Revolutionary Approach to Testing Software (20)

Mehr von STAG Software Private Limited

Mehr von STAG Software Private Limited (20)

HBT - A Revolutionary Approach to Testing Software