Software testing standards—who cares, anyway? You should! The new ISO/IEC/IEEE 29119 software testing standard, driven by representatives from twenty countries and under development for the past five years, will be released soon. As a professional tester, you need to know about this standard and how it may apply to your environment. Jon Hagar describes the standard, how it was developed, and what types of projects will be impacted by it. This new standard offers risk-based approach to software testing that can be applied to both traditional and agile projects. It is comprehensive—addressing software test basic concepts, definitions, generic test processes, documentation, and techniques—and will replace numerous IEEE and national standards. Many countries, government agencies, and private companies worldwide will start using ISO 29119 in the coming years to benchmark and improve their test practices. Join with Jon to dive in to ISO 29119 and see what it is all about.
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Introducing the New Software Testing Standard
1. T24
Concurrent Class
10/3/2013 3:00:00 PM
"Introducing the New Software
Testing Standard"
Presented by:
Jon Hagar
Consultant
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
2. Jon Hagar
Grand Software Testing
Jon Hagar is a systems-software engineer and tester consultant supporting software product
integrity and verification and validation, with a specialization in embedded and mobile software
systems. For more than thirty years Jon has worked in software engineering, particularly testing,
supporting projects including control system (avionics and auto), spacecraft, mobile-smart
devices, IT, and attack testing of smart phones.
3. 9/19/2013
Special Topics T24
Introducing the New
Software Testing Standard:
ISO/IEEE/IEC 29119
1
Jon Hagar
embedded@ecentral.com
Grand Software Testing (GST), LLC
303-903-5536
Copyright 2013
Jon Hagar
Grand Software Testing
GST
Scope
Purpose of standards
Types of standards
Development of standards
Overview of ISO29119
Applicability
Recent developments
Timeline
Future work
Copyright 2013
Jon Hagar
Grand Software Testing
GST
1
4. 9/19/2013
Some Preliminaries
Under disclosure, I am the IEEE Project Editor for
ISO/IEEE/IEC 29119 Software Testing Standard
Before you decide to roast me . . .please recognize
Standards are a fact of life for many industry professionals not just
testers
Standards form part of the business for many companies ,
customers, and countries
Many of us end up in a mixed relationship with standards
Standards can be a basis for research and change
Industry has not had a comprehensive software testing standard
Copyright 2013
Jon Hagar
Grand Software Testing
GST
What are standards?
“Guideline documentation that reflects
agreements on products, practices, or
operations by nationally or internationally
recognized industrial, professional, trade
associations or governmental bodies”
- ISO
Guideline documents—not compulsory unless
mandated
Agreements because they often reflect a certain level of
consensus within a community
Copyright 2013
Jon Hagar
Grand Software Testing
GST
2
5. 9/19/2013
How can a standard “help?”
Standards describe a current ‘body of
knowledge’ that provides the basis for a
professional discipline
Standards are a basis for:
Communication – common terminology
Professionalism and criticism
Conformance/compliance schemes
Benchmark of ‘industry practice’ (not best, may not be good
for all, etc.)
Contracts and regulations
Interoperability and consistency. . .
Copyright 2013
Jon Hagar
Grand Software Testing
GST
ISO29119 Standard - My Viewpoint
Any software standard is based on limited history (rear
looking) and developed based on consensus of many voices
Yet still may represent only some of viewpoints of industry
But does not represent all states of the art
Standards need to be inclusive and ethical
As with many “schools,” views and opinions but with NO
“best”
Not claimed to be perfect
Most projects/professionals will use “tailoring”
Copyright 2013
Jon Hagar
Grand Software Testing
GST
3
6. 9/19/2013
Standardization Bodies
Copyright 2013
Jon Hagar
Grand Software Testing
GST
Motivation for ISO29119
Conflicts in definitions, processes and procedures
“One ring to rule them all” — standards to be replaced by one
e.g., IEEE 829, IEEE 1008, BS7925-1/-2, IEEE 1028
Users do not know which standard to follow
Lacking in current standards or incomplete:
Organizational areas
e.g., Test Policy and Organizational Test Strategy
Project Test Management
BS7925 only covers unit testing
General processes
Common functional techniques missing
Coverage of non-functional testing
Copyright 2013
Jon Hagar
Grand Software Testing
GST
4
7. 9/19/2013
ISO/IEC29119 –Structure and Flow
Part 1
ISO 12207
ISO 15288
Directives
Concepts & Vocabulary
BS7925-1
Part 4
Part 2
Part 3
Testing
Techniques
Processes
Documentation
IEEE 1008
BS7925-2
IEEE 829
Part 5
Copyright 2013
ISO/IEC 33063
Keyword-Driven
Testing
Process
Assessment
Jon Hagar
Grand Software Testing
GST
A Brief Diversion– Schools of Testing
Initially proposed by Pettichord (workshop notes) et. al.
Labels used here with some liberty (to be positive)
Context driven is one, but there are others, possibly
Agile (the rage for the last 10 years)
Academic (researchers)
Quality assurance (whatever that is)
Process based
“None” (not an official name, but I see these testers all the time)
Some “schools” may not recognize themselves (by the names)
Schools are common in art, music, sciences, and other professions
But many “artists” cross the school lines of art
ISO29119 aims to be “school independent”, but will have a
process centric view but with liberal ability to tailor
Copyright 2013
Jon Hagar
Grand Software Testing
GST
5
8. 9/19/2013
Do Testers Need Standards? –
Yes, maybe, but
Standards support common communication within the topic
Common reference points
Starting point for research, usage (pro & con), critic
Maturity is an issue but a baseline serves as sounding board
and common reference point for “Scientific” methods
An international benchmark
Thinkers and researchers can prove/disprove benchmark(s)
Part of being in a profession (but only part)
Standards have dangers
Copyright 2013
Jon Hagar
Grand Software Testing
GST
My View – Potential Dangers
Not every book, school, paper, ideal will be represented
There must be a vetting, upgrade, and maintenance effort
Will not make everyone happy, so in most cases must be
tailored by anyone using them
Will be subject to “misuse” by various groups
Unthinking audits/reviews
May stop some people from thinking for themselves (but many do
that anyway)
Will (always) be imperfect
Copyright 2013
Jon Hagar
Grand Software Testing
GST
6
9. 9/19/2013
Part 1: Concepts & Vocabulary
DEFINITIONS
SOFTWARE TESTING CONCEPTS
TESTING IN DIFFERENT LIFE CYCLE MODELS
ROLES AND RESPONSIBILITIES IN TESTING
Test: Approach, Basis, Methods
- Risk Based Testing
Scope, Conformance, Normative References
ANNEXES – Metrics, Examples, Bibliography
Copyright 2013
Jon Hagar
Grand Software Testing
GST
Part 2: Testing Processes
ORGANIZATIONAL TEST PROCESS
TEST MANAGEMENT PROCESSES
DYNAMIC TEST PROCESSES
Copyright 2013
Jon Hagar
Grand Software Testing
GST
7
10. 9/19/2013
Instantiating Testing Processes
ORGANIZATIONAL TEST PROCESS
TEST MANAGEMENT PROCESSES
DYNAMIC TEST PROCESSES
Ref: S. Reid
Copyright 2013
Jon Hagar
GST
Grand Software Testing
Organizational Test Processes
Develop
Organizational
Test
Specification
(OT1)
Organizational
Test
Specification
Monitor and
Control Use of
Organizational
Test
Specification
(OT2)
Controlled
Organizational
Test
Specification
Update
Organizational
Test
Specification
(OT3)
Updated
Organizational
Test Specification
Copyright 2013
Jon Hagar
Grand Software Testing
GST
8
11. 9/19/2013
Test Management Processes
ORGANIZATIONAL TEST PROCESS
ORGANIZATIONAL
TEST
DOCUMENTATION
FEEDBACK ON
ORGANIZATIONAL TEST
DOCUMENTATION
TEST MANAGEMENT PROCESSES
TEST PLAN UPDATES
TEST
PLAN
TEST
PLANNING
TEST PLAN,
CONTROL
DIRECTIVES
TEST
MONITORING &
CONTROL
TEST
COMPLETION
TEST PLAN,
TEST COMPLETION
REPORT,
TEST MEASURES
TEST PLAN,
CONTROL
DIRECTIVES
TEST
MANAGEMENT
PROCESSES
Copyright 2013
TEST
COMPLETION
REPORT
TEST
MEASURES
DYNAMIC TEST
PROCESSES
Jon Hagar
Grand Software Testing
GST
Test Planning Processes
Understand
Context
Scope
Organize
Test Plan
Development
Identify &
Estimate Risks
Analyzed
Risks
Identify Risk
Treatment
Approaches
Schedule,
Staffing
Profile
Draft
Test Plan
Approved
Test Plan
Test
Plan
Treatment
Approaches
Design Test
Strategy
Determine
Staffing and
Scheduling
Test
Strategy
Document
Test Plan
Gain
Consensus on
Test Plan
Publish
Test Plan
Copyright 2013
Jon Hagar
Grand Software Testing
GST
9
12. 9/19/2013
Standard Primarily has a Risk-based Process Flavor
Risk based testing is a primarily “good” practice
Other bases/approaches (Agile, exploratory, model-based, mathbased, etc.) are included
All parts may need “tailoring” (few efforts would be “full” conformance)
Use of the standard implies skills, thinking, dependence on
context (“it depends”), etc.
Copyright 2013
Jon Hagar
Grand Software Testing
GST
Test Monitoring & Control Process
test status report
test
progress
info
test
plan
Report
test
control
info
[testing incomplete]
[testing
complete]
test
measures
Set-Up
Monitor
test progress
info
measures
Control
control
directives
...TEST PROCESSES...
Dynamic/Management
Copyright 2013
Jon Hagar
Grand Software Testing
GST
10
13. 9/19/2013
Dynamic Test Processes
(LEVEL) TEST MANAGEMENT PROCESS
(LEVEL)
TEST PLAN
CONTROL
DIRECTIVES
TEST
MEASURES
DYNAMIC TEST PROCESSES
TEST DESIGN &
IMPLEMENTATION
TEST
SPECIFICATION
TEST
EXECUTION
TEST
RESULTS
[NO ISSUES
NOTICED]
[ISSUE NOTICED
or
RETEST RESULT]
TEST
ENVIRONMENT
REQUIREMENTS
Copyright 2013
TEST
ENVIRONMENT
SET-UP
Jon Hagar
TEST
ENVIRONMENT
READINESS
REPORT
TEST
INCIDENT
REPORTING
Grand Software Testing
INCIDENT
REPORT
GST
Part 3 – Test Documentation
Select a subset of docs
Scope, Conformance,
Normative References
TEST DOCUMENTATION
ANNEXES - EXAMPLES
Copyright 2013
Jon Hagar
Grand Software Testing
GST
11
14. 9/19/2013
Part 3: Test Documentation
Organizational test documentation
Test policy
Test strategy
Project test documentation
Project test plan
Test project completion report
Test Level documentation
Test plan
Test specification
Test results
Anomaly reports
Level test status report
Test environment report
Test level completion report
Appendices
Examples of documents at each level of testing
Copyright 2013
Jon Hagar
Grand Software Testing
GST
Part 4 – Test Techniques
Scope, Conformance, Normative References
TEST DESIGN TECHNIQUES
Functional
Structural
TEST COVERAGE MEASUREMENT
ANNEXE – TESTING OF QUALITY CHARACTERISTICS
ANNEXE – SELECTION OF TECHNIQUES
ANNEXE – TEST TECHNIQUE EFFECTIVENESS
Copyright 2013
Jon Hagar
Grand Software Testing
GST
12
15. 9/19/2013
Part 5- Keyword-Driven Testing
Part 5 will address:
Concept
Applicability
Interfaces
Approach
Part 5 WD was sent out in May and next draft due in Nov 2013
Copyright 2013
Jon Hagar
Grand Software Testing
GST
Applicability of ISO/IEEE 29119
It is intended to be a generic standard to support all
domains
Some domains are more standards oriented such as:
safety-related
telecoms
Some domains will become more regulated such as:
financial – banks, stock markets, etc.
Needs trails in all domains and project types
Applicability to sequential, iterative and agile
Applicability to new as well as maintenance projects
Applicability to IT and embedded
Copyright 2013
Jon Hagar
Grand Software Testing
GST
13
16. 9/19/2013
Discussion Point: Thoughts?
Do you think we need standards?
If we use a standard does this mean we do not think?
What should the test community do?
Copyright 2013
Jon Hagar
Grand Software Testing
GST
My Thoughts
Thinking testers with a context-driven/based viewpoint
Use standards when a customer or project requires them e.g., in
regulated environments
Understand where standards are “good” (worthwhile) and bad (not
worthwhile)
Use standards as a professional reference work, but know when and
how to break the “rules” via tailoring
How should the test community react?
Engage to identify and address both the good and bad
Do not use standards in all situations (test projects)
Use materials for constructive criticism
They will have many concepts which need to be “over turned”
Drive the scientific review and healthy debated
Copyright 2013
Jon Hagar
Grand Software Testing
GST
14
17. 9/19/2013
Timelines – Current Status
Parts 1, 2, 3
Working Draft (WD)
Committee Draft (CD)
Draft International Standard (DIS)
Final Draft International Standard (FDIS)
Final International Standard (FIS)
Part 4
Part 5
WD
FIS
WD
CD1
CD1
WD
…
May
2010
Copyright 2013
CD2
CD3
FIS
CD2
CD3
FDIS
DIS
May
2012
May
2011
Jon Hagar
DIS
FDIS
Grand Software Testing
4
May
2013
GST
For more information
jon.d.hagar@gmail.com
For more information and details
http://softwaretestingstandard.org/
WG26 website
http://www.jtc1-sc7.org/
Access to official documents released by WG 26
Copyright 2013
Jon Hagar
Grand Software Testing
GST
15
18. 9/19/2013
Summary
First attempt at a comprehensive software test
standard is being released
Even in the presence of standards, a thinking
tester is always needed
What will you do?
Copyright 2013
Jon Hagar
Grand Software Testing
GST
Major Reused Reference Point:
Stuart Reid
Testing Solutions Group
117 Houndsditch
London EC3
UK
Tel: 0207 469 1500
Fax: 0207 623 8459
www.testing-solutions.com
Copyright 2013
Jon Hagar
Grand Software Testing
GST
16