A presentation that I did for MSCI:720

1. Software Quality:
The Elusive Target?
Syed Sohail Javaad
28th July 2011

2. Presentation Theme
• Introduction to Quality
• Software Quality
• Addressing Software Quality
• Final Discussion
• Theme for the Term Paper
Software Quality - An Elusive Target 2

3. Introduction to Quality

4. Introduction to Quality*
• What is Quality?
• How do writers define Quality?
– Conformance to requirements
– Customers’ reactions
– Notions of completeness
– Notions of cost effectiveness
• Defining Quality is a problem. Why?
– Ambiguous and subjective.
• …lies in the eye of beholder
• Depends on who is the respondent?
• Depends on requirements and psychology
• Quality Vs. Grade
* Most of the material in section is from lecture slides of Dr. Darrell Raymond from his course on
Quality Systems (Winter Quality at UW). Used with permission
Software 2011 - An Elusive Target 4

5. Introduction to Quality*
• Measurement and Scope:
– How much Quality is good enough?
– Where to start and where to stop?
• Cost-Benefit trade-off
– Example: Five 9s
• How to achieve Quality?
– Is quality about documentation only or standards or
processes or everything?
• Who is responsible for Quality?
* Most of the material in section is from lecture slides of Dr. Darrell Raymond from his course on
Quality Systems (Winter Quality at UW). Used with permission
Software 2011 - An Elusive Target 5

6. Introduction to Quality
• Misconceptions*
– Quality is only about testing and audits
– Quality is the job of Quality department
– Even with all these, you may not have actual
quality or a quality system
* Most of the material in section is from lecture slides of Dr. Darrell Raymond from his course on
Quality Systems (Winter Quality at UW). Used with permission
Software 2011 - An Elusive Target 6

7. Cost of Quality
• Cost of providing acceptable quality
• Two types of costs:
– Conformance costs: defect prevention and inspection
– Non-conformance costs: losses from defects (reworks) and
product failures
• Improvements in quality, therefore leads to
lowered cost because of reduced defects, less
process failures and opportunity costs
Software Quality - An Elusive Target 7

8. Returns to Quality
• Focusing on quality by just reducing costs may
not result in better performance.
• Focus on revenue generation is also important
Software Quality - An Elusive Target 8

9. PMBOK’s Approach
Software Quality - An Elusive Target 9

10. Software Quality

11. Characteristics of Software
• Deming: “If you do not have a competitor today, you will get
one tomorrow”
• Appreciating that software is different because it is
characterised by volatility and rapid change
• Low barriers to entry in the market
• Internet enabled market place provides variety and product
knowledge to customers
• Producing a product without defects is not sufficient
• Knowing the customer and the environment is really
important
• Cooperation and Collaboration are the keys to success
Software Quality - An Elusive Target 11

12. What is Software Quality?
• A recent survey of Chief Information Officers
(CIOs)indicates that ‘‘Improve IT quality” is one of
the top concerns facing IT executives
• But the question is: What is Software Quality?
• “The extent to which software satisfies its users”
• It is not only concerned with meeting user
requirements but also with criteria such as ease
of learning, ease of use, enjoyment to use and so
on…
Software Quality - An Elusive Target 12

13. User Developer Measurable
External quality
features x yes
speed x x yes
space x x yes
network usage x x yes
stability x x yes
robustness x x somewhat
ease-of-use x subjective
determinism x x yes
back-compatibility x yes
security x difficult
power consumption x difficult
Internal quality
test coverage x yes
testability x hard
portability x somewhat
thread-safeness x hard
conciseness x somewhat
maintainability x hard
documentation x subjective
legibility x subjective
scalability x somewhat
Software Quality - An Elusive Target 13

14. Views on Software Quality*
• Transcendental view: Ideal
• User View: Fitness for purpose; product
characteristics; product usability
• Manufacturing View: “Did you build the thing right”;
believes that process improvement will lead to
improved product quality; may lead to standards
• Product View: focus on Internal and inherent
characteristics of a product eg. quality of code
• Value based view: quality is what the customer is
willing to pay for; value for money
* D. Garvin, “What Does “Product Quality” Really Mean?” Sloan Management Review, Fall 1984, pp. 25-45
Software Quality - An Elusive Target 14

15. Measurement of Software Quality*
The way we choose to measure quality depends on the
view point we take…
•User View: Quality concept can be broken down into
directly measurable attributes
•Manufacturing View: measuring defect counts and
rework costs (cost of defect correction)
•Business View: How the product enables a business to
expand and gain more market share
* Kitchenham, B.; Pfleeger, S.L.; , "Software quality: the elusive target [special issues section]," Software, IEEE ,
vol.13, no.1, pp.12-21, Jan 1996; doi: 10.1109/52.476281
Software Quality - An Elusive Target 15

16. Some Measures of Software Quality
• Number of pre- and post-release defects
• Defect Density
• Mean Time Between Failures (MTFB)
• Customer Perceived Quality
Software Quality - An Elusive Target 16

17. Software Quality: Greek Chimera*
• Why?
– the perceived scale of the
problem,
– the diversity of quality defects
in software, and
– the difficulty of factoring high
level quality attributes down to
tangible properties
– Market Structure
– Spread of Internet
– ??
*R. Geoff Dromey, "Cornering the Chimera," IEEE Software, An Elusive Targetpp. 33-43, Jan. 1996, doi:10.1109/52.476284
Software Quality - vol. 13, no. 1, 17

18. Addressing Software Quality
Software Quality - An Elusive Target 18

19. The Software Development Process
Waterfall
Software Quality - An Elusive Target 19

20. The Software Development Process
Agile
Software Quality - An Elusive Target 20

21. The Software Development Process
• Iterative approach should not be reactive and
must be carefully planned
• Frequent and unplanned iterations may lead
to deviations from the desired results
Software Quality - An Elusive Target 21

22. Addressing Product Quality*
• Quality Model Framework:
– Links product properties that influence Quality with a set of high-level quality attributes
• Quality carrying properties (tangible):
– Internal Correctness: Component’s normal form should not be violated eg. Every loop
must have a termination point
– Contextual Correctness: Avoiding redundancy and repetition
– Descriptive: Easy to understand code, variable names etc.
• Software Quality Attributes (intangible):
– high-level attributes like functionality, reliability, efficiency and maintainability
– Process maturity (assuming that it impacts product quality)
• Define linkages:
– Example: redundancy affects efficiency and maintainability
• Implement the model for the key products of software
development: Implementation, Design and Requirements
*R. Geoff Dromey, "Cornering the Chimera," IEEE Software, An Elusive Targetpp. 33-43, Jan. 1996, doi:10.1109/52.476284
Software Quality - vol. 13, no. 1, 22

23. Addressing Product Quality*
• Implementation Quality Model:
– Role of programming languages; language structure and statements;
data (variables, constants etc)
• Requirements Quality Model:
– Must be complete, precise, consistent, explicit, understandable,
implementable, verifiable, traceable and non-redundant
– Prototyping may be used to achieve this
• Design Quality Model:
– The design must show how each of these requirements is to be realized in the
context of the overall system: focus on overall architecture
– Reduced variety of software structure leads to reduced complexity
*R. Geoff Dromey, "Cornering the Chimera," IEEE Software, An Elusive Targetpp. 33-43, Jan. 1996, doi:10.1109/52.476284
Software Quality - vol. 13, no. 1, 23

24. Addressing Product Quality
• Controlling defects controls cost
• But is it possible to prevent defects from
passing on to the customer? Maybe.
• Is testing sufficient?
• What about rapidly changing requirements?
• Maybe a process oriented view is better?
Software Quality - An Elusive Target 24

25. Software Quality - An Elusive Target 25

26. Improving Process for Improved
Software Quality
• Process Improvements:
– Total Quality Management
– Six Sigma
– Failure Mode and Effect Analysis (FMEA)
– Design reviews
– Voice of the Customers
– Continuous Improvement (Including Process
Improvements: CMMI, OPM3)
– Software Quality Function Deployment (SQFD)
Software Quality - An Elusive Target 26

27. Software Quality Function
Deployment (SQFD)
• Focuses on requirements planning phase
• Steps:
– Customer requirements are solicited
– Product specifications are generated using these
requirements
– Customers are asked to correlate between their
requirements and the product specs
– Customers are asked to prioritize their requirements
– Product specification priorities are developed
Software Quality - An Elusive Target 27

28. SQFD - Benefits
• Improved User and Management Involvement
• Shorter development cycle
• Better communication between users and
developers
• Improved quality overall
Software Quality - An Elusive Target 28

29. Total Quality Management*
• Instead of post-process inspection and testing focus
should be on continuous Improvement in processes
and response to user demands
• Improvement in development quality leads to
improved product quality
• An increase in process standardization and tool
capability can significantly increase the level of
development quality
• Does less reporting of defects by customers
necessarily mean higher quality?
*Marcus A. Rothenberger , “Total quality in Software Quality - An ElusiveAn empirical study of quality drivers and benefits
software development: Target 29
in Indian software projects” Information & Management (December 2010), 47 (7-8), pg. 372-379

30. Process Improvement
•In assembled goods industry, strong
operational excellence in internal product
development processes positively impacts
market outcomes
•In the context of software development in
some industries, the relationship between
quality and profitability may not be uniform
across all customers
Software Quality - An Elusive Target 30

31. Process Maturity
• Increased CMM levels could:
• Improved software product quality (less defects)
• lead to less operating cost
• better product reliability
• Adoption of standards by clients resulted in better
requirements (reduced requirements volatility)
• Result in higher development quality
• but…
• Reduction in variance in quality resulting in uniformity
in quality
• doesn’t mean you start producing good software
Software Quality - An Elusive Target 31

32. Software Validation*
• Did you build the right thing?
• Why?
– Usually required for regulated industries
– To ensure design is correct
– To reduce defects after changes are made
• Done in all phases of software development and
everybody participates
• Validation is considered costly, brittle and more
about documentation
• In practice, everybody looks for ways to avoid it
* Most of the material in section is from lecture slides of Dr. Darrell Raymond from his course on
Quality Systems (Winter Quality at UW). Used with permission
Software 2011 - An Elusive Target 32

33. Managing Quality on Projects
• Quality Management is a key area of project
management
• Some recent observations:
– Release the product, fixes will come later
– Minimize the cost at the expense of customer
inconvenience
Software Quality - An Elusive Target 33

34. Role of Quality Standards
• There are so many of them
• ISO, CMM, CMMI, IEEE, ANSI, AIAA
• Used for the certification function
• Can help ensure consistent process quality but
their impact on product quality is unclear
• Can standards be imposed on program
behaviour?
• Maybe be mandatory for some industries and
may also provide legal cover
Software Quality - An Elusive Target 34

35. Will COTS lead to better
Organizational Quality?
• Enterprise Software may be of good quality
but its implementation might not be
• Post-Implementation Processes are more
important than the Software itself
• But Software quality issues may aggravate the
overall implementation
• Overall IT Architecture become more
important and plays an important role in
determining overall quality
Software Quality - An Elusive Target 35

36. Quality in Outsourcing*
• Contracts as a means of managing risk and
quality on project
• Outsourcing (contracts) helps in improving
overall quality
• Clients can focus more on quality by
outsourcing some of their activities
• Fixed Price contracts provide better quality
then Time and Material
* Gopal, A. and Koka, B. R. (2010), The Role of Contracts on Quality and Returns to Quality in
Offshore Software Development Outsourcing. Decision Sciences, 41: 491–516
Software Quality - An Elusive Target 36

37. Is Open Source Software (OSS) the
Answer*
• Does OSS have more Reliability, performance,
scalability, security?
• Equivalent OSS/FS applications
are more reliable
• MySQL database (a leading
OSS/FS database) had fewer
defects than a set of 200 proprietary
programs used for comparison
• Sites using Microsoft’s IIS web serving software have over double
the average time offline than sites using the Apache software
• Surveys report that GNU/Linux systems experience fewer viruses
and successful cracks.
* http://www.dwheeler.com/oss_fs_why.html
Software Quality - An Elusive Target 37

38. Open Source Software Development
Process*
* http://cio-nii.defense.gov/sites/oss/Open_Source_Software_(OSS)_FAQ.htm
Software Quality - An Elusive Target 38

39. Quality In OSS
• Open Source Software can be of higher
quality*
• Users track bugs to their root cause and fix them
• Peer Reviews: Developers review each other’s
code
• Meritocracy
• No resource or time pressure
* http://www.dwheeler.com/oss_fs_why.html
Software Quality - An Elusive Target 39

40. Final Discussion
Software Quality - An Elusive Target 40

41. Quality Issues
• A software is not just a piece of code
• It requires:
• hardware
• People
• Processes
• Policies and
• Other software (Operating systems, databases,
languages, internet and so on)
• Example: Enterprise Software
* http://www.dwheeler.com/oss_fs_why.html
Software Quality - An Elusive Target 41

42. Quality Issues
• So Software Quality is not an absolute concept
– It is difficult to define…
– But easy to blame
• Propagation of internet is changing the
concept of quality
– More and continuous opportunity for testing
– Easy to release upgrades
– But requirements are also continuously evolving
* http://www.dwheeler.com/oss_fs_why.html
Software Quality - An Elusive Target 42

43. Role of Organizational Politics
• Quality Manager is usually
– not a position of power within the organization
– Someone without a strong sponsor
• Is usually viewed as someone
– who creates hurdles (and so increases cost)
– Who is “not on our side”
• Quality Manager has to be more like a PMBOK
Project Manager
Software Quality - An Elusive Target 43

44. Role of Quality Manager
• A principled friend
• Advisor
• Relationship builder
• Wise
• Empathetic
• Positive
• And most importantly: political
Software Quality - An Elusive Target 44

45. Theme for the Term Paper
• How has the Concept of Software Quality
evolved during the last three decades?
• What models and methods for managing
quality have evolved?
• Is there an improvement in Software Quality?
• And is there any empirical evidence for this?
Software Quality - An Elusive Target 45

46. BACK TO SLIDE 1
Thank you
Software Quality - An Elusive Target 46

47. BACK
Source: R. Geoff Dromey, "Cornering the Chimera," Quality - An Elusive Target no. 1, pp. 33-43, Jan. 1996,
Software IEEE Software, vol. 13, 47
doi:10.1109/52.476284