"Earthsoft Foundation of Guidance (EFG) is working as an NGO/NPO for students - Education & Career
guidance and for Professionals for soft skills enhancements. I am working on speading , sharing
knowledge; experience globally.It has uploaded important presentations at http://myefg.in/downloads.aspx.
Also https://dl.dropbox.com/u/83265908/Links-events.xls has links for all ppt files.
Read http://tl.gd/jm1gh5
Be mentor using your education, knowledge & experience to contribute for a social cause & do conduct
free training/ workshop seeking help of existing platforms like rotary,etc
Kindly spread to your friends.Thank you!
- Earthsoft Foundation of Guidance
Let us make earth little softer..
"
The Coffee Bean & Tea Leaf(CBTL), Business strategy case study
2 earthsoft-quality control in software organidation
1. Making earth little softer
Earthsoft Foundation of Guidance
Edge-Aggressive-Reliable-Trust-Honesty-Soft-Obedient-Fun-Transparent
2. Making earth little softer
Earthsoft Foundation of Guidance (EFG) has uploaded following
presentations at http://myefg.in/downloads.aspx
Be mentor using your education, knowledge & experience to
contribute for a social cause & do conduct free training seeking
help of existing platforms. Kindly share with your friends
•Motivation for higher study, Planning for study, Education
guidance, Career guidance, Career available after SSC & HSC
•Personality development – 3 files
•How to prepare resume, Tips to attend interview successfully
•Religion related –To understand basic religion, Do & Don’t tips
•Health related - Be vegetarian, Be healthy, Manage health
•Corporate - Project management, Assertiveness, Ownership,
Effective communication, Leadership, Be entrepreneur
•Finance - To avoid speculation in stock market
•Social - Women empowerment, Choosing life partner, conflict
resolution, stop ragging, stop alcohol, snakebite treatment
About us
3. Making earth little softer
• Organisation
• Quality
• Quality management
• Quality Assurance
• Quality Control
• Processes
• Case studies experience
• Strategic Quality Planning
Index
6. Making earth little softer
Project RAG Status
NOTE: Pl update the RAG status from spectrum
for last 4 weeks. If any of the parameter is Red /
Amber, pl update the action required and status
for the current week.
Parameter Last
Week -1
Last
Week
Current
Week
Action
Required
Status Target
Date
Budget
Scope
Effort
Schedule
Resource
Quality
7. Making earth little softer
• Functionality related gap between expectations
of client and actual delivery
• Defects are injected during Requirement
analysis, High & Low level Design and coding
phase.
• Continuous reviews help to prevent/ remove
defect injection
• Rigorous testing during Unit level, Integration
and application level testing detect the defects.
RA R Desig
n
R Coding R UT IT AT
Achieving Quality
8. Making earth little softer
Reviews
• Review helps
• To confirm client’s requirements
• To detect defects in early stages
• To reduce rework at later phases
9. Making earth little softer
RA Design Coding UT IT AT
Defectsinsystem
Various Phases of SDLC
To be at better quality after every phase
• Removing defects by reviewing & testing
• From each deliverable
• At each phase
Reviews & Testing
10. Making earth little softer
Team structure & reviews
PMO
Reviews
PMO
Reviews
Reviews of deliverables of each member at every
level by supervisor help improving quality
Creating proposal Project Plan, Reports
Following Process & best practices
Review of DD, Coding conventions
Reviews of Proposal; Reports, Escalations
Compliances of the best practices, client
management & business growth
Creating DD, Design Framework, Coding
convention, Reviews of deliverables of
Developers & testers
Creating Specification, Code, Unit
& SI test cases, Testing, etc
Completion checklist, Self review
11. Making earth little softer
Reviews
• Assuring:-
• Quality of design document impacts quality of
the code & the test cases.
• To identify the right resource for the reviews
depending on entity to be reviewed
• Assuring incorporating review comments
• Reviews are discontinued when progress is
delayed. To assure to have reviews as planned
• Tracking review efforts & duration for each
entity (like code, unit test cases, etc) (e.g.
review time per page)
12. Making earth little softer
Reviews
• Deciding on review density before starting the
activity
• Review comments expected to reduce on
repeat cycle
• Cause for concerns
• No. of review comments & No. of major
comments are not decreasing on progress as
expected is indication of unstable quality
• Volume change of entity (e.g. no of pages for
design doc, no. of test cases, etc) pre & post
review
13. Making earth little softer
Code reviews
• Build detailed code checks into the development
process
• Make it impossible to release code that violates
these checks
• Few errors would only be discovered by code
reviews or testing
• Many errors only occurred on specific datasets
14. Making earth little softer
Performance testing
• Execute to check performance of each query
• Create database as appropriate
• Compile & Check manually all the queries
individually for performance time, keep record
• Every SQL statement is run against multiple real
world customer databases
• Improve/ optimise the queries
• Log the bugs for worst performing statements
15. Making earth little softer
Issue Management
• Issues are restraints to accomplishing the
deliverables of the project.
• Typically identified throughout the project and
logged and tracked through resolution.
Issues impact cost, time, quality, most
important is reputation and branding!
Types of issue – Minor, Normal, Major,
Critical, Blocker
Minor issue is dangerous!
16. Making earth little softer
Issues/Risks (non-financial)
Highlight highest risks for management attention
17. Making earth little softer
Metrics-Development & Enhancement
NOTE: Pl highlight the summary from the
metrics report
Work
Packets
Effort
Variance
On-time
Delivery
Defect
Density
Field
Defect
Density
Productivity COQ Review
Effective
ness
Goal Actual G A G A G A G A G A G A
18. Making earth little softer
Metrics-Maintenance & Support
Provide the SLA compliance summary If you use
any tool provide the snap shot of the same.
The number of tickets includes the new received
in the reporting month as well as the backlog
from previous months
Priority Number of
Tickets/Faults
Response SLA
Compliance%
Resolution SLA
Compliance%
Remarks
Goal Actual Goal Actual
P1
P2
P3
P4
19. Making earth little softer
• The perceptions of the user determine the
purpose. Quality therefore is a user perception.
• Correctness : Does it as per users agreed on?
(Unit Level)
• Integrity : Does it maintain accurate data?
(Integration Level)
• Reliability: Does it behave right all the time?
(Application Level)
• Efficiency : Does it run as well as it could?
(Application Level)
• Robustness: Does it handle invalid data?
(Application Level)
Quality & User perceptions
20. Making earth little softer
Testing Improvements
• PM decides on type of testing - automated or
manual or both
• 400% increase in automated regression tests in
the same efforts
• Tests created for all key business flows
• Created by dedicated product QA teams
• Used for
• Ongoing maintenance/bug fixes
• Platform certification (e.g. Linux)
• Technology certifications
• eBusiness Flows certified prior to release
• Progress measured daily, and reported
21. Making earth little softer
Prerequisites of Testing
Test Cases
• Latest & Completed test cases documents from
CVS (or any Configuration management tool)
Test Environments
• Document that gives the details of H/W and S/W
required to start testing
• Identical environment which client would have
while going live
• Development environment
• Testing environment
22. Making earth little softer
Flow of Testing
• Test cases
• Test Data
• Unit Testing
• Screen Layout, look & feel
• Screen Validations
• Functionality
• Integration Testing
• Application Testing
• Users Acceptance Testing
23. Making earth little softer
Testing Guidelines
Why the testing guidelines?
• “Test cases may not cover 100% aspects thus
extend your thoughts beyond the written
statement of the test case.”
24. Making earth little softer
Who should test
There are 2 possible actors
• Developers to execute 100% unit testing
• Reasons being they take the entire ownership of
quality of the delivery
• Second possibility is dedicated testing team
executing test cases
• Testing being skill of expert!
25. Making earth little softer
Unit Testing - UI
To use a check list & test for following
• Consistency- Control sizes
• Basic functionalities
• Tables, navigation to next page
• Search listing result
• Validation error message
• Layout (as per Prototype)
• Spelling Mistakes
• Font (Type & Size)
• Icons display
• Color Scheme
• Screen resolution
26. Making earth little softer
Unit Testing - UI
• Tab sequence & Text selection
• Links (Check each link to ensure it works)
• Current selection indication
• Alignments (Text, Numbers, Row, Column)
• Compatibility with various browsers
• Duplicate display of information
• Mention of Copyright
27. Making earth little softer
Unit Testing - Validations
• Field type & Size (Numeric, Alphanumerical,
Database field width)
• Formatted fields (e.g. Date, E-mail, Phone
Numbers, Security ID)
• Mandatory Fields
• Messaging (Alert, Warning, field message,
message post submissions, sequence, etc)
• Database Records verification
• Verify the values stored in database for Radio
buttons & Checkboxes
• Test spaces trimming (leading & trailing spaces)
• Update By & Update on fields
28. Making earth little softer
Unit Testing - Functionality
To use matrix of screens & functionalities
e.g. File Upload/Download
•Try all types of files – image files, office files,
animated GIF
•Try downloading files – verify they don’t contain
garbage
•Search using special characters like (%,_,’ etc.)
Sorting functionality - Check the number of
records before and after the sorting are same
• In case of mailing functionality, check for non-
existing e-mail address. (The mail should be
returned back to some preset mail address)
29. Making earth little softer
Unit Testing - Functionality
To use matrix of screens & functionalities
•Test User login Case Sensitivity
•Test the dependant behaviors of the controls
•Enabling / Disabling based on some actions
•Data shown in Popup Screens based on some
actions (Refreshing)
•Language Support Test (Wherever applicable
Japanese / German/ French, etc Input)
•Test readability by changing encoding type in the
browser
30. Making earth little softer
Integration Testing
• Refer Integration Matrix while testing
• Enter boundary values (Min , Max, 0 , negative
etc)
• Errors occur when values passed by once
screen are used in calculations on other screen
31. Making earth little softer
• Testing Business Transactions
• Refer Screen Flow Diagrams while identifying
Business Transactions
• Identify the User Roles to perform testing (Sales
Person, Purchase Manager, Dispatch Person
OR User , Administrator)
• Ensure Test Data Variety & Test data Volume
• Roles & Rights testing (Test for access rights)
• Logging Mechanism testing (Errors and User
Actions)
• Deployment Testing (Perform Deployment as
per the Deployment Document)
Application Testing
32. Making earth little softer
• Application Menu should be in sequence of the
functionality
• User Management->Master Data Mgt-> Order
Entry->Purchase Mgt->Dispatch
• Navigation
• Using Browser ‘Back’ & ‘Forward’ buttons
• Using ‘Back Button’ & ‘Links’ in the application
• Browser dependency
• Test for all the browsers as per client request
(Min should be IE & Netscape)
• Disable Cookies and try to login or perform
Transactions
Application Testing
33. Making earth little softer
• Perform transaction using multiple instances of
application, Open other applications like (Word,
Excel, Access, Others)
• Quick Actions in the Application
• Click Submit button twice
• Switch between the screens quickly
• Enter data in one instance & try to see it
immediately in another instance
• Random operations
• Enter any number of characters and click
anywhere
• Clicks on read only area
Application Testing
34. Making earth little softer
• Copy Paste of the URLs
• Copy URL after Logging in and paste it in
another instance of browser
• Same operation while in the middle of a
business Transactions
• Session Timeout verification (test for < / = / >
defined timeout value)
• Disconnection Test (to verify there is no
inconsistent data generated)
• Disconnect Net connection in the middle of a
transaction
• Stop Database service while transaction is in
process on the client
Application Testing
35. Making earth little softer
• Reports printing
• Print Multiple Pages
• Try to print without installing any printer driver
• Verify Report Margins for the printed copy
• Status related tests (Display only ‘Active’ / ‘Live’
records)
• Multi-user Testing
• Concurrent users (Different users / users with
different roles)
• Concurrent Transactions (Add / Update /
Delete / File Upload)
• Locking Issues Test
• Common Tests…ref file
Application Testing
36. Making earth little softer
How Much Testing Is Enough?
• There is always one more bug…
• So how much testing is needed to find “all” ?
• Cost
• Time to market
• Feasibility
• Rate of detection of defects
• Industry-specific standards
• To predict how many bugs will be in a release?
• In order to provide correct budget and schedule
estimates, QA managers need the means to
predict fault levels.
37. Making earth little softer
Types of defects
• Blocker
• Severe/ Critical
• Major
• Normal
• Minor
• Trivial
• To publish phase wise defect data for each
module for above types
• Responsibility wise
Sample statistics
38. Making earth little softer
Update defect passbook at end of day and check
status and trend
Defect passbook
39. Making earth little softerConfidential 41
QAG measures Quality
QAG tracks and measures quality using following
template
Organisation
40. Making earth little softer
Quality measurement
No. of defects 1
effort in manmonths 3
Loc per hr 10
Lines of code 4800
no. of defects 1
% of defects to LoC 1*100/4800=2.1%
Achieved quality 97.9167%
• 1 defect detected by customer for 3 man
months of efforts is said to be fairly good quality
• Count each type of defect (minor, normal,
major, critical or blocker) ; Minor defect is worst!
41. Making earth little softer
Measure where are you
• To set a quality baseline & remain determined
• To seek feedback
• To measure and conclude current levels from
desired level
• To draw action plan to be at set goal
0
1
2
3
4
5
6
7
8
Vertex NTTD
Desired level
Current Level
Current level Desired level
42. Making earth little softer
Aim for Defect Metrics (sample)
Phase Metrics Unit
Lower
Limit
Avera
ge
Upper
Limit
Requirement
definition
Requirement
documents
Review
comments
Per 100
pages
9.8 15 24
Design
Detail design
(DD)
Review
comments
Per 100
pages
7.6 13 21
Program design
(PD)
Review
comments
Per 100
pages
4.9 12 30
Coding
Unit Testing
Test Case
Density
KLOC 65 110 170
Defect density KLOC 3 6.4 9.7
Integration
Testing (SI)
Test Case
Density
KLOC 16 39 92
Defect density KLOC 0.7 2.2 3.6
Testing
System Testing
(PT)
Testing density KLOC 2.3 9.5 39
Bug detection
density
KLOC 0 0.7 1.4
Entity which is measured, can be managed well
43. Making earth little softer
Bug content analysis
Phenomenon Action example
Module having
major change
Suspend test work
Review of design document, review of source
code with the designer
Enhanced test according to test items, bug
contents
Test density <
allowable range
Enhanced test according to test items, bug
contents
Bug detection
density is less
than allowable
Enhanced test according to test item, bug
contents
Bug detection
density exceeds
allowable range
As per 1 above
44. Making earth little softer
Bug content analysis
Phenomenon Action example
Bug cause
Problem of
defective
specification
Re-check specifications for client, business
Design document review with an expert
Indetermination of specifications of design
doc. ambiguous part
Bug cause
Problem of
development
from specification
Confirmation of specification to designer
Review of design document & source code
with the designer
Clarification of implementation,
implementation method
Bug - Other than
specification
Thoroughness of walk through and source
code review
Occurrence type
Environment
Specify reason according to occurrence
period, contents
Re-examination of test environment, system
45. Making earth little softer
Bug content analysis
Occurrence type Action example
Operational error,
without reproduction,
identification
mistake)
Review of test item, process
Re-adjustment of design document, test
environment
Educational guidance to designer
Coding/ modification
mistake
Thoroughness of source code review
Occurrence of
important bug that
impacts test
progress
Suspend test work
Review of design document & review of
source code
Enhanced test according to the test item,
bug contents
occurrence of similar
bugs
Check the latency of a similar bug in
another module
46. Making earth little softer
• No. of pages/ characters of design document &
LOC of the code
• Check planned No. of test cases (a statistical
index page for test cases shows the progress)
• Test cases planned & actually execution
• To capture No. of test cases executed on daily
basis and cumulative basis
• No. of defects detected on daily and
cumulative basis – check if volume is apt
• Calculate defect density–check if volume is apt
• Check if any delay in execution, understand &
remove the delay
Quantitative analysis
47. Making earth little softer
•Check the pending test cases on daily &
cumulative basis – check if volume is apt
•Bug fixing on daily and cumulative basis - check
if volume is less or more
•Check if any delay in fixing, understand &
remove the delay
Quantitative analysis
Matrix Analysis
Test density Low Target value Upper limit
Bug Density -Low Less Bugs
detection
Retest adding few
more test cases
Member skill level?
Target Value Write additional
test cases
Judge verification of
the result qualitatively
Judge verification of
the result qualitatively
Upper Limit Multiple bugs
occurred
Multiple bugs occurred Suspend testing, get
into other measure like
code review, etc
48. Making earth little softer
Quantitative analysis
# Category Details Type
1 Specification
Inadequacy
Description Inadequacy Interface Error
Logical Error
Data definition error
Table definition error
Format Error
Others
Wrong description
Ambiguous
Standard Violation
Document not corrected
Document not consistent
2 Issue while
deploying
Slipping over the specification
Lack of understanding the
Lack of confirming the specification
Insufficient investigation of specs
3 Issues
except the
matters
related to
specification
Less knowledge for language usage
Halfway publicize / communication
Standard violation
Not checked while re-using
Not checked while modifying
Unforced mistake
49. Making earth little softer
• Root cause analysis – assign root cause to
each bug & conduct analysis (fishbone diagram)
• Severity wise
• Developer wise
• Tester wise
• Defect injection phase v/s defect detection
• Defect injection phase v/s root cause
• Daily status (like a bank passbook)
Quantitative analysis
Defect detected RA-Review HLD, IT -Review LLD, UT-Review Code review UT IT AT
Defect injected
RA
HLD
LLD
Coding
50. Making earth little softer
Process
• This PCB is released as Organization PCB
• This PCB should be referred to when providing
Organizational Data to the customers
• Each project depending on the size, complexity
and customer requirement shall define the
metrics for the project.
• This would be done in Process Plan as part of
the project initiation.
• In case the project targets are outside the UCL
or LCL limits an approval would be required
from the respective VP.
51. Making earth little softer
PCB Contents
• Productivity Data
• Phase wise effort distribution
• Effort – Actual Vs. Estimated
• Schedule – Actual Vs. Estimated
• Defect Metrics
52. Making earth little softer
Productivity Data
LOC/ 1 MM
Productivity LCL Target UCL
Java 700 1000 1300
VB.Net 800 1500 2200
C# .net 700 950 1200
C 700 1000 1300
This data applies to all the efforts spent on all
SDLC Phases – from RA to Acceptance testing
To finalise parameters for data to be captured
To create PCB data for each project
Thus organisational data will be created
Analyse & use the data
53. Making earth little softer
Plan to Collect Data
• PCB shall be updated every three months, End
of every quarter - Jun, Sep, Dec, and Mar
• For development, testing & maintenance project
• Finalise project metrics during kickoff of project
• The project metrics collection plan shall be
finalized as part of the project initiation process
(Process Plan)
• The projects shall contribute Phase End data or
Milestone data every six months
• To share PCB data during project closure
54. Making earth little softer
Project Closure
To conduct a meeting with all stake holders
To share following information estimated & actual
• Efforts
• Schedule
• Defect Matrices
• Profitability
• Client happiness Index
Detailing
• Key learning
• Best performance
• What next?
Remember
55. Making earth little softer
Thank You
Author has worked at as Sr management professional
Vertex Software, now NTTData
Tech Mahindra
SoftBridge Solutions, India
& extremely thankful to these organisation for offering exciting
opportunity to work in related areas, gain experience & contribute
& a gentleman having deep passion about quality..Mukand
Bhagwat
Hinweis der Redaktion
Above processes remove the defects
Reviews at each phase during SDLC removes the defects and quality improves
Global Standards Compliance Checker - GSCC
250+ detailed code checks
Built into the development process
Impossible to release code that violates these checks
Started with 11i6, previously, these errors were only discovered by code reviews or testing
Many errors only occurred on specific datasets
Automatic Generation of Patch Database (DB) Drivers
DB driver is portion of patch that updates schema, seed data and pl/sql
DB driver errors accounted for ~20% of patch install errors
Started with 11i6, previously these were manually coded, which was extremely complex and error prone
Automated Performance Analyzer
Every SQL statement (~280,000) is run against multiple (5) real world customer databases
Worst case Explain plan is analyzed
Bugs automatically logged for worst performing statements
This activity performed every month
Fixes consolidated into next release
Critical issues fixed as one-off fixes
~1,500 issues fixed monthly
Started with 11i5, previously these were only caught by performance testing
Database group uses this on new versions of the database
100+ bugs fixed prior to release of 9iR2
Issues not easily resolved are escalated for resolution.
Issues are typically identified throughout the project and logged and tracked through resolution.
In this section of the plan the following processes are depicted:
Where issues will be maintained and tracked
The process for updating issues regularly
The escalation process
The vehicle by which team members can access documented issues
Issue… already impacting the cost, time or quality
Risk… POTENTIAL negative impact to project
Following factors make the quality software
400% increase in automated regression tests,
Tests created for all key business flows
Created by dedicated product QA teams
All products now have RTs
Used for
Ongoing maintenance/bug fixes
Platform certification (e.g. Linux)
Technology certifications (e.g. 9iR2)
eBusiness Flows certified prior to release
With 11i7, 60+ Oracle consultants worked for 2 months to pre certify Oracle implementation flows
Progress measured daily, and reported to executive management
(By changing system dates)
After server side validations such as &quot;Duplicate ID Check&quot;, retry entering valid data and save
Read only fields Test (press enter key or click to verify that no unexpected events are getting fired)
“There is always one more bug.”
The amount of testing needed to ensure quality is hard to quantify. Very few project teams are able – or are willing – to predict how many bugs there will be in a release. Testing and fixing until all bugs have been found would be a very expensive task, if it were possible. And is it reasonable? Most organizations prefer to deliver a product on time, with a few unimportant defects, rather than to deliver it six months late. Needless to say, each industry has a different level of acceptable quality, for instance, word processors versus aircraft systems.
In order to provide correct budget and schedule estimates, QA managers need the means to predict fault levels.