1. BW3
Session
6/8/16 11:30 AM
End-to-End Quality Approach:
14 Levels of Testing
Presented by:
Adrian Thibodeau
Standard & Poors Rating Services
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888---268---8770 ·· 904---278---0524 - info@techwell.com - http://www.techwell.com/

2. Adrian Thibodeau
Standard & Poors Rating Services
Vice president Adrian Thibodeau has led the QA & Metrics team at Standard &
Poor's Ratings Services since 2011 and held numerous other roles since 2005. For
the past four years, Adrian has led the Ratings QA department in a number of
transformational initiatives: adoption of the TMMi Operational Maturity Model,
revision of its automation framework, a shift to metric-driven reporting, and the
expansion of the team's quality leadership into static code analysis and data.

3. 5/11/2016
1
End-to-End Quality Approach:
14 Levels of Testing
Better Software Conference 2016
June 8, 2016
Chintan Pandya
Senior Director, QA & Metrics
Adrian Thibodeau
Vice President, QA & Metrics
S&P Global Ratings Services / IT
Copyright © 2016 by S&P Global.
All rights reserved.
• Intro and Overview of S&P Global Ratings
• Lowering the Cost of Quality
• 14 Levels of Testing
• Wrap-Up
Agenda
2

4. 5/11/2016
2
S&P Global Ratings: Global Reach
• In business for 150+ years
• Provides global reach and
local knowledge with an
office network spanning
26 countries
• Approximately 1,400 credit
analysts covering 125+
countries
• More than 1 million credit
ratings outstanding
• More than $4.3 trillion in
new debt issuances in 2014
• S&P Global Ratings analytical teams provide high-quality analysis, data
and research across government, corporate, financial sector and
structured finance entities & securities.
3
• A discrete organization within S&P Global Ratings’ IT group, providing
direct support to distinct portfolios of application development
• Represents ~6% of IO full-time staff & ~18% of project-based
consulting resources
• Overall, Ratings IT leverages a primarily off-shore consultant contract
model
• Agile/scrum model, with some waterfall on legacy projects
- Sprint duration varies from 1-3 weeks
- Practice maturity varies greatly by the project and experience of the team
S&P Global Ratings: QA & Metrics
4

5. 5/11/2016
3
Lowering the Cost of Quality
5
Early Defect Prevention
6
• Our program seeks to improve our processes around quality and the
early detection of defects. It is not a guarantee that Production
defects won’t sometimes occur.
All code, to all environments:
• IDE to Dev
• Dev to Sys Integration
• Sys Integration to QA
• QA to E2E/UAT
• UAT to Prod
Dev / IDE
Sys Int
QA
UAT
Prod
No defects
”
”
”

6. 5/11/2016
4
End-to-End Quality Approach
7
Quality
Requirements
Creation and
Story Verification
12 Levels of
‘Build’ Testing
Code
Deployment &
Production
Verification
14 Levels of Quality…
• Organization-wide accountability to:
• Our standards
• Our process
• Our users
• Our peers
• End-state quality – reduce Production defects
• Embrace transparency, improve understanding, and opportunities
for improving quality
• Manage the accumulation of technical and business debt, help
reduce maintenance costs
• Quality early and in every step of our process, and a commitment not
to pass poor quality on within our SDLC….an outcome-focused
orientation
Why It Matters
8

7. 5/11/2016
5
The Costs Add Up:
• Risk to project delivery timelines from re-work or unexpected failures
• KTLO support costs increase across IT and project budgets fall short
• Velocity of teams slows down
• Creativity and innovation hampered
• IT is less agile to delivering business values
• Business users are frustrated by delivery delays, outages, and Production
issues
Cost of Quality
9
Source: Boehm, Barry and Philip N. Papacco. ‘Understanding and Controlling Software Costs,’ IEEE Transactions on Software Engineering, v. 14, no.10,
October 1988, pp. 1262-1477
• An assessment based on the known relationship that defects found later
in the SDLC (e.g., Prod deployment) cost an organization more than those
found earlier.
• Weighting factors to the types of defect tickets raised
• Compare the ratio of tickets found in lower environments vs Production,
identifying the “escaped” effort
• This helps identify the areas & cost (lost effort) of poor early quality
Illustrative Example:
Cost of Quality Calculation
10
Tickets
Environment Data TrainingCoding Enhancement
DDP – 97% DDP – 28% DDP – 5%DDP – 97% DDP – 77%
Volume – 28% Volume – 65% Volume – 22%Volume ‐ 14% Volume – 9%
86% 7% 2%81% 80%28% 65% 22%14% 9%
Effort Contained
Cost of Rework
Ticket Volume %
Defect Detection %
Defect Detection % (DDP) = [(Total # of Defects found pre-Prod) / (Total # of Defects found pre-Prod) + (Total # of Defects found in Prod)] * 100

8. 5/11/2016
6
• Improved Quality delivered to Production: Increased confidence in
Production quality, reduction of re-work
• Early Awareness of Quality Issues: Increased transparency throughout
the SDLC of quality levels
• Repeatable / Transparent Quality Process: Steps towards the
enablement of DevOps Processes & Tools
• Alignment with our vendors: S&P Global Ratings App Dev & Testing
Standards for vendor contract and quality management
14 Levels of Testing: Target Outcomes
11
• Beyond just formal test execution, establish quality accountability
• Encourage high quality early in the SDLC, govern formal testing levels
• In the overall quality process, establish:
- Governance
- Transparency
- Traceability
- Confirmation
The Role of QA Across the 14 Levels
12

9. 5/11/2016
7
14 Levels of Testing
13
14 Levels of Testing
14
Individual
Development
Environment
(IDE)
Unit Testing
Peer Code
Review
Static Code
Review
Over the
Shoulder Testing
Dev
Tech Lead Code
Review
Continuous
Integration
testing (including
Security Testing)
System
Integration
Functional
Testing
Story Acceptance
Testing
QA
Functional
Regression
Testing
Pre-Prod / UAT
User Acceptance
Testing
End 2 End (RRG)
Testing
App Scan
Quality Requirements Creation & Story Verification
Quality Code Deployment & Production Verification

10. 5/11/2016
8
14 Levels: Execution
15
Test
Level
Test Target Environment
Accountable Owner /
Participants
1
Quality Requirements Creation & Story
Verification
Sprint Planning Dev / QA / BA / PO / SMEs
2 Unit Testing
Individual Development
Environment (IDE)
Dev
3 Peer Code Review (pre-commit) IDE Dev
4 Over the shoulder Testing IDE Dev / QA
5 Static Code Review IDE Dev
6 Tech Lead Code Review (Post commit) Development Dev
7 Continuous Code Integration Testing Development ISG / QA / Dev
8 Functional Testing System Integration QA
9 Story Acceptance Testing System Integration Product Owner (PO)
10 Functional Regression Testing QA QA
11 End-to-End Testing (RRG) End-to-End QA
12 User Acceptance Testing UAT Stakeholders / Users
13 App Scan Pre-Production DM
14 Code Deployment & Production Verification Production QA / Ops / SMEs
Levels 1-7
16
Individual
Development
Environment
(IDE)
Unit Testing
Peer Code
Review
Static Code
Review
Over the
Shoulder Testing
Dev
Tech Lead Code
Review
Continuous
Integration
testing (including
Security Testing)
System
Integration
Functional
Testing
Story Acceptance
Testing
QA
Functional
Regression
Testing
Pre-Prod / UAT
User Acceptance
Testing
End 2 End (RRG)
Testing
App Scan
Quality Requirements Creation & Story Verification
Quality Code Deployment & Production Verification

11. 5/11/2016
9
Level 1: Requirements
17
Quality Requirements Creation & Story Verification
Common
Understanding:
• Requirements
• Impact
• Scope
• Data
• Architecture
• Solution
• SMEs
Business-Value:
• Who has
requested?
• Who will use?
• How will they
use it?
• Who will
accept?
• Who will take
part in UAT?
Risk-Analysis:
• What are the
known risks of
this platform?
• What are the
risks of this
change?
• What do we
need to test?
• How will we
deliver to
target quality?
Quality Requirements Development and Story
Verification
18
Centralized Requirements
‐ Traceable & Verifiable
- Unambiguous
- Consistent & Complete
- Prioritized
- Scope
Impact Assessment
- Architecture
- Interdependent Apps
- Data
- RRG Criteria
Project Scope & Business Value
- What is the change being made?
- What value is being delivered & for whom?
-How will we know that we succeeded?
- What are our quality criteria?
Knowledgeable SMEs & Involve
the Right Teams
- Are all teams involved that need to be?
- Are the SMEs informed of impact?
- Is there agreement on the approach?
Common
Understanding
• Plan before we build, and ensure everyone has a common
understanding
• Plan for quality early, raise awareness of risks, estimate appropriately

12. 5/11/2016
10
Levels 2-5: IDE
19
Individual
Development
Environment
(IDE)
Unit Testing
Peer Code
Review
Static Code
Review
Over the
Shoulder
Testing
• Defined standard for Unit Test code coverage and methods
on new code
• Commitment towards review and coverage targets for legacy
code across all apps
• Defined process for Development peer code review
• Designated peer review rosters maintained by vendors and
managers
• Enablement of Static Code Unit Test Code coverage scanning
and reporting
• Creation of Static Code rulesets to enforce coding best-
practices
• Early engagement between Development and QA to review
complex requirements and test cases
• BA engagement and sign-off as required, eliminate Dev/QA
requirement assumptions
Levels 6-7: Development
20
Dev • Defined process for Development Tech Lead / SME code
review and sign-off
• Designated peer review rosters maintained by vendors and
managers
• Review compliance with ‘Build’ process SOPs
• Automated testing of critical integration paths
• Web Services testing
• Mandatory Static Code scanning of code repository, ‘Critical’
defects cannot pass: Security, Performance, Code-Syntax
rules

13. 5/11/2016
11
Levels 1-7 Stage Gates & Measurement
21
Test
Level
Test Measurement Goal
1
Quality Requirements Creation & Story
Verification
Stories w/ Risk Score Assigned
• Risk Score assigned for 100% of
stories
2 Unit Testing Unit Test Coverage %
• New Code: 100%
• Legacy Code: based on targeted
progression plan
3 Peer Code Review (pre-commit) % Peer Review Coverage
• 100% of stories reviewed
4 Over the shoulder Testing
Confirmation from Dev & QA of
occurrence for High Risk stories
• 100% coverage of High Risk Stories
• ‘0’ open ‘Requirement Gap/Missing
Requirement’ Defects
5 Static Code Review % of builds scanned under SCA
• ‘0’ Critical violations: Security or
Performance
6 Tech Lead Code Review (Post commit)
% Tech Lead Review Coverage • Minimum: 100% review of High &
Medium Risk Stories
7 Continuous Code Integration Testing Open Integration or SCA Defects • ‘0’ open Critical or High Defects
Levels 8-13: Formal QA
22
Individual
Development
Environment
(IDE)
Unit Testing
Peer Code
Review
Static Code
Review
Over the
Shoulder Testing
Dev
Tech Lead Code
Review
Continuous
Integration
testing (including
Security Testing)
System
Integration
Functional
Testing
Story Acceptance
Testing
QA
Functional
Regression
Testing
Pre-Prod / UAT
User Acceptance
Testing
End 2 End (RRG)
Testing
App Scan
Quality Requirements Creation & Story Verification
Quality Code Deployment & Production Verification

14. 5/11/2016
12
QA Governance and Monitoring
23
QA Portal – S&P QA developed, web-based, workflow application to help
govern and monitor ZDP and formal QA testing.
Project
Profile
Release
Management
Test
Phases
Checklist
QA
Artifacts
QA Go /
No-Go
• Resource Management
• Technology mapping
• Nature of project
• Integrated app impact
analysis
• Enforce standard test
structure in release
management tool
• Release scope
• Compliance check
• Checklist for Test phase
• Test planning, Design,
Execution, UAT, E2E and
closing
• Entry and exist criteria
verification
• Live defect summary
• Integration with central
repository
• Peer review / signoff tracker
• Process Audit reports
• QA recommendation
• Quality gates by Env
• ZDP: 8-12 monitoring
• Code Build governance
• Levels 2-7
Reporting
& Metrics
Project Name
Sprint 1
Sprint 2
QA Governance and Monitoring
Project
Profile
Release
Management
Release
June 5, 2016
Sprint N
Release
Management
Requirement
Test Case Design
Test Execution
Bug Tracking
Project Name
Release
June 5, 2016
Sprint 1
Sprint 2
Sprint N
User stories
QA Portal Test Management Tool
Release
Management
Requirement
Test Case
Design
Test Execution
Bug Tracking
Kanban Items
QA Artifacts
management
24

15. 5/11/2016
13
QA Governance and Monitoring
Release
Management
Test phases
checklist
Test Planning
Test
Designing
Test
Execution
Go TO QA
/
Conditional
Go To QA
QA
Recommenda-
tion
Test
Execution
Go TO UAT
/
Conditional
Go To UAT
UAT Testing
Go TO Prod
/
Conditional
Go To Prod
E2E Testing
Test Planning
UAT Planning
Prod
Verification
Signoff
Production
Verification
Production
verification
checklist
Reporting & Metrics
25
Story Certification (Functional Testing)
Test
Planning
Test
Designing
Test
Execution
Go TO QA
/
Conditional
Go To QA
Test Planning
• Test Plan creation
(Automated process)
• Peer Review
• Requirement review
• Impact assessment
• Risk assessment
• Test planning checklist
Test Designing
• Test case creation
• Peer Review
• Requirement traceability
• Impact assessment
• Test designing checklist
• TDD (Over the shoulder
testing)
Test Execution
• Story certification
• Requirement traceability
• SI environment set up
• Test data creation
• Test execution checklist
• ZDP check gate (QA
Recommendation)
Levels 4-5 Levels 8-9Level 1
Release
Management
Test phases
checklist
26

16. 5/11/2016
14
Regression Testing
Test
Planning
Regression
Execution
Go TO UAT
/
Conditional Go
To UAT
Test Planning
• Regression test
• Risk assessment
• Automation script update
• Test data analysis
Test Execution
• Regression Testing
• Defect triage
• Reporting and metrics
• QA environment set up
• Test data creation
• Test execution checklist
• ZDP check gate (QA
Recommendation)
Level 10
QA Deliverables
• Test summary
report
• Quality Risk log
• Estimate vs Actual
Effort
• Performance / load
test results
• Defect Metrics by
status
• Environment
performance stats
Release
Management
Test phases
checklist
27
Go TO Prod
/
Conditional Go
To Prod
UAT & E2E Testing
UAT
Planning
E2E - UAT Planning
• UAT Plan creation
• UAT preview session
• UAT script creation / peer
review
• E2E Testing Impact
analysis
• Cross org test case
review and schedule
UAT Coordination
• Ready-Set-Go Email
• Conduct UAT session
• Set up Feedback session
• UAT ticket management
& communication
• UAT summary report
• Complete UAT checklist
on Portal
Levels 11-12
E2E Testing
• Cross team E2E test
planning
• E2E Test execution
• Collect E2E signoff
• Complete E2E checklist
on QA portal
UAT TestingE2E Testing
Release
Management
Test phases
checklist
28

17. 5/11/2016
15
QA Portal
1
2
3 4
29
QA Portal: QA Recommendations
30

18. 5/11/2016
16
Levels 8-13 Stage Gates & Measurement
31
Test
Level
Test Measurement Goal
8 Functional Testing Functional Defect Counts
• ‘0’ open defects
• Rejection Rate <5%
• Re-open Rate <10%
9 Story Acceptance Testing Product Owner sign-off of stories
• 100% sign-off
• ‘0’ Enhancements from
missed requirements
10 Functional Regression Testing Regression Defect Counts
• ‘0’ open defects
• Rejection Rate <5%
• Re-open Rate <10%
• ‘0’ defect ‘Deferrals’
11 User Acceptance Testing UAT Defect Counts
• ‘0’ open defects
• ‘0’ Enhancements from
missed requirements
12 End-to-End (RRG) Testing E2E Defect Counts • ‘0’ open defects
13 App Scan
App Scan Defect Counts
• ‘0’ open defects
Level 14: Deployment Verification
32
Individual
Development
Environment
(IDE)
Unit Testing
Peer Code
Review
Static Code
Review
Over the
Shoulder Testing
Dev
Tech Lead Code
Review
Continuous
Integration
testing (including
Security Testing)
System
Integration
Functional
Testing
Story Acceptance
Testing
QA
Functional
Regression
Testing
Pre-Prod / UAT
User Acceptance
Testing
End-2-End (RRG)
Testing
App Scan
Quality Requirements Creation & Story Verification
Quality Code Deployment & Production Verification

19. 5/11/2016
17
Code Deployment & Production Verification
33
Build
Management
• Adherence to Build standards and processes, DevOps where enabled
• Maker-Checker: confirm the build for release is the same as has been
certified
• Confirmation: ZDP levels adherence, Build ID, Build Master, etc..
Change
Management
• Adherence to Change Management processes: Change ID#, Build ID
• Follow standard Deployment procedures and validations
• Be prepared with a roll-back plan
Production
Verification
• Document all planned activities in Production Control Docs
• Adhere to Production environment access policies and restrictions
• Engage business users for real-world scenario / ‘write’ validations
• Report any deviations from lower-environment certification
• In‐alignment with mandatory Build‐QA‐Deploy procedures:
Production Verification
Go TO UAT
/
Conditional
Go To UAT
Test Planning
• Create Prod verification
plan
• Get approval on prod
control document
• Break glass approval
Prod Verification
• Perform Prod verification
• Story certification
• Send notification to
project team
• Complete Prod
verification checklist
Level 14
Production
Verification
Verification
checklist
Release
Management
Test phases
checklist
34

20. 5/11/2016
18
Level 14 Stage Gates & Measurement
35
Test
Level
Test Measurement Goal
14
Quality Code Deployment &
Production Verification
• Post deployment defects
• Successful deployment
process
• ‘0’ post deployment defects
• Incident free deployment
Wrap-Up
36

21. 5/11/2016
19
• 7 levels (4, 8-12, 14) fully deployed since Q3 2015, remaining
being revised and strengthened: training, tool on-boarding, and
documented standards and procedures
Wrap-Up: Progress-to-Date
37
65% Aging Defect
Backlog
53% Production
“Bug” ticket
Volume
25% Defect
Discovery in all
QA Test Phases
3% Existing Issue
identification in
Functional testing
vs Regression or
UAT
6% Build
Configuration
Defects
Early Defect Detection
11% Coding
Issues found in
Functional testing
vs Regression or
UAT
Q3 2015 – Q1 2016
‘14 Levels of Testing’:
• Quality focus needs to be owned by all IT disciplines and a
collaborative conversation with your business
• Testing for quality begins from requirements and Development efforts
and collaborative test planning
• Understand your risks, plan for quality early
• How many “Levels” is dependent on your organization
Wrap-Up: Reducing Cost of Quality
38

22. 5/11/2016
20
Process & Engagement:
• Defined and repeatable operational policies and procedures (SOPs),
across IT disciplines
• Strive for standardization, but allow for exceptions when appropriate
• Automate where possible, and when ready
• Communicate your expectations: Document and communicate your
Development and Testing standards to your vendors
• Process metrics and reporting to facilitate the identification,
assignment, and remediation of process gaps; KPIs that inform of
opportunity, not assign blame
Wrap-Up: Reducing Cost of Quality
39
Accountability and Empowerment:
• Understand your ‘Cost of Quality’ equation
• All IT staff need to be trained on policies and procedures, and
empowered to say ‘no-go’ to poor quality
• Foster transparency into the state of quality and progress within the
testing process;
• Continuous Improvement mindset: Can we value or improve what we
don’t measure?
Wrap-Up: Reducing Cost of Quality
40

23. 5/11/2016
21
Permission to reprint or distribute any content from this presentation requires the prior written approval of
S&P Global Ratings. Copyright © 2016 by Standard & Poor’s Financial Services LLC. All rights reserved.
Thank You
Chintan Pandya
Sr. Director – QA & Metrics
chintan.pandya@spglobal.com
Adrian Thibodeau
VP – QA & Metrics
adrian.thibodeau@spglobal.com
Copyright © 2016 by Standard & Poor’s Financial Services LLC. All rights reserved.
No content (including ratings, credit-related analyses and data, valuations, model, software or other application or output therefrom) or any part thereof (Content) may be modified,
reverse engineered, reproduced or distributed in any form by any means, or stored in a database or retrieval system, without the prior written permission of Standard & Poor’s
Financial Services LLC or its affiliates (collectively, S&P). The Content shall not be used for any unlawful or unauthorized purposes. S&P and any third-party providers, as well as their
directors, officers, shareholders, employees or agents (collectively S&P Parties) do not guarantee the accuracy, completeness, timeliness or availability of the Content. S&P Parties
are not responsible for any errors or omissions (negligent or otherwise), regardless of the cause, for the results obtained from the use of the Content, or for the security or
maintenance of any data input by the user. The Content is provided on an “as is” basis. S&P PARTIES DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, ANY WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE OR USE, FREEDOM FROM BUGS,
SOFTWARE ERRORS OR DEFECTS, THAT THE CONTENT’S FUNCTIONING WILL BE UNINTERRUPTED OR THAT THE CONTENT WILL OPERATE WITH ANY SOFTWARE
OR HARDWARE CONFIGURATION. In no event shall S&P Parties be liable to any party for any direct, indirect, incidental, exemplary, compensatory, punitive, special or
consequential damages, costs, expenses, legal fees, or losses (including, without limitation, lost income or lost profits and opportunity costs or losses caused by negligence) in
connection with any use of the Content even if advised of the possibility of such damages.
Credit-related and other analyses, including ratings, and statements in the Content are statements of opinion as of the date they are expressed and not statements of fact. S&P’s
opinions, analyses and rating acknowledgment decisions (described below) are not recommendations to purchase, hold, or sell any securities or to make any investment decisions,
and do not address the suitability of any security. S&P assumes no obligation to update the Content following publication in any form or format. The Content should not be relied on
and is not a substitute for the skill, judgment and experience of the user, its management, employees, advisors and/or clients when making investment and other business decisions.
S&P does not act as a fiduciary or an investment advisor except where registered as such. While S&P has obtained information from sources it believes to be reliable, S&P does not
perform an audit and undertakes no duty of due diligence or independent verification of any information it receives.
To the extent that regulatory authorities allow a rating agency to acknowledge in one jurisdiction a rating issued in another jurisdiction for certain regulatory purposes, S&P reserves
the right to assign, withdraw or suspend such acknowledgement at any time and in its sole discretion. S&P Parties disclaim any duty whatsoever arising out of the assignment,
withdrawal or suspension of an acknowledgment as well as any liability for any damage alleged to have been suffered on account thereof.
S&P keeps certain activities of its business units separate from each other in order to preserve the independence and objectivity of their respective activities. As a result, certain
business units of S&P may have information that is not available to other S&P business units. S&P has established policies and procedures to maintain the confidentiality of certain
non-public information received in connection with each analytical process.
S&P may receive compensation for its ratings and certain analyses, normally from issuers or underwriters of securities or from obligors. S&P reserves the right to disseminate its
opinions and analyses. S&P's public ratings and analyses are made available on its Web sites, www.standardandpoors.com (free of charge), and www.ratingsdirect.com and
www.globalcreditportal.com (subscription), and may be distributed through other means, including via S&P publications and third-party redistributors. Additional information about our
ratings fees is available at www.standardandpoors.com/usratingsfees.
Australia
Standard & Poor's (Australia) Pty. Ltd. holds Australian financial services license number 337565 under the Corporations Act 2001. Standard & Poor’s credit ratings and related
research are not intended for and must not be distributed to any person in Australia other than a wholesale client (as defined in Chapter 7 of the Corporations Act).
STANDARD & POOR’S, S&P and RATINGSDIRECT are registered trademarks of Standard & Poor’s Financial Services LLC.