The document discusses testing strategies for the ACE open source project. It describes ACE and its testing challenges due to its large codebase and distributed development. It then outlines a 4-step policy developed by ACE to prioritize testing, focusing on core components and called code. This policy reduced untested functions from 275,000 to under 3,000. The document also introduces Coverity Test Advisor and how it was used by ACE to define and enforce this testing policy. Finally, it briefly describes Coverity's development testing solutions and maturity model.

1. Test What Matters Most
Johnny Willemsen, Remedy IT
Keri Sprinkle, Coverity
Jon Jarboe, Coverity

2. Agenda
2 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
• Part 1: Remedy IT and ACE
• Introduction
• ACE and Coverity Scan
• Testing challenges
• Creating an ENFORCEABLE testing policy
• 4 Step policy with a big impact
• Part 2: Coverity
• Overview of Coverity Test Advisor
• Coverity suite of development testing solutions
• Development Testing Maturity Model
• Q & A

3. Remedy IT
ACE
3For Coverity and Partner use only. Copyright Coverity, Inc., 2013

4. Remedy IT
4 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
• Company of specialists focused on middleware and
component technologies
• Dedicated to open standards and
preferable open source
• Actively involved in the development of
ACE/TAO/CIAO/DAnCE/OpenDDS
open source software suite

5. What is ACE?
5 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
ADAPTIVE Communication Environment
• Open-source object-oriented (OO) framework that
implements many core patterns for concurrent
communication software
• Strong adoption in the telecom, medical, aerospace,
defense and financial services industries

6. ACE and Coverity Scan
6 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
• What is Coverity Scan?
• Free static analysis for open source projects
• Find and fix defects in C/C++ or Java
• Tests every line of code and potential execution path
• Explains root cause of each defect making it easy to fix bugs
• ACE and Coverity Scan
• Joined in April 2011
• Fixed 3243 defects as of June 2013
• 1827 defects outstanding
• Defect density of 0.24

7. ACE testing challenges
7 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Goal: reduce risk through actionable, efficient testing
• Large, decentralized project with distributed development teams
• 7.8 million lines of C++ code
• 20 primary developers, 100 developers committing code to the repository,
about 2375 developers who have contributed additions/patches
• Last release on May 28, 2013
• Testing a priority, but not enforceable, so most code remains untested
• Current overall code coverage is 27% (57% for critical components)
• Over 275,000 functions not fully tested including examples and other code
that does not require testing

8. Creating an ENFORCEABLE test policy
8 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
• Many concerns before the project began
• Where do I start, and how do I prioritize, with so much untested code?
• How much real risk is there that I need to address ASAP?
• How do I hold developers accountable for testing their own code?
• How do I ensure code is tested as new features are developed?
• ACE developed a basic Coverity Test Advisor policy to
address these concerns
• Filtering rules to determine which code is important to test
• Analyzing untested code against those rules to identify missing tests
and prioritize them

9. 4 Step policy with a big impact
9 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Step Remaining
Violations
Testing Policy
0 275,313 Simple line coverage
All insufficiently‐ or un‐tested functions
1 12,833 Focus on core components
Premise: certain components are more important to test
than others
2 2,967 Only called code (ignore uncalled)
Premise: we only write tests for important functions
3 2,757 Exclude debug, logging code
Premise: diagnostic messages are not important to test
4 2,588
Since 6.1: 605
Exclude error cases, prioritize violations by release
Premise: testing error cases provides minimal value and it is
often difficult to trigger this code
Note: These numbers are for the exact same code

10. Using Coverity
10 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
• ACE performs a weekly build dedicated to Coverity scan
• Coverity scan analyzes the code and publishes the Scan
results
• New issues are emailed to all developers
• The ACE build czar will enforce that any new issue gets
resolved by the original developer
• Supports our goal that quality increases weekly and never
decreases

11. Addressing the issues
11 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
• Test violations in the core code are most important
• Try to resolve high priority violations with funding through
maintenance and support
• When resolving bugs, test violations are used to focus the
new unit tests on what is really needed
• When adding a feature to ACE, Test Advisor is used to
determine that the feature is well tested
• After a decrease in number of test violations, the basic Test
Advisor policy can be relaxed

12. Coverity
12For Coverity and Partner use only. Copyright Coverity, Inc., 2013

13. Automated Testing Is Not Effectively
Measured
13 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Effort to Develop Tests
%ofCodeTest
1. Diminishing return for
increased test effort
2. Not all code is testable:
• Unreachable statements
• Dead code
3. Not all tested code adds
equal value:
• Non critical code
• Debug code
• Exception handling code
100%

14. Coverity Test Advisor
14 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Improves unit testing effectiveness and efficiency
Focus testing
time where
it matters
And
don’t waste
time writing tests
you don’t need
High
Risk
Code

15. Creating an enforceable process
The high priority code that must
be thoroughly tested
Define
Analyze
Remediate
Govern
Code that has changed
and been impacted by changes
Code that has been insufficiently
tested
Manage progress to improve test
coverage
15

16. What code needs to be tested based on
your high risk criteriaDefine
Analyze
Remediate
Govern
With patent-pending techniques based
on code behavior and change impact
Surface issues in your workflow and
efficiently manage to closure
Create a testing stage gate and enforce
developer accountability
Generic example:
• All code changes for next release …
• And code impacted by those changes …
• Must have 100% coverage …
• Not counting exception handling and debug
code
Policy definition example
16 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
ACE:
• Focus on core components
• Only called code (ignore uncalled)
• Exclude debug, logging code
• Exclude error cases, prioritize violations by
release
• Exclude error code
• Prioritize violations by release

17. Remediate Testing Violations
Unified view
of test
violations and
defects
Unified view
of test
violations and
defects
Clear
description of
the violation
Clear
description of
the violation
Automatically
assign
violations to
owners
Automatically
assign
violations to
owners
Identify where
the violation
occurs in the
code
Identify where
the violation
occurs in the
code
For Coverity and Partner use only. Copyright Coverity, Inc., 2013
17

18. Coverity Connect
Coverity SAVE™
Static Analysis Verification Engine
Test
Advisor
SDLC
Integrations
Test
Execution
Third Party
Metrics
Code
Coverage
SCM
Coverity Development Testing Platform
18 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Security
Advisor
Test
Advisor
Analysis Packs
Coverity SAVE™
Static Analysis Verification Engine
Policy Manager
Quality
Advisor
Architecture
Analysis
Dynamic
Analysis
Analysis
Integration
FindBugs™ | FxCop
Analysis
Integration
Toolkit
Coverity Connect
Test
Execution
Third Party
Metrics
Build/
Continuous
Integration
ALM
HP | IBM
IDE
Code
Coverage
Defect
Tracking
SCM
Proprietary Code | Open Source Code
(Scan)

19. Development Testing Maturity Model
19 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Development Testing Adoption
IntegrationintoSDLC
Level 1
Detection of
critical quality and
security defects as
part of SW build
process.
No new defects
introduced.
Level 3
Integration into
the existing SDLC
using a common
workflow for all
defects and test
effectiveness
issues.
Level 4
Establish and
enforce consistent
source code
quality and
security policies.
Establish source
code acceptance
criteria.
Level 5
All legacy defects
eliminated, build
fails if new defects
are introduced.
All critical code
and code impacted
by change is
tested.
High
High
Level 2
Identification of
areas of risk
caused by
insufficient
automated testing.
Ensure critical
code is prioritized
and tested.

20. Q & A
20 For Coverity and Partner use only. Copyright Coverity, Inc., 2012
• Remedy IT:
www.remedy.nl
• ACE:
www.cs.wustl.edu/~schmidt/ACE.html
• Coverity:
coverity.com
• Coverity Scan:
scan.coverity.com