Presentation that gives an overview of how ACE/TAO/CIAO/DAnCE are using Coverity scan

1. Test What Matters Most
Develop Testing:
Totaalvoetbal for Software Development
12th September, Amsterdam
Johnny Willemsen, CTO Remedy IT
jwillemsen@remedy.nl

2. Remedy IT
Company of specialists focused on middleware and
component technologies
Dedicated to open standards and preferable open
source
Actively involved in the development of
ACE/TAO/CIAO/DAnCE/OpenDDS open source
software suite
For more information take a look at our website
www.remedy.nl
2 Copyright © Remedy IT

3. What is ACE?
ADAPTIVE Communication Environment
Open-source object-oriented (OO) framework that
implements many core patterns for concurrent
distributed communication software
Strong adoption in the telecom, medical, aerospace,
defense and financial services industries
TAO, CIAO, DAnCE, and OpenDDS are products
that are created on top of ACE and form together the
ATCDO open source suite.
3 Copyright © Remedy IT

4. ACE and Coverity Scan
What is Coverity Scan?
• Free static analysis for open source projects
• Find and fix defects in C/C++ or Java
• Tests every line of code and potential execution path
• Explains root cause of each defect making it easy to
fix bugs
ACE and Coverity Scan
• Joined in April 2011
• Fixed 3263 defects as of September 2013
• 1795 defects outstanding
• Defect density of 0.24
4 Copyright © Remedy IT

5. ACE testing challenges
Goal: reduce risk through actionable, efficient testing
Large, decentralized project with distributed
development teams:
• 7.8 million lines of C++ code
• 20 primary developers, 100 developers committing
code to the repository, 2375 developers contributed
Testing a priority, but not enforceable, so most code
remains untested
• Current overall code coverage is 27% (57% for
critical components)
• Over 275,000 functions not fully tested including
examples and other code that does not require
testing
5 Copyright © Remedy IT

6. Creating an ENFORCEABLE
test policy
Many concerns before the project began
• Where do I start, and how do I prioritize, with so
much untested code?
• How much real risk is there that I need to address
ASAP?
• How do I hold developers accountable for testing
their own code?
• How do I ensure code is tested as new features are
developed?
6 Copyright © Remedy IT

7. Creating an ENFORCEABLE
test policy
ACE developed a basic Coverity Test Advisor policy
to address these concerns
• Filtering rules to determine which code is important
to test
• Analyzing untested code against those rules to
identify missing tests and prioritize them
7 Copyright © Remedy IT

8. 4 Step policy with a big impact
8
Step Remaining
Violations
Testing Policy
0 275,313 Simple line coverage
All insufficiently- or un-tested functions
1 12,833 Focus on core components
Premise: certain components are more important to test
than others
2 2,967 Only called code (ignore uncalled)
Premise: we only write tests for important functions
3 2,757 Exclude debug, logging code
Premise: diagnostic messages are not important to test
4 2,588
Since 6.1: 605
Exclude error cases, prioritize violations by release
Premise: testing error cases provides minimal value and it is
often difficult to trigger this code
Note: These numbers are for the exact same code Copyright © Remedy IT

9. Using Coverity
ACE performs a weekly build dedicated to Coverity
scan
• Takes 27 hours to compile all source code with gcc
• Takes 19 hours to analyze using Coverity scan
Coverity scan analyzes the code and publishes the
Scan results
New issues are emailed to all developers
The ACE build czar will enforce that any new issue
gets resolved by the original developer
Supports our goal that quality increases weekly and
never decreases
9 Copyright © Remedy IT

10. Addressing the issues
Test violations in the core code are most important
Try to resolve high priority violations with funding
through maintenance and support
When resolving bugs, test violations are used to
focus the new unit tests on what is really needed
When adding a feature to ACE, Test Advisor is used
to determine that the feature is well tested
After a decrease in number of test violations, the
basic Test Advisor policy can be relaxed
10 Copyright © Remedy IT

11. Want to know more?
Remedy IT
Postbus 81
6930 AB Westervoort
The Netherlands
tel.: +31(0)88 053 0000
e-mail: sales@remedy.nl
website: www.remedy.nl
Twitter: @RemedyIT
Slideshare: RemedyIT
Subscribe to our mailing list
Join Coverity Scan
11 Copyright © Remedy IT