1. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Software Reliability
Journ´ee des Technologies de l’Information et de la
Communication
Baptiste Wicht
EIA-FR
June 3, 2014
Baptiste Wicht EIA-FR Software Reliability
5. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Definitions
Examples
Facts
Solutions
Definitions
Definition
Probability that a software will work properly in a specified
environment for a given amount of time.
What makes a program reliable ?
No bug
Must work under any possible condition
Must comply to the specifications
Baptiste Wicht EIA-FR Software Reliability
6. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Definitions
Examples
Facts
Solutions
Difficulties
properly, specified environment, given amount of time
No specification is perfect
Specifications can change
Software is complex
Baptiste Wicht EIA-FR Software Reliability
7. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Definitions
Examples
Facts
Solutions
Measuring reliability
Difficult problem
Hard to specify
Software is complicated to quantify
Models are available
More than 200 models exist
No single model completely represent reliability
Basic idea: Reliability is function of the number of defects
Baptiste Wicht EIA-FR Software Reliability
8. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Definitions
Examples
Facts
Solutions
Examples
1991: Telephone outage in California
Three lines of code changed
several-million lines of code program
⇒ ≈ 200’000 phone calls lost
Baptiste Wicht EIA-FR Software Reliability
9. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Definitions
Examples
Facts
Solutions
Examples
1991: Telephone outage in California
Three lines of code changed
several-million lines of code program
⇒ ≈ 200’000 phone calls lost
1991: Patriot Missile fails to intercept Scud Missile
Time stored in tenths of a second, multiplied by 1/10 to get
seconds
24 bit fixed-point register, 1/10 has 0.000000095 error
After 100 hours, error of 0.34 seconds
=> 28 dead
Baptiste Wicht EIA-FR Software Reliability
10. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Definitions
Examples
Facts
Solutions
Examples (cont.d)
1996: Ariane 5 explodes after 37 seconds of flight
16bit integer overflow
Overflow detected ⇒ useless corrections ⇒ other
malfunctions ⇒ self-destruction
System taken from Ariane 4, not used in Ariane 5
⇒ 500M $ lost
Baptiste Wicht EIA-FR Software Reliability
11. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Definitions
Examples
Facts
Solutions
Examples (cont.d)
1996: Ariane 5 explodes after 37 seconds of flight
16bit integer overflow
Overflow detected ⇒ useless corrections ⇒ other
malfunctions ⇒ self-destruction
System taken from Ariane 4, not used in Ariane 5
⇒ 500M $ lost
2014: Heartbleed (OpenSSL)
Missing bound check
Allows more data to be read than should be allowed
Introduced in 2011
⇒ Millions of website vulnerable
Baptiste Wicht EIA-FR Software Reliability
12. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Definitions
Examples
Facts
Solutions
Facts
Perfect software does not exist
The bigger the program ⇒ the more bugs
working today = working tomorrow
small change may have big impact
Baptiste Wicht EIA-FR Software Reliability
15. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Unit Testing
Integration Testing
Software Validation
Conclusion
Unit Testing
Definition
Set of tests to verify that a small unit of code is correct
Automated tests
Should be run after each change to the program and under
production-like environment
Monitor code coverage
Baptiste Wicht EIA-FR Software Reliability
16. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Unit Testing
Integration Testing
Software Validation
Conclusion
Integration Testing
Definition
Set of tests to verify that a set of units forming a component is
correct
Can be automated or performed by human
If human-performed, can be very costly
Should be performed as much as possible
Baptiste Wicht EIA-FR Software Reliability
17. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Unit Testing
Integration Testing
Software Validation
Conclusion
Software Validation
Definition
Tests to ensure that the complete system meets its specifications
Was the right software built ?
Was the software built right ?
Generally performed by Q/A specialists
Baptiste Wicht EIA-FR Software Reliability
18. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Unit Testing
Integration Testing
Software Validation
Conclusion
Conclusion
No part of the code should be neglected
No change should be neglected
Small modules easily covered by a unit test should always be
unit tested
Unit/Integration tests should always be used to avoid
regression
Unit testing should be done as part of Continuous Integration
Automated tests are less costly that human-performed tests
Baptiste Wicht EIA-FR Software Reliability
20. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Language Features
Contract Programming
Conclusion
Defensive Programming
Definition
Make a program more reliable by ensuring that code is used in the
correct way and it does what it is supposed to do.
Protection against failures and fault
Written as code
Accurate information to the source of the error
Verified at compile-time or runtime
Baptiste Wicht EIA-FR Software Reliability
21. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Language Features
Contract Programming
Conclusion
Language Features
Different languages provide different level of security
int array [100];
int b = array [100]; // What should happen ?
Baptiste Wicht EIA-FR Software Reliability
22. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Language Features
Contract Programming
Conclusion
Language Features
Different languages provide different level of security
int array [100];
int b = array [100]; // What should happen ?
C/C++: Undefined
Java/Python: Runtime error
Generally: lower level ⇒ harder to debug
Baptiste Wicht EIA-FR Software Reliability
23. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Language Features
Contract Programming
Conclusion
Assertions
Definition
Predicates (true-false statements) inserted into code indicating
what the developer thinks should be true at a certain point.
Indicates that a certain condition must be true
Typically aborts execution if condition not met
May be replaced by exceptions
Is generally disabled in production
Should not have any side effect
Baptiste Wicht EIA-FR Software Reliability
25. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Language Features
Contract Programming
Conclusion
Assertions (cont.d)
Pros
Available in almost every languages
Very simple
Generally useful to check input parameters
Baptiste Wicht EIA-FR Software Reliability
26. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Language Features
Contract Programming
Conclusion
Assertions (cont.d)
Pros
Available in almost every languages
Very simple
Generally useful to check input parameters
Cons
Only verified at runtime
Some languages supports compile-time assertions
Only local to a function
Can make the code more complex
Can be disabled
Baptiste Wicht EIA-FR Software Reliability
27. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Language Features
Contract Programming
Conclusion
Contract Programming
Contract of the code specified into the program
Pre/postconditions, Invariants, ...
Can be implemented with assertions
Coupled with powerful static analysis, some errors can be
found without running the program
Baptiste Wicht EIA-FR Software Reliability
28. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Language Features
Contract Programming
Conclusion
Conclusion
Pros
Catch errors with nice error messages
Stored with the code
Cons
Make the code more complex
May introduce new errors
May incur a runtime cost
Dependent on the developer
Baptiste Wicht EIA-FR Software Reliability
31. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Compilers
Every modern compiler produces warnings
Only when asked
May produce too much warnings
Often forgotten
Recommendation
New project: As much as possible and -Werror
Existing project: Fix existing warnings and enable warnings one
by one
CLang/GCC: -Wall -Wextra
Baptiste Wicht EIA-FR Software Reliability
32. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
cppcheck
Static Analysis for C/C++ programs
Free software, GPL license
Cross-platform
Active project, latest stable release 10 May 2014
Available on every mainline distribution
Contains a large set of checkers
Integrated in several IDE (Eclipse, Visual Studio,
Code::Blocks, CodeLite...)
Generate text, XML and HTML reports
Baptiste Wicht EIA-FR Software Reliability
33. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
CLang Static Analyzer
Static Analysis for C/C++ and Objective-C programs
Free software, MIT license
Developed as part of the LLVM/CLang project
Slower than compilation
Still young project
Several checkers implemented
Can be extended with checker plugins
Generate text and HTML reports
Baptiste Wicht EIA-FR Software Reliability
34. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Valgrind
Framework for dynamic analysis tools
Free Software, GPL License
Work on Linux and Mac OS X
Most known for Memcheck tool
Runs the program into a virtual machine
Program runs slower (4-5 times slower with None tool)
Generate text reports
Baptiste Wicht EIA-FR Software Reliability
35. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Tools
Memcheck: Memory debugger
Default and most used tool
Very large performance lost (20 to 40 times slower)
Massif: Heap profiler
Helgrind: Detect race conditions
Cachegrind: Cache profiler
Callgrind: Call-graph analyzer
exp-sgcheck: Experimental tool for memory overruns
Baptiste Wicht EIA-FR Software Reliability
37. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Memory Leak
int main (){
int* a = new int (4);
// Something
a = new int (5);
// Something else
delete a;
}
> g++ leak.cpp
> valgrind ./a.out
Baptiste Wicht EIA-FR Software Reliability
38. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Memory Leak
==15386== HEAP SUMMARY:
==15386== in use at exit: 4 bytes in 1 blocks
==15386== total heap usage: 2 allocs, 1 frees, 8 bytes allocated
==15386==
==15386== LEAK SUMMARY:
==15386== definitely lost: 4 bytes in 1 blocks
==15386== indirectly lost: 0 bytes in 0 blocks
==15386== possibly lost: 0 bytes in 0 blocks
==15386== still reachable: 0 bytes in 0 blocks
==15386== suppressed: 0 bytes in 0 block
4 bytes have been lost
No information on location
> valgrind --leak -check=full ./a.out
Baptiste Wicht EIA-FR Software Reliability
39. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Memory Leak
==6434== 4 bytes in 1 blocks are definitely lost in loss record 1 of 1
==6434== at 0x4028812: operator new(unsigned long) (vg_replace_malloc.c:319)
==6434== by 0x400641: main (in /home/wichtounet/dev/analysis-examples/a.out)
The source of the allocation is now shown
Only binary location, no source location
> g++ -g leak.cpp
> valgrind --leak -check=full ./a.out
Baptiste Wicht EIA-FR Software Reliability
40. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Memory Leak
==9604== 4 bytes in 1 blocks are definitely lost in loss record 1 of 1
==9604== at 0x4028812: operator new(unsigned long) (vg_replace_malloc.c:319)
==9604== by 0x400641: main (leak.cpp:2)
Source location (leak.cpp:2) is now complete
Valgrind will only show the sources of allocated leaked memory
Sometimes it may still be complicated to find why not freed
Baptiste Wicht EIA-FR Software Reliability
47. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Stack Overrun 2
==28007== exp-sgcheck, a stack and global array overrun detector
==28007== NOTE: This is an Experimental-Class Valgrind Tool
==28007== Copyright (C) 2003-2013, and GNU GPL’d, by OpenWorks Ltd et al.
==28007== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==28007== Command: ./a.out
==28007==
==28007==
==28007== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0
Overrun not detected!
Need a valid access before the invalid one
> cppcheck stack_overrun_2 .cpp
Baptiste Wicht EIA-FR Software Reliability
48. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Stack Overrun 2
Checking stack_overrun_2.cpp...
[stack_overrun_2.cpp:4]: (error) Array ’stack[5]’ accessed at index 5, which is out of bounds.
Overrun detected!
Tools put together are very powerful
Baptiste Wicht EIA-FR Software Reliability
49. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Complex Memory Leak
int main (){
int* array [42];
for(int i = 0; i < 42; ++i){
array[i] = new int;
}
// Something
for(int i = 0; i < 41; ++i){
delete array[i];
}
}
> cppcheck --enable=all leak_2.cpp
Baptiste Wicht EIA-FR Software Reliability
50. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Compilers
Tools
Exercises
SonarQube
Conclusion
Complex Memory Leak
Checking leak_2.cpp...
Checking usage of global functions..
cppcheck static analysis not powerful enough
> g++ -g leak_2.cpp
> valgrind --leak -check=full ./a.out
==12868== 4 bytes in 1 blocks are definitely lost in loss record 1 of 1
==12868== at 0x4C2C167: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-li
==12868== by 0x400739: main (leak_2.cpp:7)
Baptiste Wicht EIA-FR Software Reliability
59. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Conclusion
Build reliable software is hard
Software testing is very important
Numerous tools are available
No tool will ever find all bugs
Some tools are overlapping
Baptiste Wicht EIA-FR Software Reliability
60. Software Reliability
Software Validation
Defensive Programming
Software Analysis Tools
Conclusion
Recommendations
No change should be ignored when testing
Tests should be run in production-like environment
Continuous Integration and Testing
Make use of the existing analysis features at each level
No tool is perfect ⇒ Several tools can be used together
Baptiste Wicht EIA-FR Software Reliability