How to be sure that a software development and test tool works according to a dedicated specification.

1. Tool Qualification
Minimize the risk
of systematic faults in the developed product
due to malfunctions of the software tool
(introduce or fail to detect errors)
Debug and Test Tools

2. ISO 26262
Part 8 Supported Processes, Chapter 11
 Safety-Lifecycle Process
― Big portions of this procoss is covered by software tools today
 This works only if
― The risk of systematic fault in the developed product due to
malfunctions of the software tool (introduce or fail to detect errors) is
minimized
― The software tool works accurate and correct within such a process
 Detailed and up front analysis / risk assessment of the software tools used
within a safety project
Debug and Test Tools

3. Software Tools
 Software Development Tools: May introduce an error in the final
product
 Software Verification Tools: May fail to detect an error in the
final product
 Other Software Tools
― Depending on the size of a company we are talking of a big number of
tools (including in-house tools, Excel, …)
Debug and Test Tools

4. Part 1 – Overview
The Blue Box and ISO26262 Use Cases
Debug and Test Tools

5. Reference V ISO 26262-6
ISO 26262-5, 10
Hardware
Debug and Test Tools

6. iSYSTEM Tools within a Safety Project I
 Software Development (ISO 26262-6, 9)
― Not so critical
― Assembler (interrupt handler, time-critical algorithms, …)
 Software Test (ISO 26262-6, 9)
― Unit Testing
― Integration Testing
 Hardware-Software Integration and Testing (ISO 26262-5, 10/-4,8)
 System Integration Testing (ISO 26262-4,8)
 ISO 26262-6 mentions: „…The test environment … shall correspond as
closely as possible to the target environment …“
Debug and Test Tools

7. iSYSTEM Tools within a Safety Project II
- Unit Testing
isystem.connect
isystem.test
Debug and Test Tools

8. iSYSTEM Tools within a Safety Project II
- Unit Testing
isystem.connect
isystem.test
+
Trace &
Object Code
Level Coverage
Debug and Test Tools

9. iSYSTEM Tools within a Safety Project II
- Unit Testing
isystem.connect
isystem.test
Debug and Test Tools

10. iSYSTEM Tools within a Safety Project II
- Hardware-Software Integration Testing
isystem.connect
isystem.test
+ Trace & Profiler
isystem.connect
isystem.test
+ Trace &
I/O Module
Debug and Test Tools

11. Part 2 – How to get confidence
in the use of software tools
 Detailed and up front analysis / risk assessment of the software tools
used within a safety project
Debug and Test Tools

12. Debug and Test Tools

13. e?
Confidence in the use of e nc
id
software tools ISO 26262-8-11
on
f
c
and
 sk
How high is the risk that the malfunction of a tool leads to an error in the
ri
ss
final product or errors are not detected (Tool Impact, TI)?
se
as

ou
How much do you trust your own methods and procedures (your
y
processes) in order to uncover suchhmalfunction of a tool
(Tool Error Detection (TD)? hic
w
on
high se average low
ba Trust in process
minimal risk
ti on
high risk a
o rm
f
in
e
th
 Beside the tool itself also the process plays a major role in getting
is
confidence
t
a
h TCL2, TCL3: Tool Qualification applies
W
Debug and Test Tools

14. How iSYSTEM helps you
getting/creating confidence?
 Process
― SPICE, CMMI, … iSYSTEM: ISO9001:2008
― Output: Process documentation, external audits
 Transparency es
ur
― Release and test process (http://www.isystem.com/downloads/winidea/release-policy)
eas new features), item (bugs, changes, new
― Output: Test documentation, release notes (major
features, …) tracking lists l M
o na
 Documentation a ti
is be used in a safety project or to be qualified)
― Description of features/functions (to
an
rg
― Explanation how these features/functions should (not) be used
O
― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing
Guidelines, ...
― System Requirements in general or specific to a micro controller
― Description of well-known workarounds dedicated to a specific micro controller
― …
 Pre-Qualification Environment: fitIDEA
― Regression test suite, reference hardware, reference application, reference test cases
Debug and Test Tools

15. Result of Step 1
Determine a „Tool Confidence Level“ could be
l?
minimal risk, high trust = t oo
e
no tool qualificationr needed
w
a
ft
so
a
y
lif
or you see a high risk and an average trust in
your processes andqua a tool qualification would
to apply
w
Ho
high average low
Trust in process
minimal risk
high risk
Debug and Test Tools

16. ISO 26262-8-11 Tool Qualification Methods
TCL2 A B C D A-D: Measure of the
Methods safety relevance of a
1a Increased confidence from use ++ ++ ++ + malfunction
(Automotive Safety
1b Evaluation of the tool dev process ++ ++ ++ +
Integrity Level (ASIL))
1c Validation of the software tool + + + ++
1d Dev. in accordance with a safety standard + + + ++
TCL3 A B C D
1a Increased confidence from use ++ ++ + +
1b Evaluation of the tool dev process ++ ++ + +
1c Validation of the software tool + + ++ ++
1d Dev. in accordance with a safety standard + + ++ ++
+ recommended
++ highly recommended
 Qualifying a software tool means checking/discussing all 4 methods
Debug and Test Tools

17. What are we talking about?
 The company iSYSTEM
 Development and test tools
 Standard debugging and IDE functions (memory read, write,
step, memory dump, download, flash programming, etc.)
 Advanced debugging with trace and profiling (especially time
measurements)
 Software test with code coverage and unit test
Debug and Test Tools

18. TCL2 A B C D
1a Increased confidence from use ++ ++ ++ +
1b Evaluation of the tool dev process ++ ++ ++ +
1c Validation of the software tool + + + ++
1d Dev. in accordance with a safety standard + + + ++
TCL3 A B C D
1a Increased confidence from use ++ ++ + +
1b Evaluation of the tool dev process ++ ++ + +
1c Validation of the software tool + + ++ ++
1d Dev. in accordance with a safety standard + + ++ ++
Debug and Test Tools

19. Increased Confidence From Use?
 How is a software tool used today?
― Dedicated version and configuration
ll ?
ta
― For one or more specific micro controllers
a
― In conjunction with other software tools (e.g., compilers)
hod
et
 New project: m
s
― The tool setup may definitely differ fromi the previous project
th
― Very new controllers will be used er
id
ns
 co
Dynamic release policy of the software tool manufacturer
e
w
― Official and regression tested release
d
ul
ho
― Hotfixes (customer wishes)
S
― Fairly good tested software versions
Debug and Test Tools

20. YES! – Because …
 In the market since 1986
 Thousends of users, word-wide
• Similar projects successfully done in the past
 Great cooperation between you and iSYSTEM
― Transparency ISO 262
62
― Openness
― Pragmatism
Debug and Test Tools

21. How iSYSTEM helps you
getting/creating confidence?
 Process
― SPICE, CMMI, … iSYSTEM: ISO9001:2008
!
EM
― Output: Process documentation, external audits
ST
 Transparency
i SY
=
― Release and test process (http://www.isystem.com/downloads/winidea/release-policy)
― Output: Test documentation, release notes se
(major new features), item (bugs, changes, new
features, …) tracking lists u
in
ce
 Documentation en
fid
― Description of features/functions (to be used in a safety project or to be qualified)
on
― Explanation how these features/functions should (not) be used
c
d
― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing
se
ea in general or specific to a micro controller
Guidelines, ...
cr
― System Requirements
n
― DescriptionIof well-known workarounds dedicated to a specific micro controller
― …
 Pre-Qualification Environment: fitIDEA
― Regression test suite, reference hardware, reference application, reference test cases
Debug and Test Tools

22. TCL2 A B C D
1a Increased confidence from use ++ ++ ++ +
1b Evaluation of the tool dev process ++ ++ ++ +
1c Validation of the software tool + + + ++
1d Dev. in accordance with a safety standard + + + ++
TCL3 A B C D
1a Increased confidence from use ++ ++ + +
1b Evaluation of the tool dev process ++ ++ + +
1c Validation of the software tool + + ++ ++
1d Dev. in accordance with a safety standard + + ++ ++
Debug and Test Tools

23. Evaluation of the tool dev process
 Most of the software tool manufacturers do have defined and documented
their processes (and lives those, hopefully), ISO9001 or similar
 We guess that non of them have a real process model in place (such as
CMMI, SPICE, …)
 Audit/Assessment of these processes is possible (quality manual, process
descriptions, external audit, …)
 Zertificates?!
 It may be a big challenge to evaluate a dev process, again and again
 Too many software tools are already in use, also Excel and so forth
Debug and Test Tools

24. How iSYSTEM helps you
getting/creating confidence?!
s
se
 Process es
― SPICE, CMMI, … iSYSTEM: ISO9001:2008 oc
pr
st
― Output: Process documentation, external audits
te
 Transparency and
ev
― Release and test process (http://www.isystem.com/downloads/winidea/release-policy)
― Output: Test documentation, release notes d
s (major new features), item (bugs, changes, new
features, …) tracking lists M
TE
YS
 Documentation iS
e
at
― Description of features/functions (to be used in a safety project or to be qualified)
u
― Explanation how these lfeatures/functions should (not) be used
a
ev
― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing
Guidelines, ... to
e
bl
― System Requirements in general or specific to a micro controller
si
os
― Description of well-known workarounds dedicated to a specific micro controller
― … p
t is
I
 Pre-Qualification Environment: fitIDEA
― Regression test suite, reference hardware, reference application, reference test cases
Debug and Test Tools

25. TCL2 A B C D
1a Increased confidence from use ++ ++ ++ +
1b Evaluation of the tool dev process ++ ++ ++ +
1c Validation of the software tool + + + ++
1d Dev. in accordance with a safety standard + + + ++
TCL3 A B C D
1a Increased confidence from use ++ ++ + +
1b Evaluation of the tool dev process ++ ++ + +
1c Validation of the software tool + + ++ ++
1d Dev. in accordance with a safety standard + + ++ ++
Debug and Test Tools

26. Validation of the software tool
 It is also the responsibility of the tool manufacturer to provide means and
methods to enable a customer to perform a validation
 Ideal: Availability of the manufacturer’s test tool suite for a customer.
Documentation, Test suite with reference hardware, reference applications
and appropriate test cases.
isystem.connect for Python wrapper
isystem.connect test
testIDEA
isystem.connect
iSYSTEM Development Environment & Debugger winIDEA
Reference Target
Reference Application
Debug and Test Tools

27. How iSYSTEM helps you
getting/creating confidence?
e!
 Process
― SPICE, CMMI, … iSYSTEM: ISO9001:2008
os
― Output: Process documentation, external audits rp
pu
ct
 Transparency
exa
― Release and test process (http://www.isystem.com/downloads/winidea/release-policy)
s
hi
― Output: Test documentation, release notes t(major new features), item (bugs, changes, new
features, …) tracking lists
f or
A
 DE
itI (to be used in a safety project or to be qualified)
Documentation
f
― Description of features/functions
s
de
― Explanation how these features/functions should (not) be used
vi
― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing
o
Guidelines, ... pr
EM
― System Requirements in general or specific to a micro controller
― Description T well-known workarounds dedicated to a specific micro controller
Sof
― …
i SY
 Pre-Qualification Environment: fitIDEA
― Regression test suite, reference hardware, reference application, reference test cases
Debug and Test Tools

28. fitIDEA - Tool Pre-Qualification Environment
Test GUI
Test Cases
Consulting &
Test Reports
Integration
Safety Manual &
Services
Documentation
isystem.connect for Python wrapper
isystem.connect test
testIDEA
isystem.connect
iSYSTEM Development Environment & Debugger winIDEA
Reference Target
Reference Application
Debug and Test Tools

29. Tool Pre-Qualification Environment
• If a specific function of an iSYSTEM tool has to get verified in the
customer’s use case context
• Consists of reference hardware plus test cases to verify several functions
of an on-chip debug and trace tool such as:
― Standard debugging and IDE functions, e.g. memory read, write, step, memory
dump, download, flash programming, etc.
― Advanced debugging with trace and profiling (especially time measurements)
― Software test with code coverage and unit test
―…
• Is an extract of iSYSTEM’s internally used regression test tool suite
• May be ported/adapted to a customer’s target system
Debug and Test Tools

30. TCL2 A B C D
1a Increased confidence from use ++ ++ ++ +
1b Evaluation of the tool dev process ++ ++ ++ +
1c Validation of the software tool + + + ++
1d Dev. in accordance with a safety standard + + + ++
TCL3 A B C D
1a Increased confidence from use ++ ++ + +
1b Evaluation of the tool dev process ++ ++ + +
1c Validation of the software tool + + ++ ++
1d Dev. in accordance with a safety standard + + ++ ++
Debug and Test Tools

31. Development in accordance with a
saftey standard
 View and adapt
recommendations of
such standards
 Transfer know-how to
software developers
(how others have to do
it)
Debug and Test Tools

32. How iSYSTEM helps you
getting/creating confidence?
 Process
― SPICE, CMMI, … iSYSTEM: ISO9001:2008
― Output: Process documentation, external audits
 Transparency n!
ar
le
― Release and test process (http://www.isystem.com/downloads/winidea/release-policy)
to
― Output: Test documentation, release notes (major new features), item (bugs, changes,
new features, …) tracking lists g n
i
i ll
w
 Documentation
is
EM
― Description of features/functions (to be used in a safety project or to be qualified)
T
― Explanation how these features/functions should (not) be used
SHardware Reference Manual, Getting Started Guides, PCB
― Manuals: User Manual,Y
iS
Desing Guidelines, ...
― System Requirements in general or specific to a micro controller
― Description of well-known workarounds dedicated to a specific micro controller
― …
 Pre-Qualification Environment: fitIDEA
― Regression test suite, reference hardware, reference application, reference test cases
Debug and Test Tools

33. 62
ISO 262
Summary
 Detailed and up front analysis / risk assessment
of the software tools used within a safety project
 Beside the tool itself also the process plays a
major role in getting confidence
 It is also the responsibility of the tool manufacturer to provide means and
methods to enable a customer to perform a validation
 Tool manufacturers will improve their development and test processes
 Transparency first, qualification second
Debug and Test Tools