1. Sub-Header Text
Welcome
1

2. About ICS
Established in 1987, Integrated
Computer Solutions, Inc. (ICS)
delivers innovative software
solutions with a full suite of
services to accelerate development
of successful next-gen products.
ICS is headquartered outside
Boston in Waltham, Mass. with
offices in California, Canada and
Europe. Currently 160+ people.

3. www.ics.com
Delivering a Full Suite of
Medtech Services
● IEC 62366-UX/UI Design
● Custom Frontend & Backend Software Development
● Development with IEC 62304-Compliant Platform
● Low-code Tools that Convert UX Prototype to
Product
● Medical Device Cybersecurity
● AWS and Azure Cloud Services and Analytics
● ISO 14971-Compliant Hazard Analysis
● Software Verification Testing
● Digital Twins for Training and Simulation
● Complimentary Software Technology Assessments

4. Quality and Test in Medical Devices
1. Part 1 - Recap
2. Part 2 - Test Methods, Pros, Cons
3. The complete picture Quality and Test
Page #

5. Part 1 Recap
1. Usability to systematically reduce Use Errors through product design - impacts quality
2. Architecture for quality - maintainable, testable, extensible, reliable, available
a. Tiered layers - visibility / access
b. Error injection - difficult to test sub-systems
c. Dependency inversion - simulators and visibility
3. Code analyzers
a. A quality tool, with tradeoffs
b. Necessary but not sufficient (for complex projects)
Usability and Architecture for Quality
5

6. Part 2
➢ Test Canon
● Automated GUI testing
● System testing
● Integration testing
● Unit tests
➢ High-Value testing
○ Key goals
○ Example
➢ Verification vs. test canon
➢ Software “Reliability"
Pros and Cons of Testing
6

7. Automated UI
Testing
Page #

8. Using a 3rd party application to automate mouse clicks on a GUI (Graphical User Interface)
Test tools manage scripts and collections and support automation
Good for:
- Stable User Interfaces (not undergoing changes)
- Enduring value - runs ‘for free’
- Functional/system/black-box testing
Automated Graphical
User Interface Testing
8
Lambda Test
Selenium
AutoHotKey
Squish
Test Complete
Cypress

9. Challenges
- Brittle - Superficial changes to UI can cause tests to fail
- Test strategy - create modular/reusable tests that can be daisy-chained
- No variability - tests the same thing over and over
- Mostly tells you something changed
- Changes system - additional process running on system with threading considerations
- i.e. a graphic bug example (triangles)
Automated User Interface Testing
9
Lambda Test
Selenium
AutoHotKey
Squish

10. System Testing
Page #

11. UI vs. System testing → same?
● Depends on architecture and functionality of the UI
● Are all system behaviors available from the UI?
System Testing
● May require additional test software/framework (i.e. LIS, Cloud services)
● Focuses on positive and negative test cases
● Can be harder to automate
Both
Are tested vs. product requirements
System Testing
11

12. Integration Testing
Page #

13. Why integration testing is important
- Behavior obscured by system
complexity
- FDA language indicating that functional
testing (Black box), is not sufficient as it
does not exercise hidden dependencies
- Rigorously test behavior for each block
and each interface
- Essential for large or complex products
Integration Testing
13
Pro
- Essential, irreplaceable form of test for
a complex system
Con
- Labor intensive
- Time-consuming

14. Unit Tests
Page #

15. Unit Tests
Background
- The smallest unit of test
- Software functions that call product functions
- They usually are numerous, multiple test functions for
a single product function. Executed by a framework in
an automated build/test sequence.
- Necessary, but far from adequate
As valuable as we thought?
15

16. Unit Tests
Pros
- Finds bugs early, at check-in
- Easily automated - Part of Continuous Integration
- Automated coverage metrics
- Enduring value
- Mature tool set (Google Test, xUnit, Mocha)
- Gold standard
Cons
- Increases initial development effort
- Finds few bugs
- Static
- Obvious/trivial
- Low value/effort ratio
- Small code changes can take much longer
- False confidence in code quality
Pros / Cons?
16
Commonly seen:
- Low value unit tests that rarely find
a bug
- Unit test code > product source
code - making changes difficult
- High effort to get the last 10%
coverage - with little apparent
return
- But,... still essential

17. Unit Tests
Keep it light
Don’t try for 100% everywhere
Differentiate highly critical vs. non-critical
Pin down the corners vs. exhaustive testing
Don’t:
Add high levels of test if product in flux
Add trivial testing to improve code coverage metrics
Best applied to
Stable products
When losing experienced staff
Sophisticated/complex designs
Automation: Continuous Integration, automated emails
Best practices
Page #
Best practice
Non-trivial tests written for stable
products with low staffing levels in an
automated environment
Worst practice
Simple tests written for 100% code
coverage on products undergoing change
TL;DR

18. High Value Testing
Page #

19. ➢ Analyze software sub-components for criticality
○ Comes from IEC 62304: risk based
analysis for each software
sub-component
○ Adjust your process according to risk
➢ Allows you to discriminate between critical
sub-systems and non-critical sub-systems
➢ Overall safety class = MAX(each functional
aspect)
High Value Testing
19
C B C C B C C A B A C A

20. Key goals
1) Balance test resources around criticality of
component to Intended Use, Safety, Efficacy
2) Minimize development overhead
3) Minimize regulatory burden
Example
- “Offline Data Access” omits unit, integration,
system testing
Pro
- Less budget overall
- More budget on critical areas
Con
- Expertise: More complicated to plan and apply
- Hard to keep track of →Test Plan
High Value Testing
20
A
C B C C B C C A B C A

21. ➢ Test vs. Verification
➢ Reliability
Page #

22. Quality
team
V&V
team
Engineering
Quality Assurance
- Confirm process
- Checking that SDP steps were followed
- Issuing/processing CAPAs
Engineering Test Engineer
- Test automation (scripts, CI integration)
- Ad-hoc testing
- Smoke testing
- Build management
Verification Test Engineer
- Formal test protocol writing
- Formal test protocol execution
- Test coverage
- Requirement focus
Validation Test
- Formal test protocol writing
- Formal test protocol execution
- Clinical focus
- Statistical test treatment
HF/UE Test
Engineer
- Summative test
HF/UE Test
Engineer
- Formative test
Engineering
Vs
Quality
Developer
- Unit tests
- Ad-hoc testing
Quality driven
Plan → Review → Report
Plan
Accounts for engineering tests
Analyze/Review
Determines outcome
Report
Formally documents
Engineering driven
- Development process
- Includes ad-hoc
Engineering Test vs. Verification & Validation

23. ● Software is logical not physical so traditional concepts of failure
don’t apply
○ Mechanical or electrical systems have Bathtub failure curves
● Software Reliability is a nuanced concept different from Software
Quality
○ Reliability is Software Quality over time
○ i.e. How long has the software been meeting it’s requirements
● Reliability
○ Considers Total Inputs vs Total Outputs
○ Tests a sample of Total Inputs
○ Probability = Number failing cases / Total cases
○ Testing types: Feature/Load/Regression test
○
● Takeaway:
○ Reliability is a sample based, predictive metric different from
QA/QC/Test
○ A complete, quantitative, system for assessing software
Software Reliability
23

24. Tying it together:
Part 1 & 2
Quality vs. Test
Page #

25. Quality → Test
What’s the relationship between quality and test?
Quality as an outcome, Test as a method?
Can you test quality into a product?
Usability
Use Errors
Prevent unintended behaviors
Architecture
Separation
Clear responsibilities
Access / Visibility
Prevent unintended functionality
Test canon (unit, integration, system, UI)
Quality/Test relationship
25
Quality Assurance
Quality Control
Testing
Prevent Defects
Correct Defects
Detect Defects

26. Thank You
Page #