Presentation on the promises and pitfalls of applying Agile in a Quality Management System. How do you get the benefits of agile while maintaining quality and regulatory compliance?
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Agile for Software as a Medical Device
1. O R T H O G O N A L
orthogonal.io
A G I L E I N A Q M S
Prepared for Bayer, October 2016
2. ABOUT ME
2 Agile in a QMS orthogonal.io
Bernhard Kappe
CHIEF EXECUTIVE OFFICER, ORTHOGONAL
Bernhard is an expert in lean innovation and the
application of agile methods in regulated
industries. He is the author of the eBook Agile in
an FDA Regulated Environment and a frequent
conference speaker on wireless medical devices
and mHealth. He works with Orthogonal clients
on connected care strategy and at the application
of lean and agile methods.
5. PRESENTATION AGENDA
orthogonal.ioAgile in a QMS5
Introduction
Agile in 5 Minutes
Agile for Medical Devices
How to be Agile in a QMS
Automation
How to be Agile in a QMS (for Non Developers)
Agile Implementation Mistakes
Resources
6. A G I L E I N F I V E
M I N U T E S
A Quick recap of the why and what of Agile
orthogonal.ioAgile in a QMS6
9. ITERATIVE METHODS
Fast Feedback Loops to Reduce Uncertainty and
Efficiently Manage Change:
• OODA Loops
• Shewhart/Deming Cycles (PDCA)
• Six Sigma (DMAIC)
• Toyota Production System
• Lean Manufacturing
• Lean Startup
• Lean UX
orthogonal.ioAgile in a QMS9
10. AGILE
APPROACH
orthogonal.ioAgile in a QMS10
• Break functionality into the smallest units of customer value: the user
story.
• Organize the stories into a prioritized list called a backlog
• Organize development efforts into short (one to two week0 efforts called
iterations or sprints.
• The goal of each iteration: Demonstrate working software
• Take feedback and improve the process through retrospectives and
the software through refactoring.
• Self Organizing Teams
11. BEHAVIOR DRIVEN DEVELOPMENT
(BDD)
orthogonal.ioAgile in a QMS11
“Behavior-driven development is an
“outside-in” methodology. It starts at
the outside by identifying business
outcomes, and then drills down into
the feature set that will achieve those
outcomes. Each feature is captured as
a “story”, which defines the scope of
the feature along with its acceptance
criteria.”
• —Dan North
Title (one line describing the story)
Narrative:
As a [role]
I want [feature]
So that [benefit]
Acceptance Criteria: (presented as Scenarios)
Scenario 1: Title
Given [context]
And [some more context]...
When [event]
Then [outcome]
And [another outcome]...
Scenario 2: ...
12. AGILE ITERATIONS
12
• Story Baking – Detailed specification of user stories, including acceptance criteria and ties to FURPS+.
• Story Points – Unit of estimation measuring complexity and size.
• Iteration Planning Meeting – a negotiation between the team and the product owner about what the team will do during the next
Iteration. The product owner and all team members agree on a set of Iteration goals, which are used to determine which product
backlog items to commit from the uncommitted backlog to the Iteration. Often new backlog items are defined during the meeting.
• Daily Standup Meeting - a 10-minute daily meeting for the team
• Pair Programming – two programmers work together on one user story.
• Test Driven Development (TDD) – repetition of a very short development cycle: first the developer writes a failing automated test case
that defines a desired improvement or new function, then produces code to pass that test and finally refactors the new code to
acceptable standards.
• Continuous Integration – code is checked into a continuous integration environment/build server, which runs automated tests (unit,
integration, static and dynamic analysis, etc.)
• Demo/Design Review - On a per iteration basis, functional software is reviewed by stakeholders, including source code review
• Retrospective - what went well and what to improve in the next Iteration.
16. A G I L E F O R
M E D I C A L
D E V I C E S
Why we need Agile and why it’s taken so long to adopt it
orthogonal.ioAgile in a QMS16
17. CHANGE AND UNCERTAINTY
• New Product Development
• Complex Systems
• System Interdependencies
• More Users
• Technology Changes
• Market Changes
• Regulatory Changes
orthogonal.ioAgile in a QMS17
18. orthogonal.ioAgile in a QMS18
MEDICAL DEVICE
CHALLENGES
The Target
Business Value/
Real-World
Outcomes
Patient Safety
Process
Efficiency
Regulatory
Compliance
19. BARRIERS TO AGILE
• Agile evolved in software, and Medical Devices used to not have much
software.
• Regulations, standards and guidance documents (CFR 21 Part 820, ISO
13485, IEC 62304) were mostly defined before agile, and are written in a
waterfall way.
• Medical Device regulations and standards have additional requirements
(risk management, design controls, verification and validation) that add
complexity to Agile.
orthogonal.ioAgile in a QMS19
20. THE FDA DESCRIBES RATHER THAN
PRESCRIBES
“The quality system requirements do not dictate the types of design process that a manufacturer must use.
Manufacturers should use processes best suited to their needs. However, whatever the processes may be,
it is important that the design controls are applied in an appropriate manner.”
FDA Design Control Guidance for Medical Device Manufacturers, p. 8
“Based on the intended use and the safety risk associated with the software to be developed, the software
developer should determine the specific approach, the combination of techniques to be used, and the level
of effort to be applied. While this guidance does not recommend any specific life cycle model or any
specific technique or method, it does recommend that software validation and verification activities be
conducted throughout the entire software life cycle.”
General Principles of Software Validation (GPSV), p. 1
orthogonal.ioAgile in a QMS20
21. RESOURCES: AAMI TIR45:2012
• Provides recommendations for complying
with international standards and FDA
guidance documents when using agile
practices to develop medical device
software.
• Agile brings value to medical device
software: continuous focus on safety, risk
management and control, impact analyses,
. . .
• Designs coherent (locally constant) and
highly cohesive (vs. tightly coupled)
• Recognized as a standard by the FDA in
2013
21
22. H O W T O B E
A G I L E I N A Q M S
Mapping Agile to IEC 62304 and CFR 21 Part 820
orthogonal.ioAgile in a QMS22
23. IEC 62304: MEDICAL DEVICE
SOFTWARE LIFECYCLE
PROCESSES
23
8. Software Configuration Management
9. Software Problem Resolution
7. Software Risk Management
5. Software Development Process
5.2 SW
Requirements
Analysis
5.3 SW
Architectural
Design
5.4 SW
Detailed
Design
5.7 SW
System
Testing
5.8 SW
Release
5.1 SW
Development
Planning
5.5 SW Unit
Implement.
& Verification
5.6 SW
Integration &
Integration
Testing
26. V-MODEL: BENT WATERFALL OR
AGILE?
orthogonal.ioAgile in a QMS26
User Requirements
System
Requirements
Specification
Software Detailed
Design
User Acceptance
Testing
System &
Integration
Testing
Unit Testing
Quality Assurance / Risk Control • Relationship between
artifacts are time-
independent
• Finish to finish parallelism
(vs. start to finish)
• Design Inputs ↔ Design
Outputs
• Synchronize Approvals
27. VERIFICATION DRIVEN AGILE
orthogonal.ioAgile in a QMS27
Verification Test
• Avoids different interpretations of
requirements and specification that are
typical in waterfall development
• This avoids “verification hell” that can double
the length and cost of a project.
• Allows for continued feedback from end
users on working software – further reducing
risk
28. R I S K
M A N A G E M E N T
I N A G I L E
It’s Iterative
orthogonal.ioAgile in a QMS28
29. THE ITERATIVE NATURE OF
RISK
orthogonal.ioAgile in a QMS29
“Risk can be introduced throughout the product lifecycle, and risks that
become apparent at one point in the life-cycle can be managed by
action taken at a completely different point in the life cycle.”
ISO 14971: 2007 Medical devices -- Application of
risk management to medical devices
30. orthogonal.ioAgile in a QMS30
ITERATIVE RISK MANAGEMENT
• Iterative Risk Analysis Takes inputs from and
provides outputs to Human Factors and
Development
• Risk Mitigations are reflected in new stories.
• Risk Mitigations are verified and validated
• The Iterative Risk Management cadence can
be different than the development cadence
31. HUMAN FACTORS
ENGINEERING
orthogonal.ioAgile in a QMS31
• Lean and agile cadence ships insights and/or UI each iteration
• User experience architecture maps and assesses human
interaction models and requirements
• Usability studies conducted with real users to identify and assess
the effectiveness of vital system interactions
• Interactive prototypes to simulate interactive functions of the
software
• User interface design of software interface using color, animation,
text and graphics
• Result:
– User Engagement
– Real-World Outcomes
32. A U T O M A T I O N
The key to maintaining quality at speed
orthogonal.ioAgile in a QMS32
33. TESTING AUTOMATION
• Full Code Coverage: Unit, Integration, Acceptance Testing built into continuous
integration environment.
• White-box and black-box testing for all elements of the stack,
from hardware to firmware to web services
• Result:
– Maintain Velocity on Larger, More Complex Projects
– Faster Parallel Development of Hardware and Software
– Faster, More Efficient Maintenance
orthogonal.ioAgile in a QMS33
34. TDD+ BDD + TESTING AUTOMATION
34
From this:
… to this
38. orthogonal.ioAgile in a QMS38
Agile development is
wasted If none of the
related processes can
iteratively provide and
receive input.
Risk
Management
Human
Factors
Verification &
Validation
Post Market
Surveillance
Product
Management
Document
Managment
40. PUTTING IT ALL TOGETHER
orthogonal.ioAgile in a QMS40
Business Value/
Real-World
Outcomes
Patient Safety
Process
Efficiency
Regulatory
Compliance
Better usability,
adherence/compliance = Improved
Engagement and Outcomes
Iterative Risk Management = Safer
Products and Reduced
Regulatory Risk
Faster Development
Streamlined Maintenance and
Manufacturing
Platform Development/Reusability
41. A G I L E
I M P L E M E N TAT I O N
M I S T A K E S
Common mistakes we’ve seen, and how to avoid them.
orthogonal.ioAgile in a QMS41
42. orthogonal.ioCompany Overview42
Area Finding (Gap) Best Practice Consequences of Gap Recommended Approach Dependencies
Quality Management Development Methodology
and Quality Management
System (QMS) are Tightly
Coupled
Loose Coupling of Quality
Management and Development
Approach.
Changes in development methodology as
needed by individual projects may result in
changes and re-approvals of parts of the
QMS which takes time and consumes
significant organizational resources
Allow for different development methodologies by
stating in QMS that any methodology based on a
standard and that meets relevant regulations is
allowed as long as it is approved by Quality.
Consider templated approach at SDP level.
Approvers of QMS (Executive
management, Quality,
Regulatory)
Risk Management Risk Management is not
iterative and integrated into
agile process
QMS states that Risk Management
Plan determines how risk control is
performed throughout the product
lifecycle.
Risk Management that is not iterative and
integrated into agile process may result in an
"Agile Falls" lifecycle where risk "in-stream"
risk analysis and mitigation suffers and risk
management activities become a bottleneck
for other project activities to continue
QMS does not specify risk management
methodology, only that risk control activities will be
performed throughout the project lifecycle as
required by relevant regulations. Risk
Management Plan outlines risk control activities
and their rationales on a per project basis.
QMS Risk Management
Process.
Verification and Validation Verification and Validation
(VnV) is not iterative and
integrated into agile process
QMS states that Validation Plan
determines how VnV is performed
throughout the product lifecycle
VnV that is not iterative and integrated into
agile process may result in an "Agile Falls"
lifecycle where verification testing fails to
indicate when an issue appears and VnV
activities become a bottleneck for other
project activities to continue.
QMS includes a VnV process definition that is
iterative and can be integrated into agile
processes.
QMS Verification And
Validation Process.
Requirements Management Requirements PRDs do not
incorporate Agile User
Stories
Include "story points" in User Stories
as user requirements
User requirements may miss important user
requirements
Include "story points" in User Stories as user
requirements. Also consider using Human Factors
Engineering and Stakeholder (e.g. clinician and
patient) flow analyses as a way of logically
grouping User Stories.
QMS Requirements
Management Process
Requirements Management Inadequate Software Tools
for Requirements
Management and
Traceability
Analyze and determine which tools
for Requirements Management and
Traceability would best fit the
organization.
Poorly specified requirements, inadequate
traceability; inability to analyze logically
categorize requirements
Follow recommendation for which tools to use for
Requirements Management and Traceability that
would best fit the organization
QMS
Testing Automation No unit or integration tests Implement a "test first" approach to
Unit Tests when specifying low level
requirements (i.e. how would this
requirement be tested?).
Poorly specified and tested requirements. Automate unit and integration testing in a
Continuous Integration Environment
QMS Test Plan, Tools
Validation
Testing Automation No Continuous Integration
Environment
Specify interface requirements and
sub-system integration tests during
system development.
Poorly specified and tested requirements. Automate Continuous Integration Environment;
justify when able to do so.
QMS Test Plan, Tools
Validation
Testing Automation No Acceptance tests Specify user acceptance tests when
constructing User Requirements.
Poorly specified and tested requirements. Automate acceptance testing (as reasonably
possible)
QMS Test Plan, Tools
Validation, Continuous
Integration Environment
Acceptance test automation
43. #1: TIGHT COUPLING
Finding: Development Methodology and Quality Management System (QMS) are Tightly Coupled
Consequences: Constraints on Implementing Agile. Changes in development methodology as needed by
individual projects may result in changes and re-approvals of parts of the QMS which takes time and
consumes significant organizational resources.
Best Practice: Loose Coupling of Quality Management and Development Approach.
Recommended Approach: Allow for different development methodologies by stating in QMS that any
methodology based on a standard and that meets relevant regulations is allowed as long as it is approved
by Quality. Consider templated approach at SDP level.
Dependencies: Approvers of QMS (Executive management, Quality, Regulatory)
orthogonal.ioAgile in a QMS43
44. #2: WATERFALL RISK MANAGEMENT
Finding: Risk Management is not iterative and integrated into agile process
Consequences: Risk Management that is not iterative and integrated into agile process may result in an "Agile
Falls" lifecycle where risk "in-stream" risk analysis and mitigation suffers and risk management activities become a
bottleneck for other project activities to continue
Best Practice: QMS states that Risk Management Plan determines how risk control is performed throughout the
product lifecycle.
Recommended Approach: QMS does not specify risk management methodology, only that risk control activities
will be performed throughout the project lifecycle as required by relevant regulations. Risk Management Plan
outlines risk control activities and their rationales on a per project basis.
Dependencies: QMS Risk Management Process.
orthogonal.ioAgile in a QMS44
45. #3: V&V NOT INTEGRATED
Finding: Verification and Validation (V&V) is not iterative and integrated into agile process
Consequences: V&V that is not iterative and integrated into agile process may result in an "Agile Falls"
lifecycle where verification testing fails to indicate when an issue appears and V&V activities become a
bottleneck for other project activities to continue.
Best Practice: QMS states that Validation Plan determines how V&V is performed throughout the product
lifecycle.
Recommended Approach: QMS includes a V&V process definition that is iterative and can be integrated
into agile processes.
Dependencies: QMS Verification And Validation Process
orthogonal.ioAgile in a QMS45
46. #4: NO USER STORIES IN THE
PRD
Finding: Product Requirement Documents do not incorporate Agile User Stories
Consequences: Requirements may miss important user requirements, and require additional translation to
agile-usable requirements
Best Practice: Include story definitions in User Stories as user requirements
Recommended Approach: Include story definitions in User Stories as user requirements. Also consider
using Human Factors Engineering and Stakeholder (e.g. clinician and patient) flow analyses as a way of
logically grouping User Stories.
Dependencies: QMS Requirements Management Process
orthogonal.ioAgile in a QMS46
47. #5: NO AGILE RM TOOLS
Finding: Inadequate Software Tools for Requirements Management and Traceability
Consequences: Poorly specified requirements, inadequate traceability; inability to analyze logically
categorize requirements, lots of extra work.
Best Practice: Analyze and determine which tools for Requirements Management and Traceability would
best fit the organization.
Recommended Approach: Follow recommendation for which tools to use for Requirements Management
and Traceability that would best fit the organization
Dependencies: QMS, Tools Validation
orthogonal.ioAgile in a QMS47
48. #6: NO TDD OR BDD
Finding: No or insufficient unit, integration and acceptance test coverage.
Consequences: Poorly specified and tested requirements. Bugs, more laborious maintenance
Best Practice: Implement a "test first" approach to Unit Tests, Integration Tests and Acceptance tests
when specifying requirements (i.e. how would this requirement be tested?).
Recommended Approach: TDD and BDD, Automate unit, integration and acceptance testing in a
Continuous Integration Environment
Dependencies: QMS Test Plan, Tools Validation
orthogonal.ioAgile in a QMS48
49. #7: NO PAIRING
Finding: No Developer Pairing
Consequences: Slower development, greater dependency on individual developers (lower “bus number”),
no built-in design review, less ability to scale projects up and down, higher training costs
Best Practice: Implement Promiscuous Pairing.
Recommended Approach: Training and Mentoring by developers experienced with pairing.
Dependencies: Developers who can pair
orthogonal.ioAgile in a QMS49
50. R E S O U R C E S
A selection of connected care solutions Orthogonal has
developed for the medical industry.
orthogonal.ioAgile in a QMS50
51. RESOURCES: AAMI TIR45:2012
• Provides recommendations for complying
with international standards and FDA
guidance documents when using agile
practices to develop medical device
software.
• Agile brings value to medical device
software: continuous focus on safety, risk
management and control, impact analyses,
. . .
• Designs coherent (locally constant) and
highly cohesive (vs. tightly coupled)
• Recognized as a standard by the FDA in
2013
51
52. RESOURCES: ORTHOGONAL
EBOOK
• Addresses the shortcomings of waterfall
development specifically in regulatory
environments.
• How agile development meets the
safety, reliability and regulatory needs of
the medical device and diagnostics
industry.
• How agile development can help ensure
delivery of successful software.
52
53. Q U E S T I O N S ?
Bernhard Kappe
bkappe@orthogonal.io
orthogonal.ioAgile in a QMS53
(As Ken said, I’m Bernhard Kappe, CEO of Orthogonal, formerly Pathfinder Software.)
Orthogonal is a medical software development company, we build connected care systems for medical devices. We build everything except the devices and the firmware. Connectivity, mobile, web, cloud, integration to midleware for clinical systems. Most falls under medical device regulations, Class I, MDDS, Class II, some class 3.
We work primarlly with medical device companies, both startups and large companies. We started out as an agile software development shop, working in a number of industries, and ended up doing work in medical devices, and dealing with the challenges of making Agile work. We learned a lot along the way. At some point I started writing a book about it, and talked to quite a few folks as part of that, including folks at the FDA and folks who had pioneered the use of agile, We picked up a number of things out of that and incorporated them into what we do.
So today I’ll start with a quick intro/recap of agile, then talk about why it makes sense in a medical device context, get down to brass tacks on how to be agile in a QMS, talk about automation, agile for non developers, highlight some common implementation mistakes, and then give you a couple of resources.
OK, Agile in 5 minutes. How many of you have done agile? Are familiar with it?
Look at traditional waterfall development process – how we used to do everything, including software. This works great when problems and solutions are well known. But the problem is that when you’re doing new product development, with new and changing underlying technologies – that’s not the case.
That’s where you run into the problems with waterfall – the cost of change curve. If you have change, you have to go back to the beginning. There’s no mechanism for dealing with change.
This is not unique to software development. That’s where iterative methods have evolved, that use fast feedback loops to deal with uncertainty and change.
All based on a build-measure-learn loop and making adjustments.
From military Observe, Orient, Decide and Act. – if your OODA loop is faster, you increase your chances of winning.
To the Toyota production system and all the lean methodologies which came from that.
Agile comes out of the same area, and is applied to software development. Lot’s of flavors, scrum, crystal, etc.
Agile does not attempt to lock down all requirements up front. It assumes that the requirements and design will evolve and change as more is learned and discovered during the process.
It works by breaking projects down into user stories - little bits of user functionality that create value for the user - prioritizing them, continuously delivering them in short one to three week cycles called iterations, and getting feedback on the working software to incorporate back into the development process.
So another key concept here is behavior driven development which fits in well with agile.
Stories – requirements. Acceptance Criteria: Specification. Acceptance Tests: Verification Protocols
So let’s walk through an agile iteration.
OK, so let’s look at the typical agile cadence. Review – planning as required. Activities in one iteration to prepare for the next, testing automation afterwards.
So the effect of these methods on the cost of change curve – they uncover a lot of issues earlier and allow you to adjust.
The effect is a higher up front cost to set up, but then a much more gradual cost of change curve.
So this has been accepted in a lot of other industries – why do we need it in this industry, and if we need it, why has it taken so long?
Software has gotten more and more important in medical devices, and now with connected care, even more so. New product development, etc.
So in an environment of change and uncertainty, you need business value and real world outcomes, you want process efficiency, both of which agile can deliver on, but you also want patient safety and regulatory compliance.
So let’s start with the FDA. The FDA does not dictate process. It describes but does not prescribe.
A number of folks pioneered the use of agile in medical devices, both at small companies and at large companies. But in a safety conscious industry, we like standards. So some of these folks took a crack at that through AAMI TIR:45, the guidance document on applying agile practices. I can highly recommend this document. It was recognized as a standard by the FDA in 2013.
OK, so let’s dig into this a little more
What AAMI TIR:45 does is focus on the software development process in IEC 62304, the generally accepted standard for medical device software development. For Agile to be successful, you need to address some of the other processes, risk management, configuration management, and problem resolution, but we’ll get to those later
This is the key diagram from TIR:45, slightly modified to include risk management. The activities are mapped at different levels
– project level software development planning, requirements analysis and architecture
- At the release level – any planning changes, integration and integration testing, system and regression testing, and release
The next level – increment – related group of stories typically – the same set
and then the meat of it is at the user story level. Requirements, emergent architectural design, detailed design (software implementation, acceptance criteria, unit testing, integration and system testing.
Risk management is a separate process, which we’ll get to later, but the yellow boxes represent inputs and outputs of those processes.
One of the biggest things that trips people up is this old graph from the 1970’s that’s still in the FDA guidance. Conceptually it’s correct, but people look at it as a waterfall type process map.
The v model is a better representation. You trace down from user requirements (at system and story level) through specifications (architecture, story acceptance criteria) and then up from unit testing through system and integration testing to user acceptance testing.
The key thing here is that the relationship between artifacts are time independent. It’s not start to finish parallelism, but finish to finish parallelism. You can’t finish the outputs before the inputs, but you can start them. That gives you flexibility in Agile.
So if you start with test driven development (write the unit test first, then the unit) and extend it to bdd – story first, story acceptance criteria – you have the start of verification driven agile - verification protocols composed mainly of story and system level acceptance tests. Stories – requirements. Acceptance Criteria: Specification. Acceptance Tests: Verification Protocols
BDD drives validation. Lose sight of intended use.
Let’s talk about risk management – design controls and risk management the core of what you need to do.
Here’s what ISO 14971 says about risk management -
Start with preliminary hazard analysis.
Risk management core to the process
Responsive to inputs
Safer!
Another iterative process is human factors – doing formative and summative human factors is part of the FDA’s guidance. Use errors are discovered through this, fed into risk management, risk mitigations can be tested, designs and usability. This feeds into development (faster to test and check in prototype. That’s a whole ‘nother talk.
So if you stop there and don’t look at automation – Agile is going to be pretty hard. Automation is the key to maintaining quality at speed, especially for long, complex projects.
If you don’t do this, bug propagation, full manual retesting every time.
Conceptually, when you do this – combine tdd and bdd with testing automation – you shorten the bug cycle and prevent propagation.
Another thing that is a great benefit is Tooling to manage the process, lightweight, manage agile requirements, traceability, hazard analysis, and documentation. We’ve found that the right tools let you set up worflows and approvals, dependencies between entities, include risk management – output documentation.
OK, let’s talk through agile in a QMS for non developers
The story of pigs and chickens is famous in agile. The pigs run the process – they’re committed. They need to manage the chickens, educate the chickens, work with the chickens. If the chickens run it, or you end up waiting on the chickens, slow down the process, gum up the works. As a chicken, you need to understand that as well.
I’d actually go a little further – a lot of agile development is wasted if none of the related processes can iteratively provide and receive feedback. You can still deal with technology change and architectural issues, missed requirements, but a lot of the benefits are there if you can actually use that input.
Ideally you have a series of iterative processes feeding into each other.
If you put it all together – you get beter usabilty,
Faster development, streamlined maintenance, etc.
So fi we have time, I’d lover to cover some of the implementation mistakes.
We’ve seen some of the same mistakes over and over again.