Recording of the March 19, 2012 IIBA Pharma/Biotech Special Interest Group Webinar given by Griffin Jones, titled What about...? What I wished I had asked the regulatory and testing people.
11. Requirements Analysis (Chapter 6 in the BABOK® Guide)
describes the business analysts’ role in the verification
and validation of requirements. During that
task, business analysts’ are in direct contact with the
stakeholder roles of tester and regulator.
Why are those roles special in an FDA regulated
context, and how can a business analyst more
effectively interact with them?
IIBA Pharma/Biotech SIG 3/19/2012 11
12. Outline three current topics relevant to the
tester and regulatory roles
Opportunity to ask your questions
Leave with a better understanding and
empathy for the concerns and issues of
testing FDA regulated products and services
IIBA Pharma/Biotech SIG 3/19/2012 12
13. Introduction
◦ SW Testing and Regulatory Compliance
◦ Medical devices and Clinical Research Organization
Three current topics in Testing & Regulatory
◦ How to do Perfect Testing
Jim Bullock’s simple summary
◦ The FDA’s initial thinking on Agile practices
Summary of AAMI Agile TIR SW Committee’s
“Guidance on the use of AGILE practices in the development
of medical device software”
◦ Why testing experts disagree
Status on the “Schools of Testing” conversation
Audience Questions
IIBA Pharma/Biotech SIG 3/19/2012 13
14. James Bullock’s “The Big Book of Perfect
Testing”
◦ http://www.qasig.org/past_meetings.html
Review James’s Five Principles
Follow up with Jerry Weinberg’s
◦ Perfect software: And Other Illusions about Testing
Read it yourself, then share it with others
IIBA Pharma/Biotech SIG 3/19/2012 14
15. 1 - Testing is an informational function.
◦ You are paying to learn something.
Any time your bury or misinterpret information you get from
testing you are corrupting what you are paying for.
2 - Testing produces information reliably
grounded in the observed behavior of a system.
◦ Observation is what separates testing from
analysis, induction, deduction and speculation.
Any time you argue yourself out of an observation because
what you see "just doesn't make sense" or create
circumstances that corrupt the observing, you are defeating
the purpose of testing.
One of the most powerful observations that can come from
testing is noticing what is not being, or cannot be reliably
observed. Bury that information at your peril.
IIBA Pharma/Biotech SIG 3/19/2012 15
16. 3 - Information is valuable if you act on it.
◦ Otherwise it is entertainment. So, each bit of information
produced by testing needs a customer; a
willing, enthusiastic customer eager to use the
information they hope to get from testing to do a better
job.
If you are a tester without customers, your job description is
inoperative.
If you are a manager whose testers don't have eager
customers, your testing is irrelevant.
If you work at creating artifacts, especially software, and
your testers aren't your best friends why are you so sure you
are right?
IIBA Pharma/Biotech SIG 3/19/2012 16
17. 4 - The techniques of testing are endless - literally.
◦ The correct technique is the one that best gets you the
information you need to take a better next step than you would
otherwise.
If you are stuck on a technique rather than the results, you have
descended into ritual.
If you or your testers are unwilling to discover, embrace or if
need be invent whatever technique it takes to find the
information you need, you aren't doing testing, you have
descended into administration.
◦ So what do you really want to know about your system, and how
do you find that out?
5 - Act always according to these four principles and you
will do perfect testing, every time.
◦ The rest is details.
IIBA Pharma/Biotech SIG 3/19/2012 17
18. Is there one principle that resonates with you?
1. Testing is an informational function.
You are paying to learn something.
2. Testing produces information reliably grounded
in the observed behavior of a system.
3. Information is valuable if you act on it.
4. The techniques of testing are endless - literally.
Take a better next step.
5. Act always according to these four principles.
IIBA Pharma/Biotech SIG 3/19/2012 18
19. Search for the current draft of AAMI Agile TIR
SW Committee’s
◦ “Guidance on the use of AGILE practices in the
development of medical device software”
http://www.aami.org/committeecentral/Committee/CommitteeDocume
nts.cfm?ComID=0SW0000 (Note that access to the doc is currently
restricted)
Expect that this will be applied across all SW
◦ regulated by FDA & other non-US regulatory bodies
For AGILE to be established in the medical
device software industry, guidance is needed
to adapt it to fit that unique context.
IIBA Pharma/Biotech SIG 3/19/2012 19
20. Agile adds value and can be adapted to this
context
The values of AGILE add to the QMS values
Demonstrate how your incremental SW dev.
lifecycle satisfies regulatory requirements
Agile needs strong change management
systems
Agile needs a functioning QMS to live inside
This is an important document!
Read the detailed recommendations as
soon as you can.
IIBA Pharma/Biotech SIG 3/19/2012 20
21. Has anyone else had access to the document?
Who else has started implementing Agile
practices? What works well? Challenges?
IIBA Pharma/Biotech SIG 3/19/2012 21
22. Schools of Software Testing
◦ Share some of Bret Pettichord’s slide deck
◦ http://www.prismnet.com/~wazmo/papers/four_schools
.pdf
Debate: Are Testing “Schools” a good idea? –
Doug Hoffman & James Bach
◦ http://www.associationforsoftwaretesting.org/2012/03/08/debate-
are-testing-schools-a-good-idea-doug-hoffman-james-bach/
IIBA Pharma/Biotech SIG 3/19/2012 22
23. Recent separation in the Context-Driven school
◦ Cem Kaner
http://context-driven-testing.com/?page_id=9
◦ James Bach
http://www.satisfice.com/blog/archives/724
◦ Michael Bolton
http://www.developsense.com/blog/2012/03/i-might-be-
wrong/
◦ Scott Barber
http://scott-barber.blogspot.com/2012/02/is-testing-
dead-dunno-but-context.html
◦ Matthew Heusser
http://xndev.blogspot.com/2012/03/lets-call-calling-off-
off.html
IIBA Pharma/Biotech SIG 3/19/2012 23
35. Ifind the school model to be useful to
explain my behavior, or as a line of inquiry to
understand someone else’s behavior.
Ifind “Standard” and “Quality” schools are
common in Pharma/Biotech industries
Thinkabout what school you might belong
to, and why. Perhaps you are a blend?
(Griffin shares his school)
IIBA Pharma/Biotech SIG 3/19/2012 35
36. What questions do you have?
IIBA Pharma/Biotech SIG 3/19/2012 36
38. Safe and Effective
Risk-based
Under Control
Accountability
Data driven decisions
IIBA Pharma/Biotech SIG 3/19/2012 38
39. Is it really an FDA or regulator requirement
that users have to do scripted system testing?
◦ No. Why people think/say that is interesting.
Been trying to push back because what they
have been doing as UAT is really just users
doing functional system testing.
◦ All that tells you is whether the systems fulfills the
requirements
◦ It doesn’t tell you if the user actually accepts and
can use the solution delivered.
◦ Agree - See Jim Bullock’s five principles
IIBA Pharma/Biotech SIG 3/19/2012 39
40. are they documented?
where do we find them?
how should we document or refer to them in
our requirements?
◦ How do you currently handle the regulatory
requirements for
21CFR Part 11 Electronic Records;
Electronic Signatures?
IIBA Pharma/Biotech SIG 3/19/2012 40
41. How to “cover yourself” in case you are
audited?
◦ YOU are the expert
◦ Explain why what you are doing is sufficient
◦ Too many requirements or too much detail
is as bad as too little detail
IIBA Pharma/Biotech SIG 3/19/2012 41
42. How those changes are documented
◦ Follow your processes
◦ If your processes are broken/unsustainable –
expose the pain to help get the processes fixed.
IIBA Pharma/Biotech SIG 3/19/2012 42
43. What is different in the Pharma/Biotech world
related to verification & testing?
What is the same?
IIBA Pharma/Biotech SIG 3/19/2012 43
44. Blending regulatory compliance with Agile
practices to create sustainable validation
compliance
Validation solutions that are practical and
tailored to their business model context
Alternative options to difficult software
validation problems
◦ E.g., Quintiles white paper: “The Art of Compliance”
http://www.quintiles.com/information-library/white-
papers/sustainable-regulatory-compliance/
IIBA Pharma/Biotech SIG 3/19/2012 44
45. The regulatory is not your business partner
The regulatory has police powers
See my other presentation
◦ http://www.associationforsoftwaretesting.org/?dl_name=What_do_auditor
s_expect_from_testers_-_Griffin_Jones.pdf
IIBA Pharma/Biotech SIG 3/19/2012 45
46. I have another presentation that may be
helpful for FDA regulated project teams
members
◦ “What do auditors expect from testers?
The audit survival heuristics of an FDA regulated
exploratory testing team”
http://www.associationforsoftwaretesting.org/?dl_name=What_do_audit
ors_expect_from_testers_-_Griffin_Jones.pdf
IIBA Pharma/Biotech SIG 3/19/2012 46
47. Griffin Jones
Congruent Compliance
Griffin.Jones@CongruentCompliance.com
IIBA Pharma/Biotech SIG 3/19/2012 47
