The purpose of this article is to provide Sponsors and CROs with a point-by-point review
of the differences between the Final FDA PRO Guidance and the Draft, highlighting the
choices made by FDA during the 3 years following the Draft PRO Guidance. These choices
reveal the FDA deliberations and resulting emphasis, and we also suggest in our review
what some of the differences might imply. Note that where terms appear highlighted
or emphasized in quotes from the Final Guidance, the emphasis has been done in the
original FDA document.
1. The Difference between the FDA Draft Guidance and Final Guidance:
How these modifications affect Sponsors and CROs
Contents The FDA Final Guidance1: Key Considerations for
PRO or ePRO Instrument Sponsors Collecting PRO and ePRO Data
PRO instrument review 2 Clinical researchers who gather data directly from patients have anticipated the FDA
Final PRO Guidance for more than 3 years. The document released in December 2009
PRO instrument definition 2 provides constructive support for collecting PRO and ePRO (electronic PRO) data with
PRO instruments measure concepts 2 scientific rigor. It establishes that FDA reviewers will evaluate protocols with respect
to the targeted labeling claims, an endpoint model, conceptual framework of PRO
PRO instrument validation 3 instruments and the content validity of PRO items. Each of these key elements is
defined and explained in the Final Guidance itself. The collaborative effort extended in
Reasons for changing a PRO instrument 4
developing the Final PRO Guidance should help clinical researchers to rely on patient
Evaluating a modified PRO instrument 5 self-reported information in support of market authorizations and advertising claims.
Specific concerns when using ePRO 8 The purpose of this article is to provide Sponsors and CROs with a point-by-point review
instruments of the differences between the Final FDA PRO Guidance and the Draft, highlighting the
choices made by FDA during the 3 years following the Draft PRO Guidance. These choices
Proving that the concept is measurable, 9 reveal the FDA deliberations and resulting emphasis, and we also suggest in our review
and the instrument is the measure of what some of the differences might imply. Note that where terms appear highlighted
the concept or emphasized in quotes from the Final Guidance, the emphasis has been done in the
Endpoint Data
original FDA document.
The endpoint model 3 The focus of the Final Guidance has been altered from a review of best practices for PRO
instrument development to FDA review considerations for PRO instruments, establishing
Safety outcomes are not endpoints 3 guidelines for evaluating existing, modified or newly created [e] PRO instruments.
The Final Guidance also provides more precise directives on how to leverage PRO
Proof of data entry times required 5 for labeling claims, and greater direction to Sponsors and CROs. The Final Guidance
Concerns about unintentional and 6 recommends that Sponsors should begin [PRO or ePRO] instrument development and
intentional unblinding evaluation early in medical product development, and should also engage the FDA in a
discussion about a new or unique PRO [or ePRO] instrument before confirmatory clinical
Missing data from patient withdrawal 6 trial protocols are finalized.
Design requirements for multiple 7 3 Key Takeaways:
endpoints
1. The Final Guidance emphasizes three aspects of PRO instruments used to support
Interpreting data beyond statistical 7 claims in approved medical product labeling: the conceptual framework, endpoint
significance model, and content validity.
A proxy-reported outcome is not a PRO 10 2. The Guidance includes an Appendix to help Sponsors prepare a dossier to be
submitted to FDA that explains and justifies the PRO instruments planned for
PRO and ePRO Collection Process an investigation.
Enhanced Wheel & Spokes diagram 4 3. PRO instrument development and use should be completed before commencing
5 criteria used to demonstrate 5 confirmatory trials. PHT suggests that Sponsors discuss planned PRO measures
content validity with us during the development phase so that we can help them optimize item and
instrument selection to suit the trial objectives and to obtain scientifically compelling
How to demonstrate a treatment 9 data directly from patients as they experience a new medical therapy.
benefit
It is critical that the clinical trial protocol define the endpoint measures and the criteria
Appendix as PRO and ePRO dossier 9 for the statistical analysis and interpretation of results, including a specification of
the conditions for a positive clinical trial conclusion, because determination of these
criteria and conditions after data are unblinded will not be credible.
1 http://www.phtcorp.com/
2. 2
The Difference between the FDA Draft Guidance and Final Guidance:
How these modifications affect Sponsors and CROs
Key Learnings Guidance for Industry (Final)
Draft Guidance February 2006
for Sponsors and CROs December 2009
PRO Sponsors and CROs who plan to rely [Lines 21-24] This guidance describes [Section I] This Guidance describes
instrument on PRO and ePRO instruments to how the FDA evaluates patient-reported how the Food and Drug Administration
review support claims in approved medical outcome (PRO) instruments used as (FDA) reviews and evaluates existing,
product labeling will know how FDA effectiveness endpoints in clinical modified, or newly created patient-
will review them. trials. It also describes our current reported outcome (PRO) instruments
thinking on how sponsors can develop used to support claims in approved
and use study results measured by medical product labeling.
PRO instruments to support claims in
approved product labeling.
PRO Sponsors should define how their PRO [Lines 45-49] In particular, the [Section I] A PRO instrument (i.e., a
instrument instruments measure treatment benefit, term instrument refers to the actual questionnaire plus the information
definition and should establish suitability of the questions or items contained in a and documentation that support its
measures before patient enrollment in questionnaire or interview schedule use) is a means to capture PRO data
confirmatory trials. along with all the additional used to measure treatment benefit or
information and documentation risk in medical product clinical trials.
that supports the use of these items [Section III.B. paragraph 5] We suggest
in producing a PRO measure (e.g., that an instrument’s measurement
interviewer training and instructions, properties be well established before
scoring and interpretation manual.) enrollment begins for confirmatory
clinical trails. Therefore, sponsors
should begin instrument development
and evaluation early in medical product
development, and engage the FDA in a
discussion about a new or unique PRO
instrument before confirmatory clinical
trial protocols are finalized.
PRO A Dossier explains how a concept being [Lines 82-89] Note, however, that PRO [Section II, paragraph 1] In clinical
instruments measured relates to clinical benefit, instruments that measure a simple trials, a PRO instrument can be used
measure endpoints and claims. concept may not be adequate to to measure the effect of a medical
concepts Sponsors and CROs may elect to utilize substantiate a more complex claim. intervention on one or more concepts
Section III as their protocol outline for For example, PRO-based evidence of (i.e., the thing being measured, such
collecting PRO and ePRO, since it lists improved symptoms alone generally as a symptom or group of symptoms,
key considerations for content: is not sufficient to substantiate a effects on a particular function or group
claim related to improvement in a of functions, or a group of symptoms
• Endpoint Model patient’s ability to function or the or functions shown to measure the
• Choice of PRO Instrument patient’s psychological state. Rather, severity of a health condition.)
• Conceptual Framework of a PRO to substantiate such a general claim,
Instrument a sponsor should develop evidence to
• Content Validity show not only a change in symptoms,
• Reliability, Other Validity, and Ability but how that change translates into
to Detect Change other specific endpoints such as
• Instrument Modification ability to perform activities of daily
• PRO Instruments Intended for Specific living, or improved psychological state.
Populations Accordingly, many PRO instruments
are specifically designed to assess
both symptoms and other possible
consequences of treatment.
3. 3
The Difference between the FDA Draft Guidance and Final Guidance:
How these modifications affect Sponsors and CROs
Key Learnings Guidance for Industry (Final)
Draft Guidance February 2006
for Sponsors and CROs December 2009
The endpoint Sponsors and CROs should define [Lines 791-798] A PRO instrument could [Section II.A.] Sponsors should define
model the role of a PRO or ePRO endpoint be the primary endpoint measure of the the role a PRO endpoint is intended to
within the protocol, and plan the study, a co-primary endpoint measure play in the clinical trial (i.e., a primary,
endpoint model. in conjunction with other objective or key secondary or exploratory endpoint)
physician-related measurements, or so that the instrument development
a secondary endpoint measure whose and performance can be reviewed in
analysis would be considered according the context of the intended role, and
to a hierarchical sequence. appropriate statistical methods can
be planned and applied. It is critical to
plan these approaches in what can be
called an endpoint model.
Safety Sponsors and CRO trial designers [Lines 164-165, Table 1] The intended [Section III.B.] Claims representing
outcomes and should be wary of combining efficacy use of the measure is general concepts often are not
endpoints measures and measures of adverse supported, even though the PRO
consequences to measure a general • To define entry criteria for study instrument was developed to measure
concept. Instead, it is recommended populations the general concepts, because the
that they separate measures • To evaluate efficacy instrument may not distinguish adverse
of treatment effectiveness from • To evaluate adverse events side effects of treatment that affect
measures of that treatment’s adverse the general concept that may not be
consequences into separate domains [Lines 269 – 271] The PRO instrument known at the time the clinical trials
that can be clearly related to proposed can be developed for a variety of roles, are designed. If adverse effects are
claims. including defining trial entry criteria, captured, PRO instruments should aim
including excessive severity, evaluating to measure the adverse consequences
treatment benefit, or monitoring of treatment separately from the
adverse events. effectiveness of treatment. As with any
clinical trial evaluating FDA-regulated
medical products, all adverse events
detected with a PRO instrument should
be included in the clinical trial report.
PRO Sponsors and CROs should include all [Lines 501-504] The FDA recognizes [Appendix Section V. Content Validity
instrument validation transcripts, as detailed in that the validation of an instrument is Documentation.] Evidence that
validation Appendix Section V and Appendix C - an ongoing process and that validity instrument captures all of the most
Transcripts. relates to both the instrument itself clinically important concepts and items,
and how it is used. Sponsors should and that items are complete, relevant
consider a PRO [or ePRO] endpoint for (appropriate), and understandable
evidence of content-related validity, to the patient. This evidence applies
the instrument’s ability to measure the to both existing and newly created
stated concepts, and the instrument’s instruments and is specific to the
ability to predict future outcomes, as planned clinical trial population and
illustrated in Table 4. indication.
4. 4
The Difference between the FDA Draft Guidance and Final Guidance:
How these modifications affect Sponsors and CROs
Key Learnings Guidance for Industry (Final)
Draft Guidance February 2006
for Sponsors and CROs December 2009
Enhanced Sponsors and CROs should follow the [Figure 1] The PRO Instrument [Section III.C., Figure 3] Development
Wheel & Wheel & Spokes objectives and specific Development and Modification Process of a PRO Instrument: An Iterative
Spokes action steps. provides general action steps did Process includes 5 objectives with
diagram not include specific details about the specific action steps: (i.) Hypothesize
development process. Conceptual Framework, (ii.) Adjust
Conceptual Framework and Draft
Instrument, (iii.) Confirm Conceptual
Framework and Assess Other
Measurement Properties, (iv.) Collect,
Analyze, and Interpret Data, (v.) Modify
Instrument.
Reasons for Sponsors and CROs are encouraged [Lines 590 – 670] The FDA intends to [Section III.C.] Table 1. Common
changing to utilize ePRO for subsequent trial consider a modified instrument as a Reasons for Changing Items during PRO
a PRO phases, since the reasons for moving different instrument from the original Instrument Development
instrument to electronic data capture can be easily and will consider measurement
demonstrated within these new table properties to be version-specific. The • Clarity or relevance…
of reasons. FDA recommends additional validation to • Response range…
support the development of a modified • Variability…
PRO instrument when one or more of the • Reproducibility…
following modifications occur. • Inter-item correlation…
• Ability to detect change…
1. Revised Measurement Content… • Item discrimination…
2. Application to a New Population or • Redundancy…
Condition… • Recall period…
3. Changed Item Content or Instrument
Format…
4. Changed Mode of Administration…
5. Changed Culture or Language of
Application…
6. Other Changes…
5. 5
The Difference between the FDA Draft Guidance and Final Guidance:
How these modifications affect Sponsors and CROs
Key Learnings Guidance for Industry (Final)
Draft Guidance February 2006
for Sponsors and CROs December 2009
5 criteria Sponsors and CROs can provide [Lines 1101-1109] Validation – [Section III.D.] Content validity is
used to evidence of content validity from these The process of assessing a PRO the extent to which the instrument
demonstrate sources, as outlined in the Appendix: instrument’s ability to measure measures the concept of interest.
content a specific concept or collection of Content validity is supported by
validity A. Literature review and concepts. This ability is described in evidence from qualitative studies
documentation of expert input terms of the instrument’s measurement that the items and domains of an
B. Qualitative study protocols, properties that are derived during the instrument are appropriate and
interview guides, and summary validation process. At the conclusion comprehensive relative to its intended
of results for focus group testing, of the process, a set of measurement measurement concept, population,
open-ended patient interviews, properties is produced that are specific and use..
and cognitive interviews to the specific population and the
C. Origin and derivation of items specific form and format of the PRO [Glossary] Content validity -
with chronology of events for instrument tested. The validity process Evidence from qualitative research
item generation, modification, involves: demonstrating that the instrument
and finalization measures the concept of interest
D. Qualitative study summary that • Identifying the concept to be including evidence that the items
supports content validity for item measured and domains of an instrument are
content, response options, recall • Assessing the content validity appropriate and comprehensive
period and scoring (i.e., being sure the items in the relative to its intended measurement
E. Summary of qualitative studies questionnaire cover the important concept, population, and use. Testing
demonstrating how item pool aspects of the concept from the other measurement properties will not
was generated, reduced, and patient perspective) replace or rectify problems with content
finalized. • … validity.
Proof of Sponsors and CROs should choose an [Lines 334 - 337] … If a patient diary [Section III.D.] … If a patient diary or
data entry ePRO System that can prove data entry or some other form of unsupervised some other form of unsupervised data
times times, prove what steps are taken to data entry is used, we plan to review entry is used, we plan to review the
required ensure that patient entries are authentic the clinical trial protocol to determine clinical trial protocol to determine what
and accurate; and include this proof in what measures are taken to ensure that steps are taken to ensure that patients
the archive for reconstruction. patients make entries according to the make entries according to the clinical
study design and not, for example, just trial design and not, for example, just
before a clinic visit when their reports before a clinic visit when their reports
will be collected. will be collected.
Evaluating a Sponsors and CROs are required [Lines 176 -181] A new PRO instrument [Section III.F.] … When a PRO
modified PRO to prove a modified instrument’s can be developed or an existing instrument is modified, sponsors
instrument adequacy. instrument can be modified is sponsors generally should provide evidence
determine that none is available, to confirm the new instrument’s
adequate, or applicable to their adequacy. That is not to say that
product development program. When every small change in application or
considering an instrument that has format necessitates extensive studies
been modified from the original, the to document the final version’s
FDA generally plans to evaluate the measurement properties. Additional
modified instrument just as it would a qualitative work may be adequate,
new one. depending on the type of modification
made…
6. 6
The Difference between the FDA Draft Guidance and Final Guidance:
How these modifications affect Sponsors and CROs
Key Learnings Guidance for Industry (Final)
Draft Guidance February 2006
for Sponsors and CROs December 2009
Concerns Sponsors and CROs should design [Lines 725 – 726] The importance [Section IV.A.1.] Open-label
about questions that minimize the effects of of blinding can be determined, in clinical trials, where patients and
unintentional possible unblinding, such as using part, by the characteristics of the PRO investigators are aware of assigned
and response items that ask for current instrument used. therapy, are rarely adequate to
intentional status, not giving patients access [Lines 729-731] Questions that ask support labeling claims based on
unblinding to previous responses, and using for current status, or PRO instruments PRO instruments.
instruments that include many items that ask many questions, are harder to To prevent influencing patient
about the same concept. answer in a biased way when previous perspectives, PRO instruments
answers are not available. administered during a clinic visit
[Lines 735-738] There are certain should be administered before other
situations, particularly in the clinical assessments or procedures.
development of medical devices, If the treatment has obvious effects,
where blinding is not feasible and such as adverse events, the clinical
other situations where there is no trial may be at risk for unintentional
reasonable control group (and therefore unblinding.
no randomization). When a PRO Suspicion of inadvertent unblinding
instrument appears useful in assessing can be a problematic review
patient benefit in those situations, the consideration for the FDA when
FDA encourages sponsors to confer with assessing PRO endpoints. Therefore,
the appropriate review division. when PRO instruments are included in
a clinical trial, we encourage sponsors
to include a single item during or at
the end of the trial to ask patients to
identify the clinical trial arm in which
they believe they participated.
Missing data Sponsors and CROs should use PRO [Lines 765- 768] We recommend the [Section IV.A.2] The clinical trial
from patient instrument administration techniques study protocol describe how missing protocol should describe how missing
withdrawal to minimize unblinding. data will be handled in the analysis. It data will be handled in the analysis.
could also establish a process by which Patients should remain in the clinical
PRO measurement is ascertained before trial, even if they have discontinued
or shortly after patient withdrawal treatment, and should continue to
from treatment exposure due to lack of provide PRO data. The protocol should
efficacy or toxicity. also establish a process by which PRO
measurement is obtained before or
shortly after patient withdrawal from
treatment should early withdrawal be
unpreventable.
7. 7
The Difference between the FDA Draft Guidance and Final Guidance:
How these modifications affect Sponsors and CROs
Key Learnings Guidance for Industry (Final)
Draft Guidance February 2006
for Sponsors and CROs December 2009
Design Sponsors and CROs should design [Lines 796- 798] The FDA recommends [Section IV.D] It is critical that the
requirements protocols with the end in mind, a that the study protocol define the clinical trial protocol define the
for multiple standard practice for PHT and the PHT study endpoint measures and the endpoint measures and the criteria
endpoints PROVision Science Team. criteria for the statistical analysis and for the statistical analysis and
interpretation of results, including a interpretation of results, including a
clear specification of the conditions for specification of the conditions for a
a positive study conclusion. positive study conclusion, because
determination of these criteria and
conditions after data are unblinded
will not be credible. Sponsors should
avoid separate consideration of PRO
endpoints from the clinical trial’s
primary objectives in terms of clinical
trial design or data analysis. Sponsors
also should avoid cherry picking or
post hoc selective picking of PRO
endpoint results for inclusion in
proposed labeling.
Interpreting Sponsors and CROs should define [Lines 474-475] The FDA generally [Section IV.E.] Planning for Clinical
data beyond and develop the responder definition intends to review a PRO instrument Trial Interpretation Using a Responder
statistical early in trial preparation, rather than for: reliability, validity, ability to detect Definition. Regardless of whether the
significance describing the minimally important change, and interpretability (e.g., primary endpoint for the clinical trial
difference. minimum important difference). is based on individual responses to
[Lines 802-807] The FDA recommends treatment or the group response,
that sponsors discuss with the it is usually useful to display
appropriate review division how best individual responses, often using an
to plan for the interpretation of study a priori responder definition (i.e. the
findings. In some cases, the FDA may individual patient PRO score change
request an a priori definition of the over a predetermined time period that
minimum observed difference between should be interpreted as a treatment
treatment group means (i.e., MID) that benefit.) The responder definition is
will serve as a benchmark to interpret determined empirically and may vary by
whether study findings are conclusive. target population or other clinical trial
In other cases, the FDA may request design characteristics. Therefore, we
an a priori definition of a treatment will evaluate an instrument’s responder
responder that can be applied to definition in the context of each specific
individual patient changes over time. clinical trial.
8. 8
The Difference between the FDA Draft Guidance and Final Guidance:
How these modifications affect Sponsors and CROs
Key Learnings Guidance for Industry (Final)
Draft Guidance February 2006
for Sponsors and CROs December 2009
Specific Sponsors and CROs should choose an [Lines 847-857] Sponsors should also [Section IV.F]… Sponsors also should
concerns ePRO System that plan to avoid the following:9 avoid the following:
about ePRO
Instruments • Archives eSource data; • Direct PRO data transmission from • Direct PRO data transmission from
• Documents all data changes with an the PRO data collection device to the the PRO data collection device to
electronic audit trail; sponsor (i.e., the sponsor should not the sponsor, clinical investigator,
• Provides database backup; have exclusive control of the source or other third party without an
• Prevents eSource modifications document) electronic audit trail that documents
except by Investigator or designated • The existence of only one database all changes to the data after it leaves
site staff (not the Sponsor, not the without backup (i.e., risk of data the PRO data collection device.
ePRO provider.) corruption or loss during the trial • Source document control by the
• Ensures retention of any adverse with no way to reconstitute or verify sponsor exclusively.
event data captured by the system; the data • Clinical investigator inability to
• Prevents premature access to • Removal of investigator accountability maintain and confirm electronic PRO
unblinded data; for confirming the accuracy of the data accuracy. The data maintained
• Ensures timely transmission of data by the clinical investigator should
important PRO safety data to the • Loss of adverse event data include an audit trail to capture
clinical investigator responsible for • Access to unblinded data any changes made to the electronic
the patient; and • Inability of an FDA inspector to PRO data at any point in time after
• Enables full trial reconstruction inspect, verify, and copy the data at it leaves the patient’s electronic
from archival records by an FDA the clinical site during an inspection device.
investigator at each clinical site. • An insecure system that allows for • The existence of only one database
easily alterable records. without backup (i.e., risk of data
corruption or loss during the trial
9
The FDA specifically welcomes with no way to reconstitute or verify
comment and additional information the data).
that will inform these policies as new • Ability of any entity other than
electronic PRO technology is developed the investigator (and/or site staff
and used in the medical product designated by the investigator) to
development setting. modify the source data.
• Loss of adverse event data.
• Premature or unplanned access to
unblinded data.
• Inability of an FDA investigator to
inspect, verify, and copy the data at
the clinical site during an inspection.
• An insecure system where records
are easily altered.
• Direct PRO data transmission
of important safety information
to sponsors, clinical research
organizations, and/or third parties,
without ensuring the timely
transmission of the data to the
clinical investigator responsible for
the patients.
9. 9
The Difference between the FDA Draft Guidance and Final Guidance:
How these modifications affect Sponsors and CROs
Key Learnings Guidance for Industry (Final)
Draft Guidance February 2006
for Sponsors and CROs December 2009
How to Sponsors and CROs should elicit PRO [Lines 109-119] Patients Provide a [Section V.E.] Because statistical
demonstrate data to characterize the treatment Unique Perspective on Treatment significance can sometimes be
treatment effect, and be prepared to explain the Effectiveness. PRO instruments achieved for small changes in PRO
benefit mean improvements within different can be developed to measure what measures that may not be clinically
patient subsets. patients want and expect from meaningful (i.e., do not indicate
their treatment and what is most treatment benefit), we encourage
important to them. When used to sponsors to avoid proposing
measure study endpoints, PRO labeling claims based on statistical
instruments can augment what is significance alone.
known about the product based on To demonstrate treatment benefit,
the clinician perspective or physiologic we find it informative to examine the
measures. This is important because cumulative distribution function (CDF)
improvements in clinical measures of responses between treatment groups
of a condition may not necessarily to characterize the treatment effect
correspond to improvements in how the and examine the possibility that the
patient functions or feels… mean improvement reflects different
responses in patient subsets…
Proving that Protocol designers should use PRO [Lines 49-52] The term conceptual [Glossary] Conceptual framework
the concept is instruments to measure treatment framework refers to how items are of a PRO instrument - an
measurable, benefit, and should examine the results grouped according to subconcepts or explicit description or diagram
and the in ways that reveal whether medical domains (e.g., the item walking without of the relationships between the
instrument is therapies work best only for certain help may be grouped with another questionnaire or items in a PRO
the measure individuals or subsets in the treatment item, walking with difficulty, within the instrument and the concepts measured.
of the population. domain of ambulation, and ambulation The conceptual framework of a PRO
concept. may be further grouped into the concept instrument evolves over the course of
of physical activity. instrument development as empiric
evidence is gathered to support item
grouping and scores. We [FDA} review
the alignment of the final conceptual
framework with the clinical trial’s
objectives, design, and analysis plan.
Appendix as Sponsors and CROs should include all No dossier outline was provided. [Appendix] Information on a PRO
PRO and ePRO relevant Appendix components within Instrument Reviewed by the FDA. The
dossier their PRO dossier. following topics represent areas that
should be addressed in PRO documents
provided to the FDA for review.