Performance metrics are an essential element of the management review process.
Quality metrics may include elements such as customer satisfaction, supplier performance, manufacturing defects, complaints, cycle times and many other internal or external processes.
This presentation provides a framework for establishing right quality indicators for evaluating the performance of the quality system.
💎VVIP Kolkata Call Girls Parganas🩱7001035870🩱Independent Girl ( Ac Rooms Avai...
Quality Metrics
1. This presentation is compiled by “ Drug Regulations” a
non profit organization which provides free online
resource to the Pharmaceutical Professional.
Visit http://www.drugregulations.org for latest
information from the world of Pharmaceuticals.
11/28/2015 1
2. This presentation is compiled from freely
available resources like the websites of FDA,
EMA, WHO.
“Drug Regulations” is a non profit
organization which provides free online
resource to the Pharmaceutical Professional.
Visit http://www.drugregulations.org for
latest information from the world of
Pharmaceuticals.
11/28/2015 2
Drug Regulations : Online
Resource for Latest Information
3. This presentation has been updated based on
the FDA Draft Guidance on Quality Metrics
released in July 2015
Visit http://www.drugregulations.org for
latest information from the world of
Pharmaceuticals.
11/28/2015 3
Drug Regulations : Online
Resource for Latest Information
4. ◦ Performance metrics are an essential element of the
management review process.
◦ Quality metrics may include elements such as customer
satisfaction, supplier performance, manufacturing defects,
complaints, cycle times and many other internal or external
processes.
◦ This presentation provides a framework for establishing
right quality indicators for evaluating the performance of
the quality system.
11/28/2015 4
Drug Regulations : Online Resource for Latest
Information
5. Accountability
◦ Metrics indicate how well an operation or project is
performing
Alignment
◦ A core objective for a strategy focused organization
is to align the entire team on a common set of
strategic objectives.
11/28/2015 5
Drug Regulations : Online Resource for Latest
Information
6. Attention
◦ Metrics incorporated into management review
provide a defined set of expectations of senior
management and are reviewed regularly
Behaviour
◦ Setting the right metrics drives the right behaviour.
11/28/2015 6
Drug Regulations : Online Resource for Latest
Information
7. Expectations
◦ Metrics allow an entire organization a common
understanding of what outcomes are most valued
by leadership.
Feedback
◦ Metrics provide short and long term feedback on
the effectiveness of projects and activities.
11/28/2015 7
Drug Regulations : Online Resource for Latest
Information
8. Motivation
◦ People like knowing their efforts are worthwhile.
Metrics provide short and long term measurement
of progress.
Objectivity
◦ Metrics form the basis for a fact-based
organization
11/28/2015 8
Drug Regulations : Online Resource for Latest
Information
9. Problem Solving
◦ Metrics enable management to uncover problems
earlier
Visibility
◦ A common set of metrics used throughout an
organization creates transparency.
11/28/2015 9
Drug Regulations : Online Resource for Latest
Information
10. ◦ There should be a standardized meaning of the
measurement
◦ The data should help the decision making process
◦ It should provide worthwhile information
◦ It should be easy to install
◦ It can be benchmarked or used elsewhere
11/28/2015 10
Drug Regulations : Online Resource for Latest
Information
11. ◦ Selecting metrics that take into account each of
these factors is a challenge
◦ Easy metrics to collect offer little value
◦ More worthwhile information requires effort to
establish a measurement process
11/28/2015 11
Drug Regulations : Online Resource for Latest
Information
12. ◦ Many metrics are possible to fully characterize a quality
system.
◦ Large number of metrics wastes resources and distracts
leaders.
◦ Implementation of too many metrics may not result in a
compliant quality system.
◦ Metrics must be carefully chosen to ensure they align
with overall strategy.
11/28/2015 12
Drug Regulations : Online Resource for Latest
Information
13. ◦ Good metrics system consists of leading and lagging
indicators.
◦ Lagging indicators measure the outcomes of what
already happened.
◦ Leading indicators provide information that may be able
to predict future outcomes.
◦ More mature organizations will typically have a greater
ratio of leading to lagging metrics.
11/28/2015 13
Drug Regulations : Online Resource for Latest
Information
14. ◦ Most compliance metrics are inherently lagging indicators and are
typically easier to define.
◦ Establishing predictive leading indicators can improve a quality
system.
◦ Begin developing metrics with lagging indicators
◦ Then ask what actions or conditions cause this outcome to
brainstorm leading indicators
◦ It is necessary to understand the cause and effect
relationships.
◦ Possible to focus on impacting drivers which will ultimately
change the outcomes.
11/28/2015 14
Drug Regulations : Online Resource for Latest
Information
15. ◦ Important to consider whether the metric measures time, quality
or quantity/cost.
◦ Time-based metrics are used to measure the duration of a
process.
Metrics such as the percentage of CAPAs completed on-time is an
example.
◦ Quantity/cost metrics track the frequency or productivity of the
process.
In a balanced scorecard these could include the cost of the process.
◦ Using financial metrics for compliance is not recommended.
11/28/2015 15
Drug Regulations : Online Resource for Latest
Information
16. ◦ Quality metrics should
Focus on effectiveness
Measure that the right things are being done correctly.
◦ Often the focus is on efficiency measured in financial
terms.
◦ Effectiveness first eliminates waste & improves
efficiency.
◦ Quality metrics are difficult to establish
◦ Provide excellent insight into the health of the quality
system.
11/28/2015 16
Drug Regulations : Online Resource for Latest
Information
17. ◦ Include representatives from all departments to design data
system successfully
◦ Align performance measurements to overall company strategic
plan
Quality objectives will support one or more elements of a strategic plan.
◦ Involve individuals from all levels
◦ Have a dialog between levels and across organization
Increase alignment,
Focus the organization
Develop buy-in
11/28/2015 17
Drug Regulations : Online Resource for Latest
Information
18. ◦ Document the metrics definitions.
◦ Helps in collection of consistent metrics across
organization over time.
11/28/2015 18
Drug Regulations : Online Resource for Latest
Information
19. Definition One sentence explaining the Metric
Per cent of CAPA completed within agreed time
Benefit Quality objective to be achieved
Are non conformances resolved in a timely manner ?
Formula Specific formula to calculate the parameter
Target Give desirable goal
90 % , 10 % improvement over last year ,…..
Frequency Give desired frequency
Monthly , Quarterly , Yearly ,……
11/28/2015 19
Drug Regulations : Online Resource for Latest
Information
20. Status Rules Colors could be used for performance
Red ≤ 70 % , Yellow = 71% to 89% , Green =90% to 100%
Data Source Indicate where the data is availble
CAPA system
RACI Responsible: person(s) influencing the metric performance;
Accountable: person(s) ultimately responsible for the metric
performance.
Consulted: person(s) with knowledge in improving the metric
performance.
Informed: person(s) likely to benefit from the updates on the metric
performance.
Responsible: auditors; Accountable: Senior Audit Manager,
Consulted: none; Informed Quality council, Quality Director.
11/28/2015 20
Drug Regulations : Online Resource for Latest
Information
21. ◦ Set targets and goals in alignment with strategy
◦ Have realistic expectations
◦ Collect baseline data for several reporting periods
◦ Establish current performance level with baseline data
◦ Establish improvement targets
◦ Measure performance improvement against the baseline
◦ Targets =“SMART-A”
Specific, Measureable, Attainable, Realistic, Time-bound and
Aligned.
11/28/2015 21
Drug Regulations : Online Resource for Latest
Information
22. Evolution of a Measure
◦ % of acceptable computer reports delivered
on time
◦ Number of acceptable computer reports
delivered on time
◦ Number of computer reports delivered on
time
◦ Number of computer reports delivered
11/28/2015 22
Drug Regulations : Online Resource for Latest
Information
Extent
Efficiency
Effectiveness
23. ◦ Ensure that metric is performing as intended.
◦ Change the metric definition with caution as some
changes may invalidate older measurements.
◦ Need for a metric may change over time.
◦ Remove the metrics if it does not add value
11/28/2015 23
Drug Regulations : Online Resource for Latest
Information
24. ◦ Establish Quality performance metrics based on
linkages to organizational objectives.
◦ Link to product or service performance, quality
costs, quality improvement and customer
satisfaction.
◦ Share the Quality goals across the organization.
11/28/2015 24
Drug Regulations : Online Resource for Latest
Information
25. ◦ Quality metrics may be collected and reported by
the quality function
◦ Ownership of the issue belongs to operating
function
◦ Pareto principle applies to quality problems
◦ 80% of the quality problems are outside the control
of the quality function, while 20% may be internal.
11/28/2015 25
Drug Regulations : Online Resource for Latest
Information
26. ◦ Organization wide problems could include
Supplier quality requirements
R&D product designs
Distribution system problems
◦ Internal quality problems may include
Lab test issues
Auditing procedures
Sampling procedures
11/28/2015 26
Drug Regulations : Online Resource for Latest
Information
27. ◦ Performance against regulatory agency inspections
Basic indicator of compliance
Commercialization of products
Manufacturing authorization.
11/28/2015 27
Drug Regulations : Online Resource for Latest
Information
28. ◦ Internal Audit by an independent Auditor
Snapshot of performance .
On-going feedback on the health of the quality
system.
Not an assurance of health of the quality system.
◦ Management Review to evaluate complete Quality
System
11/28/2015 28
Drug Regulations : Online Resource for Latest
Information
29. ◦ Favourable outcomes help confirm compliance and
conformance.
◦ Unfavourable outcomes
significant business interruptions
Business loss
Impact public perceptions and
Decrease the value of a company’s brand and image
11/28/2015 29
Drug Regulations : Online Resource for Latest
Information
30. ◦ Performance metrics against external inspections
are reactive measures.
◦ Do not rely on Regulatory inspection as a single
indicator.
◦ Conduct Internal audits more regularly
Will enable a proactive response to avoid an external
finding.
Functional area self-audits offer an excellent
opportunity for most proactive measurement.
11/28/2015 30
Drug Regulations : Online Resource for Latest
Information
31. Outcome Measure Method Other Considerations
% of inspections with no citations Trend Analyse citations according their
nature and details
Develop Pareto analysis of citations
Evaluate whether there are relative
differences between inspecting
authorities
% of audits with no adverse findings Trend Evaluate difference between Internal &
External performance.
Evaluate difference between sites ,
departments
Frequency and severity of
findings
Trend Evaluate difference between Internal &
External performance.
Evaluate difference between sites ,
departments
11/28/2015 31
Drug Regulations : Online Resource for Latest
Information
32. Process Measure Method Other Considerations
% of inspections with Timeliness of
internal audits against schedule
Trend Evaluate whether there are relative
differences between departments or
auditors
Indicators that audits covered
all quality systems, sites and
functions
Trend Evaluate differences between auditors
or departments
Number/per cent of open
inspection and audit actions
Trend Probe for reasons for open items.
Timeliness of closure of
inspection and audit actions
Trend Probe reasons for delay
Evaluate whether there are relative
differences between departments or
Auditors
11/28/2015 32
Drug Regulations : Online Resource for Latest
Information
33. ◦ Analyse the findings according their nature and
details.
◦ Track with regulation citation number being
violated.
◦ Develop a pareto of finding by citations.
◦ Focus of global improvement efforts should be the
highest frequencies of findings
◦ Compare & calibrate internal audit group to
external authorities
11/28/2015 33
Drug Regulations : Online Resource for Latest
Information
34. ◦ Refine audit program if internal audits do not
identify similar trends in findings as external
inspections
◦ Perform external benchmarking.
◦ Compare with studies showing pareto analysis of
finding by agencies such as the FDA.
◦ Carefully examine trends of external findings
frequency.
11/28/2015 34
Drug Regulations : Online Resource for Latest
Information
35. ◦ Increase emphasis on emerging issues
◦ Adjust the internal audit plan to include covering
emerging issues
◦ Prepare organization for the frequent inspection
areas.
11/28/2015 35
Drug Regulations : Online Resource for Latest
Information
36. ◦ Crucial to develop appropriate metrics.
◦ Important window to compliance and the health of
the quality system.
◦ Frequently a focus area for external inspectors.
◦ CAPA’s not addressed thoroughly & promptly raise
red flag during inspection
Points to unresolved problems
Indicates unresponsive management
11/28/2015 36
Drug Regulations : Online Resource for Latest
Information
37. Outcome Measure Method Other Considerations
% Investigations which discover root
cause
Trend Confirm the level of reoccurrences,
root cause and effectiveness and that
there are not any adverse trends or
patterns (current value against target,
trending).
Consider whether there are any
products, processes or departments
with a disproportionate number of
issues.
% of Nonconformance recurrence Trend
11/28/2015 37
Drug Regulations : Online Resource for Latest
Information
38. Process Measure Method Other Considerations
% Investigations completed within
defined timelines
Trend Evaluate any investigations that are
open for extended periods of time.
Probe the reasons for the delay
% of actions completed within
defined timelines
Trend Evaluate actions that are open for
extended period of time.
Probe the reason for delay
Frequency of Non conformances Trend Evaluate the location of the event,
cause, product, process step
11/28/2015 38
Drug Regulations : Online Resource for Latest
Information
39. ◦ Analyse additional CAPA system information.
◦ Categorize the nature and details of the nonconformities.
Categorization by
Product family
Part number
Type of process
Equipment used
◦ Understand the cause and resolution of problems.
Equipment failure
Human error
Supplier
11/28/2015 39
Drug Regulations : Online Resource for Latest
Information
40. ◦ Evaluate problems attributed to human error
◦ Indicates failure to identify true root cause
◦ Monitor non-conformances & CAPA that are classified as
high risk
◦ Give priority for completion and confirmation
◦ When “non conforming” products are disposed off “as is”
make the decision visible to management.
◦ Give higher priority to CAPA for recalls, and complaints.
11/28/2015 40
Drug Regulations : Online Resource for Latest
Information
41. ◦ Systems critical to the product safety & patient.
◦ Regulators Evaluate these systems very closely
◦ Complaint and Product Action process measures
determine the timeliness of key process steps
◦ Complaints open for a long period of time are a red
flag
Points to a bigger problem
May also indicate lack of resources or focus
◦ Adverse events have defined reporting time frame
11/28/2015 41
Drug Regulations : Online Resource for Latest
Information
42. Outcome Measure Method Other Considerations
Frequency of complaints
confirmed as a non-conformance
Trend
Pareto
Evaluate by product and product
family
Frequency of product recalls Trend Impact on other batches 0r products
Frequency of adverse events Trend Evaluate by product and product
family
11/28/2015 42
Drug Regulations : Online Resource for Latest
Information
43. Process Measure Method Other Considerations
Frequency of complaints by
product
Trend
Pareto
Frequency of complaints by
nature of complaint
Trend
Pareto
% of complaints closed within
defined timeframe
Trend Evaluate any complaints that are open
for extended periods of time
Regulatory notification filing
timeliness
Trend
11/28/2015 43
Drug Regulations : Online Resource for Latest
Information
44. ◦ Complexity of supply chain has increased
◦ Significant importance for supplier quality performance
◦ Rigorous Quality system essential for Supplier
◦ Suppliers may include any one or more
Providers of direct manufacturing materials including commodities,
Starting materials,
Chemicals,
Excipients,
API (active pharmaceutical ingredients),
Drug product,
Finished dosage and
Third party manufacturers of finished products.
11/28/2015 44
Drug Regulations : Online Resource for Latest
Information
45. ◦ A separate metrics may used for
CRO for testing
CRO for clinical studies
Pest control service
Calibration services
IT services
11/28/2015 45
Drug Regulations : Online Resource for Latest
Information
46. ◦ Part of a company’s quality system
◦ Act within their own quality system
◦ Monitor both perspectives with metrics
◦ Internal and external non-conformance are
examples of supplier driven issues
◦ Look at a supplier as a whole quality system
◦ A scorecard could be developed for critical
suppliers
11/28/2015 46
Drug Regulations : Online Resource for Latest
Information
47. ◦ Scorecard
Audit outcomes
Lot failure rates
Delivery performance and
Responsiveness
◦ Prepare a composite metric for each supplier
◦ Prepare a composite metric from all critical supplier
scorecards
11/28/2015 47
Drug Regulations : Online Resource for Latest
Information
48. Outcome Measure Method Other Considerations
Frequency of non-conformances
due to supplier
Trend
Pareto
Evaluate overall and by supplier
Frequency of Incoming testing
failures
Trend
Pareto
Evaluate overall and by supplier
Frequency of In-process failures due
to supplier
Trend
Pareto
Evaluate overall and by supplier
11/28/2015 48
Drug Regulations : Online Resource for Latest
Information
Process Measure Method Other Considerations
Timeliness of internal supplier
audits against schedule
Trend
Pareto
49. 11/28/2015 49
Drug Regulations : Online Resource for Latest
Information
Analytical Lab metrics
◦ Frequency of OOS (out of specification)
Calibration metrics
◦ Frequency of calibration OOS (out of specification)
◦ Calibration schedule conformance
Cleanroom metrics
◦ Frequency of cleanroom OOS
54. In July 2012, the Food and Drug
Administration Safety and Innovation Act
(FDASIA) was enacted.
This act requires the formation of a task force
to develop and implement a strategic plan for
enhancing the Agency's response to
preventing and mitigating drug shortages.
11/28/2015 54
Drug Regulations : Online Resource for Latest
Information
55. The Task Force is specifically interested in seeking
public input on amongst others :
“In an effort to address the major underlying causes
of drug and biological product shortages, FDA is
seeking new ideas to encourage high-quality
manufacturing and to facilitate expansion of
manufacturing capacity”
To assist in the evaluation of product manufacturing
quality, FDA is exploring the broader use of
manufacturing quality metrics.
11/28/2015 55
Drug Regulations : Online Resource for Latest
Information
56. ◦ FDA is now required to do risk based inspections
◦ Several risk factors are to be considered in the evaluation
(next slide)
◦ FDA can collect records otherwise available on inspection
“in advance or in lieu of inspection”
◦ Quality Metrics are considered potentially the sort of
information that could be provided in advance.
11/28/2015 56
Drug Regulations : Online Resource for Latest
Information
57. FDA plans to use its authority to collect records
"in advance of or in lieu of" an inspection,
under section 704(a)(4)(A) of the FD&C Act to
gather various quality metrics data records.
The agency says it will use these records to
"further develop [its] risk-based inspection
scheduling."
11/28/2015 57
Drug Regulations : Online Resource for Latest
Information
58. A. The compliance history of the establishment.
B. The record, history, and nature of recalls linked to the establishment.
C. The inherent risk of the drug manufactured, prepared, propagated,
compounded, or processed at the establishment.
D. The inspection frequency and history of the establishment, including
whether the establishment has been inspected pursuant to section 704
within the last 4 years.
E. Whether the establishment has been inspected by a foreign
government or an agency of a foreign government recognized under
section 809.
F. Any other criteria deemed necessary and appropriate by the Secretary
for purposes of allocating inspection resources.
11/28/2015 58
Drug Regulations : Online Resource for Latest
Information
59. Implications of Providing Quality Metrics :
◦ Identify lower risk products, processes & sites
◦ Regulatory flexibility of lower reporting
Prior approval supplement to an annual report
◦ Identify high quality site needing lower inspection
◦ Pathway to more objective inspections
11/28/2015 59
Drug Regulations : Online Resource for Latest
Information
60. ◦ To assist in the evaluation of product manufacturing quality, FDA
is exploring the broader use of manufacturing quality metrics.
◦ With that in mind, FDA would like input on the following issues:
What metrics do manufacturers currently use to monitor production
quality?
To what extent do purchasers and prescribers use information about
manufacturing quality when deciding how to purchase or utilize
products?
What kinds of manufacturing quality metrics might be valuable for
purchasers and prescribers when determining which manufacturers to
purchase from or which manufacturers' products to prescribe?
What kinds of manufacturing quality metrics might be valuable for
manufacturers when choosing a contract manufacturer?
How frequently would such metrics need to be updated to be
meaningful?
11/28/2015 60
Drug Regulations : Online Resource for Latest
Information
61. The FDA released a draft guidance detailing
how it plans to use quality metrics in July
2015
Based on this FDA proposes to
◦ Improve its ability to conduct risk-based
inspections and
◦ Predict or mitigate potential drug shortages.
11/28/2015 61
Drug Regulations : Online Resource for Latest
Information
62. New draft guidance Request for Quality
Metrics
FDA is giving more details to industry on its
approach to quality metrics,
FDA is also asking industry for its input in
specific areas that are still being
considered.
11/28/2015 62
Drug Regulations : Online Resource for Latest
Information
63. For companies with robust quality metrics data,
this information could be used to "reduce the
inspection frequency at an establishment."
In cases where the data raises issues, FDA says
having the quality metrics data will allow it to
"improve the efficiency and effectiveness" of the
inspections.
For now, the agency foresees its use of quality
metrics to be used primarily for reducing
inspection frequencies.
11/28/2015 63
Drug Regulations : Online Resource for Latest
Information
64. FDA says it will request the following 10 baseline quality
metrics from companies as part of its analysis:
◦ The number of lots attempted of the product.
◦ The number of specification-related rejected lots of the product,
rejected during or after manufacturing
◦ The number of attempted lots pending disposition for more than 30
days
◦ The number of out-of-specification (OOS) results for the product,
including stability testing.
◦ The number of lot release and stability tests conducted for the
product.
11/28/2015 64
Drug Regulations : Online Resource for Latest
Information
65. FDA says it will request the following 10 baseline quality
metrics from companies as part of its analysis:
◦ The number of product quality complaints received for the product.
◦ The number of lots attempted which are released for distribution or
for the next stage of manufacturing the product.
◦ If the associated annual product reviews (APRs) or product quality
reviews (PQRs) were completed within 30 days of annual due date for
the product
◦ The number of APRs or PQRs required for the product
11/28/2015 65
Drug Regulations : Online Resource for Latest
Information
66. The agency is also asking for input on five "additional, optional
metrics as evidence of manufacturing robustness and a
commitment to quality."
◦ Senior Management Engagement—Was each APR or PQR reviewed and approved
by the following: (1) The head of the quality unit, (2) the head of the operations
unit, (3) both, or (4) neither?
◦ Corrective Action and Preventive Action (CAPA) Effectiveness—What percentage
of your corrective actions involved re-training of personnel (i.e., a root cause of
the deviation is lack of adequate training)?
◦ Process Capability/Performance—A “yes” or “no” value of whether the
establishment's management calculated a process capability or performance
index for each critical quality attribute as part of that product's APR or PQR.
11/28/2015 66
Drug Regulations : Online Resource for Latest
Information
67. The agency is also asking for input on five "additional,
optional metrics as evidence of manufacturing robustness
and a commitment to quality.”
◦ Process Capability/Performance—A “yes” or “no” value of whether the
establishment's management has a policy of requiring a CAPA at some
lower process capability or performance index.
◦ Process Capability/Performance—If “yes” to the previous question—What is
the process capability or performance index that triggers a CAPA? If “no”
to the previous question—please do not respond.
11/28/2015 67
Drug Regulations : Online Resource for Latest
Information
68. Keep it simple
Improve objectivity
Potential for regulatory flexibility
Build consensus
Promote right behavior
11/28/2015 68
Drug Regulations : Online Resource for Latest
Information
69. This presentation was compiled from freely
available resources like the websites of FDA,
EMA, WHO.
“Drug Regulations” is a non profit
organization which provides free online
resource to the Pharmaceutical Professional.
Visit http://www.drugregulations.org for
latest information from the world of
Pharmaceuticals.
11/28/2015 69
Drug Regulations : Online
Resource for Latest Information