Establishing Performance Standards

1. Establishing Performance Standards:
A Practical Approach
Establishing Performance Standards:
A Practical Approach
by David G. Rhoads, Ph.D., DABCC
Copyright 2005 by David G. Rhoads Associates, Inc.

2. 2
Topics
Topics
z Define performance standards and show why they
are important.
z List sources for calculating performance standards
z Show formulae for those specific calculations
z Define process for figuring out how to calculate a
performance standard.
z Show examples for specific performance standards.

3. 3
What are Performance Standards?
What are Performance Standards?
z Performance Standards define the required quality of
the basic product of the clinical laboratory, patient
results.
z Performance Standards are used in two ways:
– To define allowable systematic error (i.e. bias)
– To define allowable random error (target SD values for
routine QC).
z These two metrics define the quality specifications of
our product.

4. 4
Status of Performance Standards
Status of Performance Standards
z There is no consensus in our industry that we need
performance standards except perhaps for the CLIA
’88 PT limits.
z There is no consensus on how to set performance
standards.
z There is no consensus on what those performance
standards should be.
z Achieving a consensus on these issues may take a
decade. We need to start the process now.

5. 5
What are Performance Standards?
What are Performance Standards?
z Performance Standards for Clinical Laboratory Tests
is expressed in terms of Experimental Error.
– I like the term Performance Standards because it is
positive. I use it instead of Total Allowable Error
(TEa) because the latter has negative connotations
and is technical.
z Understanding experimental error is essential to
establishing and maintaining a program of good
quality assurance and good quality control.
z We will see the consequences of QA failure in the
next few slides.

6. 6
Regulatory Effect of QC Failure
Regulatory Effect of QC Failure

7. 7
Clinical Effect of QC Failure
Clinical Effect of QC Failure
z Many clinical protocols are triggered by results
exceeding a certain medical decision point. They
assume reasonably accurate results.
z What happens if
– Some specimens with true results below this value in
fact exceed the MDP?
– Some specimens with true results above this value in
fact do NOT exceed the MDP?
z Both can cause incorrect diagnoses plus extra cost
and pain for both the health care system and the
patient as the system deals with consequences of
poor lab results.

8. 8
Healthcare System Costs –
Incorrect Calibration
Healthcare System Costs –
Incorrect Calibration
z If Calcium has a bias to the high side, a whole series
of follow-up tests may be ordered in order to confirm
the initial diagnosis of hypercalcemia.
z The costs of the follow-up lab tests alone ranges
from $60 to 199 million per year nationally depending
on the magnitude of the calibration error.
– Clin Lab News, Aug 2004. How much does Test Calibration Error Cost?
z This is a very conservative estimate based only on
the Medicare costs of possible follow-up tests. It
does not include patient or physician costs for the
follow-ups.

9. 9
A Documented Instance of Error
A Documented Instance of Error
z Dr. George Klee at Mayo Clinic did an experiment in
which he evaluated the change in the number of
patients with cholesterol values exceeding the cutoff
value in the presence of varying amounts of bias.
z If bias changed 1%, there was a 5% change in the
number of patients exceeding the 200 mg/dL cutoff.
z If bias changed 3%, the corresponding change was
15%.

10. 10
Costs of Quality Failure
Costs of Quality Failure
z If lab results causing incorrect diagnoses amounted to
0.12% of the results for a lab producing 1 million tests a
year, that would mean 1,200 patients per year would be
incorrectly diagnosed (i.e. about 3.5 per day).
z If the average additional cost for each of these incorrectly
diagnosed patients was only $1,000, then these incorrect
results would result in an additional $1.2 million cost per
year for the health care system.
z This cost is a significant fraction of the total budget for a lab
performing this volume of work. The budget for a lab this
size is approximately $3 million per year.
z Plebani and Carraro reported 49 medically significant errors from 40,490 tests
(0.12%). A medically significant error is one which resulted in an inappropriate
care or evaluation of a patient. Ref: CCJ 43, 1348 (1997)

11. 11
Business Costs of QC/QA Failure – 1
Business Costs of QC/QA Failure – 1
z Loss of business
– Poor quality of chemistry results in one Southern
hospital caused physicians to send their patients to
another hospital in the same city.
• Private communication to DG Rhoads – April, 2004
z Loss of jobs
– At Maryland General Hospital in Baltimore, after QC
failures for HIV testing were widely publicized, hospital
and lab management were replaced.
– A “vast majority” of 460 patient results were repeated.
99.6% were unchanged. (Two did change.)
• Testimony before a subcommittee of the House Committee of Government
Reform, May 18, 2004

12. 12
Business Costs of QC/QA Failure – 2
Business Costs of QC/QA Failure – 2
z Hospital failed!!
– At St. Agnes Medical Center in Philadelphia, 3 patients
died from inaccurate hemostasis results in 2001.
Hospital closed in 2004.

13. 13
Key Role of Clinical Laboratory
Key Role of Clinical Laboratory
z Clinical laboratory costs are 3-4% of the total U.S.
health-care budget.
z At Mayo Clinic, 94% of the objective data in patients’
charts were from the laboratory.
R. Forsman, Clinical Lab News, July 2004, p 12.
z Lab data leverages 60-70% of all critical decisions.
R. Forsman, Clinical Chemistry, 42, 813 (1996)
z In other words, while we are but a small
component of the total healthcare system,
our data control a large majority of the
critical decisions.

14. 14
Scope of TEa Problem
Scope of TEa Problem
z Object is to obtain values which are useful,
defensible and attainable.
z Many analytes (>1000)
z Many classes of analytes.
– Endogenous - Chemistry
– Hematology (CBC, Diff)
– Tox – TDM and DAU
– PCR type tests (Viral loads)
– Serology tests (Quant -> Qual result)
– Urinalysis (Qual and semi-quant)

15. 15
Performance Standards Definitions
Performance Standards Definitions
z Three classes
– Clinical
– Establishment
– Deployment

16. 16
Clinical Definition
Clinical Definition
z These define the clinical goals without specifying
how they are to be accomplished.
– Definition #1: The amount of error that can be
tolerated without invalidating the medical usefulness of
the analytical result" (Carey and Garber - 1989).
– Definition #2: The amount of error which will allow one
to calculate the probability that a change in clinical
condition has occurred since that patient was tested
previously. (Biological variation)

17. 17
Establishment Definitions
Establishment Definitions
z These definitions are used to define TEa values.
(preferred definitions are higher on list.)
– Medical Requirements
– Biological Variation
– Reference Interval
– Regulatory Requirements (CLIA ’88 or EQAS)
– Achievable Error
• Peer Group Survey Results
• Process SD
• CLSI (formerly NCCLS) EP 21
– Manufacturer’s Claim for Method

18. 18
Deployment Definitions
Deployment Definitions
z These definitions define algorithms which convert a
TEa into target SD’s for daily QC and allowable bias.
– Allocation by fixed fraction
• 25% Rule
• Six Sigma
– Specification of QC Rules based on magnitude of
systematic and random errors.
• Westgard Approach
– Biological Variation

19. 19
Establishment Definitions
Establishment Definitions
z Medical Requirements
z Biological Variation
z Reference Interval
z Regulatory Requirements (CLIA ’88 or EQAS)
z Achievable Error
– Peer Group Survey Results
– Process SD
– CLSI (formerly NCCLS) EP 21
z Manufacturer’s Claim for Method

20. 20
Medical Requirements Theory
Medical Requirements Theory
z Several possible approaches to establishing quality
specifications:
– For specific clinical situations
– Based on medical opinion
– Based on national or international groups
– Based on institutional or expert individuals

21. 21
Approach – Based on Medical
Interpretation Requirements
Approach – Based on Medical
Interpretation Requirements
z From NCEP
– Cholesterol, HDL, and LDL
z From the American Diabetic Association
– Glucose and HbA1c
z From NKEP
– Creatinine
z Others

22. 22
TEa from Med Req -- Examples
TEa from Med Req -- Examples
z Cholesterol
– 5% vs 10% from CLIA ‘88
z Glucose
– 10% (ADA??)

23. 23
Establishment Definitions
Establishment Definitions
z Medical Requirements
z Biological Variation
z Reference Interval
z Regulatory Requirements (CLIA ’88 or EQAS)
z Achievable Error
– Peer Group Survey Results
– Process SD
– CLSI (formerly NCCLS) EP 21
z Manufacturer’s Claim for Method

24. 24
Biological Variation - Explained
Biological Variation - Explained
Taken from Callum Fraser’s “Biological Variation: From Principle to Practice”

25. 25
Approach based on Biological Variation
Approach based on Biological Variation
z Are for inter- and intra- individual variations.
z For most analytes, the intra-individual variation is
less than the inter-individual variation. For a few
tests, it is larger.
z Due to the nature of the calculations, they are
expressed in percent terms (% CV).

26. 26
BV TEa – Advantages
BV TEa – Advantages
z If one uses this approach, it is possible to calculate
the probability that a significant clinical change has
occurred in a patient based on the difference
between his serial results.
z Many workers in the field consider this to be the
preferred approach.

27. 27
BV TEa – Issues
BV TEa – Issues
z Specifications currently available only for about 280
analytes out of a total lab menu of >1000 tests.
z Assumes that care-givers know significance of this
specification and are able to use it.
z Not applicable for many analytes (i.e. drugs, certain
disease markers, qualitative tests).
z Some TEa’s are too large or too small.
z Use of BV makes the assumption that it is constant
across all populations.

28. 28
BV Sources
BV Sources
z Two sources (about 280 analytes):
– Ricos et al (Westgard’s website – 270 tests)
– Lacher et al (CCJ – Feb, 2005 – 42 tests)

29. 29
BV TEa – Examples
BV TEa – Examples
Desirable TEa (%)
Ricos et al Lacher et al
ALT 32.1 33.4
Albumin 3.9 4.6
Total Bilirubin 31.1 32.0
Calcium 2.4 4.2
Cholesterol 9.0 12.7
Glucose 6.9 10.6
Sodium 0.9 1.6

30. 30
Establishment Definitions
Establishment Definitions
z Medical Requirements
z Biological Variation
z Reference Interval
z Regulatory Requirements (CLIA ’88 or EQAS)
z Achievable Error
– Peer Group Survey Results
– Process SD
– CLSI (formerly NCCLS) EP 21
z Manufacturer’s Claim for Method

31. 31
TEa from Reference Intervals
TEa from Reference Intervals
z I have found two rules to calculate TEa as a fraction
of a reference interval.
z Tonk’s Rule
TEa = 25% * RI
z CLIA ’88 Rule
TEa = 50% * RI
– I propose that this 50% RI rule be established as the
maximum value for TEa.

32. 32
Reference Intervals – Issues
Reference Intervals – Issues
z There are many tests for which RI’s are not available.
In particular, these have cutoff values. Examples
include Troponin, CKMB and HbA1c.
z Most reference intervals are not industry standard.
They vary between labs.

33. 33
TEa from Ref Intervals – Example
TEa from Ref Intervals – Example
Calcium TEa (mg/dL)
Reference
Interval Range
Tonks Rule
25% of Range
CLIA Rule
50% of Range
8.5 – 10.5
(Traditional)
2.0 0.5 mg/dL
(0.5/9.5 = 5.3%)
1.0
(10.5%)
8.9 – 10.1
(Recent)
1.2 0.3
(3.2%)
0.6
(6.3%)
z Major implications of this table with respect to setting routine
QC target SD values, especially if 25% Rule is applied.
– 1 SD would range from 0.8% (0.08 units) to 2.5% (0.25 units).

34. 34
Establishment Definitions
Establishment Definitions
z Medical Requirements
z Biological Variation
z Reference Interval
z Regulatory Requirements (i.e. CLIA ’88)
z Achievable Error
– Peer Group Survey Results
– Process SD
– CLSI (formerly NCCLS) EP 21
z Manufacturer’s Claim for Method

35. 35
PT Approach
PT Approach
z Values are available for about 75 analytes in USA.
z Values were set administratively in early ’90’s so they
may not be the best
z However since these values exist and they are not
grossly out of line with appropriate values, they are
a good starting point.
z When available, CLIA ’88 and EQAS values set
upper limits on TEa.

36. 36
TEa from CLIA ’88 Regs – Examples
TEa from CLIA ’88 Regs – Examples
Analyte TEa - CLIA ’88 Limits
Erythrocyte count (RBC) +/- 6%
Prothrombin time +/- 15%
Calcium +/- 1.0 mg/dL
ALT (SGPT) +/- 20%
Blood gas pO2 +/- 3 SD
Glucose +/- 6 mg/dL or 10% (greater)
HCG +/- 3 SD
Digoxin +/- 20% or 0.2 ng/mL (greater)

37. 37
Establishment Definitions
Establishment Definitions
z Medical Requirements
z Biological Variation
z Reference Interval
z Regulatory Requirements (CLIA ’88 or EQAS)
z Achievable Error
– Peer Group Survey Results
– Process SD
– CLSI (formerly NCCLS) EP 21
z Manufacturer’s Claim for Method

38. 38
Achievable Total Error - 1
Achievable Total Error - 1
z This represents the actual observed total error of a
method as compared with a goal established using
clinical or population data.
z This is the fall-back approach when the other
methods are not available.

39. 39
Establishment Definitions
Establishment Definitions
z Medical Requirements
z Biological Variation
z Reference Interval
z Regulatory Requirements (CLIA ’88 or EQAS)
z Achievable Error
– Peer Group Survey Results
– Process SD
– CLSI (formerly NCCLS) EP 21
z Manufacturer’s Claim for Method

40. 40
TEa from Peer Group Surveys
TEa from Peer Group Surveys
z This concept is based on the PT specification of
target +/- 3SD.
z Preferable source is PT or EQAS results. Need
number of eligible specimens to be at least 5, more
can be used if available.
z Fundamental advantage of this approach is its
accessibility. It is available for almost every test
performed in most clinical laboratories.
z Disadvantage is that it is based on what can be
done, not on medical or BV requirements.

41. 41
Establishment Definitions
Establishment Definitions
z Medical Requirements
z Biological Variation
z Reference Interval
z Regulatory Requirements (CLIA ’88 or EQAS)
z Achievable Error
– Peer Group Survey Results
– Process SD
– CLSI (formerly NCCLS) EP 21
z Manufacturer’s Claim for Method

42. 42
Process SD or CV
Process SD or CV
z Represents the actual performance of the process.
z One must be careful to select values which truly
represent the long-term precision of the method as
obtained by conscientious workers.
– It is entirely possible to use values here which mis-
represent the precision of the system, so one must be
very careful about which values to use.

43. 43
TEa from Process SD or CV values
TEa from Process SD or CV values
z Calculate TEa from a Process SD or CV.
z This is the SD or CV observed over a period of time
as judged by routine monthly QC results.
z This is NOT necessarily the target SD used on the
Levy-Jennings charts.

44. 44
TEa from Process SD or CV values – 2
TEa from Process SD or CV values – 2
z This is the lower limit for TEa. It defines what is
routinely attainable by your method.
z Keep in mind that this is what is attainable. Whether
it is the ideal value is another issue.

45. 45
Establishment Definitions
Establishment Definitions
z Medical Requirements
z Biological Variation
z Reference Interval
z Regulatory Requirements (CLIA ’88 or EQAS)
z Achievable Error
– Peer Group Survey Results
– Process SD
– CLSI (formerly NCCLS) EP 21
z Manufacturer’s Claim for Method

46. 46
CLSI EP21
CLSI EP21
z This document defines an approach similar to an
EP9 method comparison experiment to calculate a
Total Achievable Error.
z If you are interested in this approach, you may buy a
spreadsheet implementing this document from CLSI
for $350.

47. 47
Establishment Definitions
Establishment Definitions
z Medical Requirements
z Biological Variation
z Reference Interval
z Regulatory Requirements (CLIA ’88 or EQAS)
z Achievable Error
– Peer Group Survey Results
– Process SD
– CLSI (formerly NCCLS) EP 21
z Manufacturer’s Claim for Method

48. 48
TEa from Manufacturer’s Claims
TEa from Manufacturer’s Claims
z Calculate TEa from manufacturer’s specifications for
accuracy and precision.
z This approach is to be used only when
nothing else is available.

49. 49
Calculation Details
Calculation Details
z We will show calculation details for several data
sources.
z Issue:
– What multiplier factor (Z score) should be used?
TEa = SE + (Z * RE)
• What confidence level should it reference?
Some possibilities are 95%, 99.7%, 99.9997%.
• Traditionally, our industry has used 95% confidence
which corresponds to 50,000 errors per million. Six
Sigma (99.9997%) places that rate at 3 errors per
million.

50. 50
TEa from Medical Requirements
TEa from Medical Requirements
z If the Medical Requirement (MedReq) specifications
include both an accuracy component (SE) and a
precision component (RE), use the formula
TEa = SE + (2 * RE)
z If the specifications only include a precision
component, use the formula
TEa = 3 * RE
– This is not as satisfactory as the first, but often those
who create the specifications neglect the accuracy
component.

51. 51
BV Calculations
BV Calculations
z The general equation is:
TEa = f * ((0.125 * B) + (1.65 * 0.25 * P))
• Where B and P represent Bias and Precision components
respectively. B and P are calculated from intra- and inter-
individual biological variation.
z 3 specifications:
– Optimal (f= 1)
– Desirable (f= 2) (Quoted on Westgard’s site)
– Minimal. (f= 3)

52. 52
Error Profile
Error Profile
z Definition: The amount of random error observed or
expected over the analytical range of an instrument.
z It is expected that the amount of error will vary as the
concentration varies.

53. 53
Idealized Error Profile
Idealized Error Profile
0
5
10
15
20
25
30
10 100 1000
Concentration
SD
0
5
10
15
20
25
CV
SD
CV

54. 54
Idealized Error Profile
Idealized Error Profile
0
5
10
15
20
25
30
10 100 1000
Concentration
SD
0
5
10
15
20
25
CV
SD
CV
Concentration Percent

55. 55
TEa from Peer Group Surveys – 2
TEa from Peer Group Surveys – 2
z For each of a series of specimens for a given
instrument, calculate the CV. Want the mean result
to be in the mid- to upper-range for that analyte.
Determine the median result.
TEa = 3 * (median CV)
– Use this approach to calculate the percent component
of TEa.

56. 56
TEa from External QC results – 3
TEa from External QC results – 3
z Sometimes it is important to determine the
concentration component of TEa. In that case,
determine the median or representative SD for
results near the low end of the reportable range.
TEa = 3 * median SD

57. 57
TEa from External QC –
CAP Survey Example
TEa from External QC –
CAP Survey Example
HCG (VITROS ECi)
n=63
Spec ID Mean SD CV
C-11 26.97 1.65 6.1
C-12 68.29 4.54 6.6
C-13 90.61 6.39 7.1
C-14 52.13 3.57 6.8
C-15 82.47 4.84 5.9
Median
CV

58. 58
Error Profile – Plot of CAP Survey Data
Error Profile – Plot of CAP Survey Data
HCG (VITROS ECi)
0
2
4
6
8
10
1 10 100 1000
Concentration (mIU/mL)
SD
0.0
2.0
4.0
6.0
8.0
10.0
CV
SD
Range
CV
Reportable Range

59. 59
TEa from External QC – Example
TEa from External QC – Example
NYS PT Results - 2002-04
0
2
4
6
8
10
12
14
16
18
1 10 100 1000
HCG (mIU/mL)
CV
or
SD
SD
% CV
1.6 units 5.6%

60. 60
TEa from External QC – Example
TEa from External QC – Example
z From CAP Results
– TEa = 3 * Median CV = 3 * 6.6% = 19.8% = 20%
z From NYS Results
– TEa = 3 * Median CV = 3 * 5.6% =16.8% = 17%
– TEa = 3 * Median SD = 3 * 1.6 = 4.8 units
TEa = 5 units or 17% whichever is greater

61. 61
TEa from Manufacturer’s Claims
TEa from Manufacturer’s Claims
z From published specifications for an instrument or
method.
TEa = SE + (1.65 * RE) (95% confidence)
TEa = 3 * RE (when SE is not specified)

62. 62
Calculating TEa – Rigorous approach
Calculating TEa – Rigorous approach
1. Calculate minimum and maximum TEa’s.
2. Calculate TEa from the available resource highest
on the list.
3. If selected TEa (sTEa) is between the minimum and
maximum values then use it.
– If sTEa is < minimum, use the minimum value.
– If sTea is > maximum, use the maximum value.

63. 63
Calculating TEa – Easier Approach
Calculating TEa – Easier Approach
z If analyte has nationally specified medical
requirements, use them!!
z Otherwise, use TEa’s based on either CLIA PT limits
or PGS whichever is smaller.

64. 64
Calculating the Minimum TEa
Calculating the Minimum TEa
z The absolute minimum TEa
TEa = 3.0 * process SD or CV

65. 65
Calculating the Maximum TEa
Calculating the Maximum TEa
z TEa should never exceed 50% of the RI or the CLIA
’88 PT limits whichever is smaller.

66. 66
Our Data Sources
Our Data Sources
z Biological Variation (2 sources)
– Ricos et al (Westgard’s website) 270 tests
– Lacher et al (CCJ paper – Feb, 2005) 42 tests
z External QC
– Used NY State PT results for the instrument with the highest
population for a given test.
– TEa was calculated as 3 * median CV across all 5 specimens.
We used results from mid- to upper- range specimens.
z Process CV
– Object was to obtain results which could be reasonably expected
from a responsible clinical laboratory.
– Median of monthly QC results for 1 - 10 Canadian labs over a
period of more than 1 year. Data shown here are for the mid- or
high- level control.

67. 67
Case Study: ALT
Case Study: ALT
z Medical Requirements: n/a
z Biological Variation:
– TEa: 32%
z Reference Interval:
– 10 to 50 IU/L
z CLIA ’88 PT Limits: 20%
z 3 * Peer CV: 4.6% (NYS PT Survey)
z Process SD: 1.6%

68. 68
ALT TEa - Calculation
ALT TEa - Calculation
z Maximum TEa calculation
– 50% of RI = (40 / 2) = 20 units or 67% at 30 units
– CLIA ’88 PT limit: 20%
– Since CLIA ’88 limit is smaller,
• TEa (max) = 20%
z Minimum TEa calculation
– TEa (min) = Process CV * 3
= 3 * 1.6% = 4.8%.
z BV TEa: 32% (greater than max TEa)
z Set TEa to: 20%

69. 69
Case Study: Calcium
Case Study: Calcium
z Medical Requirements: n/a
z Biological Variation:
– TEa: 2.4%
z Reference Interval: 8.9 – 10.1 mg/dL (Recent)
z CLIA ’88 PT Limits: 1.0 mg/dL (10.6% at 9.5 mg/dL)
z 3 * Peer CV: 7% (NYS PT Survey)
z Process SD: 1.1%

70. 70
Calcium TEa - Calculation
Calcium TEa - Calculation
z TEa (max) = 50% of RI or PT value
50% of RI = 0.6 mg/dL (6.3%)
PT value = 1.0 mg/dL (10.5%)
z TEa (min) = 3 * Process SD
= 3 * 1.1% = 3.3%
z BV Goals
– TEa = 2.4%
z Set point for TEa 3.3% (round up to 4%?)

71. 71
Case Study – Total Protein
Case Study – Total Protein
z Medical Requirement: n/a
z CLIA ’88 PT Limit: 10%
z BV TEa: 3.4%
z Reference interval: 6.7 to 8.1 g/dL
z 3 * Peer CV: 5.3%
z 3 * Process SD: 3.8%

72. 72
Total Protein – Calculations (Rigorous)
Total Protein – Calculations (Rigorous)
z TEa (max) = 50% of RI or PT limit
50% of RI = (8.1 – 6.7)/2 =1.4/2 = 0.7
= 0.7 / 7.4 = 9.5%
PT limit = 10%
z TEa (min) = 3 * Process CV
= 3.8%
z BV TEa: = 3.4%
z BV TEa is < TEa (min)
Set TEa to: 3.8%. Maybe round to 4.0%.

73. 73
Total Protein – Calculations (Easier)
Total Protein – Calculations (Easier)
z Medical Reqs: n/a
z CLIA PT Limit: 10%
z 3 * PGS: 5.3%
z Set TEa to: 5.3%
– Round up to: 6%

74. 74
Final Thoughts
Final Thoughts
z My object in this exercise has been to start a
discussion on the approach that should be taken to
defining performance standards for the general case.
z I make no claim that what I propose here is the final
answer.
z Remember each lab serves its own customers, so
the clinical use to which its data is used may be
different from yours.
z I welcome your ideas and discussion.
z I trust that our collective wisdom will produce a far
better solution than any one of us.

75. 75
Discussion
Discussion
z To discuss this presentation, send email to
dgrhoads@dgrhoads.com
z If there is sufficient interest, we will set up a
discussion forum on our web site.