1. Statistics for usability tests with zero response
Zero response
If n tested items of the LOT show a zero response, i.e. zero non-conforming items are found,
a probability exists that non-conforming items are still in the LOT. For n/M ≤ 0.05, i.e. large
LOT size compared to the sample size, the binomial distribution can be used to calculate the
upper limit of non-conforming items for a given confidence level C: [1]
P(k) =
n
k
pk
(1 − p)n−k
(1)
with n the sample size, k the number of non-conforming items and p the fraction of
non-conforming items in a LOT. For k = 0:
P(0) = (1 − p)n
. (2)
One can ask for the largest p the makes P(0) reasonably small. Therefore, we set
P(0) = 1 − C, where C is the confidence level for the upper limit of non-conforming items pu:
1 − C = (1 − pu)n
(3)
or
pu = 1 − n
√
1 − C (4)
Table 1: Maximal fraction of non-conformance items in a LOT in percent as a function of
samples n taken and zero non-compliance items k are found for confidence intervals
between 75% and 99%.
Samples Upper limit in percent
n C: 0.75 0.90 0.95 0.975 0.99
1 75.00 90.00 95.00 97.50 99.00
3 37.00 53.58 63.16 70.76 78.46
5 24.21 36.90 45.07 52.18 60.19
7 17.97 28.03 34.82 40.96 48.21
9 14.28 22.57 28.31 33.63 40.05
11 11.84 18.89 23.84 28.49 34.21
17 7.83 12.67 16.16 19.51 23.73
21 6.39 10.38 13.29 16.11 19.69
25 5.39 8.80 11.29 13.72 16.82
Sometimes it’s very convienient to calculate the sample size n for a given confidence level C
and a upper fraction of non-conforming items 1 − C. This leads to
Cn
+ C − 1 = 0. (5)
Solved for n:
n =
ln(1 − C)
ln(C)
. (6)
Title: Usability stats
Revision: 0
Effective: 2007-10-25
Page 1 of 3
ID-Number:
Author: Dr. Peter Drechsler
Developed: 2007-10-25
Phone: +49 89 5008-4163
Fax: +49 89 5008-4163
peter.drechsler@tuev-sued.de
TÜV R
TÜV SÜD Product Service GmbH
Ridlerstr. 65
80339 Munich
Germany
2. Statistics for usability tests with zero response
0.5 0.6 0.7 0.8 0.9 1.0
0.00.10.20.30.4 n = 11
Confidence level
Fractionofnon−conformingitems
Figure 1: Confidence level versus upper fraction of non-conforming items for n = 11.
References
[1] ASTM E 2334. Standard practice for setting an upper confidence bound for a fraction or
number of non-conforming items, or a rate of occurence for non-conformities, using
attribute data, when there is a zero response in the sample, 2003.
Title: Usability stats
Revision: 0
Effective: 2007-10-25
Page 2 of 3
ID-Number:
Author: Dr. Peter Drechsler
Developed: 2007-10-25
Phone: +49 89 5008-4163
Fax: +49 89 5008-4163
peter.drechsler@tuev-sued.de
TÜV R
TÜV SÜD Product Service GmbH
Ridlerstr. 65
80339 Munich
Germany
3. Statistics for usability tests with zero response
Table 2: Sample size n for a given confidence level C.
C/% pu n
99.99 0.0001 92099
99.90 0.0010 6904
99.5 0.0050 1057
99 0.0100 458
97 0.0300 115
95 0.0500 58
93 0.0700 37
90 0.1000 22
70 0.3000 4
50 0.5000 1
Title: Usability stats
Revision: 0
Effective: 2007-10-25
Page 3 of 3
ID-Number:
Author: Dr. Peter Drechsler
Developed: 2007-10-25
Phone: +49 89 5008-4163
Fax: +49 89 5008-4163
peter.drechsler@tuev-sued.de
TÜV R
TÜV SÜD Product Service GmbH
Ridlerstr. 65
80339 Munich
Germany
![Statistics for usability tests with zero response
Zero response
If n tested items of the LOT show a zero response, i.e. zero non-conforming items are found,
a probability exists that non-conforming items are still in the LOT. For n/M ≤ 0.05, i.e. large
LOT size compared to the sample size, the binomial distribution can be used to calculate the
upper limit of non-conforming items for a given confidence level C: [1]
P(k) =
n
k
pk
(1 − p)n−k
(1)
with n the sample size, k the number of non-conforming items and p the fraction of
non-conforming items in a LOT. For k = 0:
P(0) = (1 − p)n
. (2)
One can ask for the largest p the makes P(0) reasonably small. Therefore, we set
P(0) = 1 − C, where C is the confidence level for the upper limit of non-conforming items pu:
1 − C = (1 − pu)n
(3)
or
pu = 1 − n
√
1 − C (4)
Table 1: Maximal fraction of non-conformance items in a LOT in percent as a function of
samples n taken and zero non-compliance items k are found for confidence intervals
between 75% and 99%.
Samples Upper limit in percent
n C: 0.75 0.90 0.95 0.975 0.99
1 75.00 90.00 95.00 97.50 99.00
3 37.00 53.58 63.16 70.76 78.46
5 24.21 36.90 45.07 52.18 60.19
7 17.97 28.03 34.82 40.96 48.21
9 14.28 22.57 28.31 33.63 40.05
11 11.84 18.89 23.84 28.49 34.21
17 7.83 12.67 16.16 19.51 23.73
21 6.39 10.38 13.29 16.11 19.69
25 5.39 8.80 11.29 13.72 16.82
Sometimes it’s very convienient to calculate the sample size n for a given confidence level C
and a upper fraction of non-conforming items 1 − C. This leads to
Cn
+ C − 1 = 0. (5)
Solved for n:
n =
ln(1 − C)
ln(C)
. (6)
Title: Usability stats
Revision: 0
Effective: 2007-10-25
Page 1 of 3
ID-Number:
Author: Dr. Peter Drechsler
Developed: 2007-10-25
Phone: +49 89 5008-4163
Fax: +49 89 5008-4163
peter.drechsler@tuev-sued.de
TÜV R
TÜV SÜD Product Service GmbH
Ridlerstr. 65
80339 Munich
Germany](https://image.slidesharecdn.com/5e04d42c-de95-4245-95fe-9d59d067c81e-160117100610/85/zero_response-1-320.jpg)
![Statistics for usability tests with zero response
0.5 0.6 0.7 0.8 0.9 1.0
0.00.10.20.30.4 n = 11
Confidence level
Fractionofnon−conformingitems
Figure 1: Confidence level versus upper fraction of non-conforming items for n = 11.
References
[1] ASTM E 2334. Standard practice for setting an upper confidence bound for a fraction or
number of non-conforming items, or a rate of occurence for non-conformities, using
attribute data, when there is a zero response in the sample, 2003.
Title: Usability stats
Revision: 0
Effective: 2007-10-25
Page 2 of 3
ID-Number:
Author: Dr. Peter Drechsler
Developed: 2007-10-25
Phone: +49 89 5008-4163
Fax: +49 89 5008-4163
peter.drechsler@tuev-sued.de
TÜV R
TÜV SÜD Product Service GmbH
Ridlerstr. 65
80339 Munich
Germany](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)