1. © TNS
Statistical Significance and Margin of Error
in TRI*M Studies

2. © TNS
Statistical Significance and Margin of Error
in TRI*M Studies
2
Generally, all standard rules for statistical significance also apply to TRI*M studies. In order to
determine the level of statistical margin the following parameters have to be taken into
account:
 Sample size
 Customized sample designs (e.g. stratified sampling of different customer segments)
 Size of universe
 Standard deviation
 Significance level
The mathematical calculation of the statistical margin of error for the TRI*M Index for various
sample sizes is based on:
 Normal distribution of TRI*M Index
 Infinite universe
 Significance level of 95%, 90% and 80%
 The challenge is to determine the effect of high quality sample designs. We therefore
calculate the standard deviation from all TRI*M Indices recorded in the TRI*M database. All
these scores are based on highly customized sample designs of our customer research
programs

3. © TNS
Statistical Significance and Margin of Error
in TRI*M Studies
3
The standard deviations for the following error rates for the TRI*M Index are calculated from
the TRI*M Database that includes more than 17,000 TRI*M Customer Retention studies with
more than 15 Mio. TRI*M Customer Retention interviews for 1,600+ companies.
 Most importantly, in order to apply these error rates to TRI*M Index comparisons over time
it is absolutely essential to keep the sample design stable and robust from wave to wave
 Any changes in the design impact the index and error rates!
Sample Size Confidence Coefficient
95% 90% 80%
n = 30 +/- 10.7 +/- 9.0 +/- 7.0
n = 50 +/- 8.3 +/- 7.0 +/- 5.4
n = 75 +/- 6.8 +/- 5.7 +/- 4.4
n = 100 +/- 5.9 +/- 4.9 +/- 3.8
n = 200 +/- 4.2 +/- 3.5 +/- 2.7
n = 300 +/- 3.4 +/- 2.8 +/- 2.2
n = 500 +/- 2.6 +/- 2.2 +/- 1.7
Example:
In case of a sample size of
n=200, the „true“ TRI*M
Index lies with a probability
of 90% within a range of
+/- 3.5 points

4. © TNS
Significant vs. meaningful differences
4
Statistical significance:
In statistics, a result is called statistically significant if it is unlikely to have occurred by chance.
Statistical significance heavily depends on the number of respondents in a survey.
Example:
 In a survey with n=8,000 respondents, a difference in the TRI*M Index (e.g. between two waves of the
survey) of 0.5 points can be considered as significant on a 90% significance level.
 This means that with a probability of 90% (or more), the TRI*M Index in the two groups (e.g. waves)
really is different (= not equal).
 But a significance does not give any information about the extent of this difference.
Meaningful differences:
A difference in results is called meaningful if it is of (business) relevance for the company. The
question, whether a difference in the TRI*M Index is meaningful or not, is completely independent of the
number
of respondents.
In the example, the difference in the TRI*M Index of 0.5 would be statistically significant (= not occurred by
chance), but the difference is small enough to be utterly unimportant. The company will not perceive any
difference in customer behaviour.
For the TRI*M Index, a difference of 3 points can be considered as meaningful
The use of the word significance in statistics is different from the standard one, which
suggests that something is important or meaningful