This is a new hypothesis testing method that allows you to test whether something during a "treatment period" made a difference even in the absence of Controls. Even a sample of 1 with no Controls will work.

1. New Hypothesis Testing Method Guy Lion November 2009

7. Building the model I am measuring a variable Y that trended upward from 80 in January 2007 to 95 in August 2009. The most recent three months, to the right of the red dotted line, consist of the “treatment” period. The latter had an effect. And, its “Effect Size” is growing as the store is performing above trend.

8. Quantifying the Effect Size The Effect Size of the treatment for the first month (June 2009) is equal to the regressed coefficient 0.74, as the treatment bumped the Y value by that much. Using the standard error of the coefficient, the output builds a 95% confidence interval (0.42 to 1.05). The p value of this coefficient is 0%, so we are nearly 100% confident that this coefficient is different from zero.

10. Technical Appendix

11. Autocorrelation of residuals Time series often break a linear regression assumption that residuals are independent when they are not. Large positive errors follow large positive ones. And, large negative errors follow large negative ones. Autocorrelation of residuals does not affect the regression coefficients. But, it affects the standard error and confidence intervals around such coefficients. To fix this problem use the lagged (t-1) residual of the original regression as an extra variable in a second regression. This example does not exhibit autocorrelation of residuals since the Durbin Watson score is close to 2.0. The seasonality variable and the focus on dummy variables reduce the risk of running into this problem.

12. Heteroskedasticity Linear regression assumes the variance of the errors remains constant (homoskedasticity). Time series often exhibit a change in the variance of errors (heteroskedasticity). Just like autocorrelation of residuals, heteroskedasticity does not affect the regression coefficients. But, it affects the standard error and confidence intervals around the coefficients. To fix this issue, transform the Y variable into a % change or log. This example does not demonstrate heteroskedasticity. The trend variable and the focus on dummy variables reduce the risk of running into this situation. Also, the Y variable being leveled (it increases by only 20% from beginning to end of series) is unlikely to be heteroskedastic.