This document summarizes a simulation study evaluating causal inference methods for assessing the effects of opioid and gun policies. The study used real US state-level data to simulate the adoption of policies by some states and estimated the effects using different statistical models. It found that with fewer adopting states, type 1 error rates were too high, and most models lacked power. It recommends using cluster-robust standard errors and lagged outcomes to improve model performance. The study aims to help identify best practices for policy evaluation studies.

1. Beth Ann Griffin
RAND Corporation
Editor, Annals of Applied Statistics

2. • Grateful for support from the
National Institute on Drug Abuse of
the NIH and Arnold Ventures
• All views and opinions expressed
are my own
• No conflicts of interest
• Colleagues I would like to acknowledge
• Elizabeth Stuart, PhD
• Megan Schuler, PhD
• Andrew Morral, PhD
• Terry Schell, PhD
• Rosalie Liccardo Pacula, PhD
• Bradley D. Stein, MD, PhD
• Mary Vaiana, PhD
• Stephen Patrick, MD PhD
• Elizabeth McNeer, MS
• Rosanna Smart, PhD
• David Powell, PhD
• Matt Cefalu, PhD

3. • Why assessing state opioid and gun policies is so challenging
• Our approach to evaluate performance of methods
• Key simulation results & their implications for practice
• Need for new methods & better dissemination of new methods

4. • Selection bias: States that choose to adopt policies may differ from
states that do not
• Sparse data: Policies may be adopted by a limited number of states,
or adopted recently (limited number of post-policy observations)
• Policy heterogeneity: States may implement related, but distinct,
versions of a specific policy
• Concurrent policies: States may have multiple policies targeting
opioids / guns

5. • Use counterfactual approach:
Compare what actually happened
in a state with what we estimate
would have happened without the
policy change

6. 0
10
20
30
40
50
60
2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
Year
Opioid policy evaluation studies published annually

7. • 85% longitudinal design vs 15% cross-sectional design
• 70% multi-state design vs 30% single-state design
• Multi-state longitudinal studies (n=79):
• 45% Difference-in-difference design
• 30% Comparative interrupted time series
• 15% GEE

8. • Strategy:
• Use simulation studies on existing data to identify the best causal
inference methods for assessing the effects of opioid and gun policies
• Tactic:
• Using existing data (state-level for now), simulate the effect of a policy
on outcomes, and determine which method(s) most accurately detect
the effect

9. • Longitudinal, repeated measures data collected on an annual basis
• States are the units of interest, observations are clustered within state
• Analyses comprised all 50 states
• Staggered implementation of given policy across “treatment” states

10. Simulate policy
effect in real
data
Estimate effect
with statistical
models
Compare
model
performance
Generated simulated
dataset by augmenting
real state-level data;
N = 50 states
Compared 17 different
statistical models under
multiple conditions (5,000
replications each)
Four performance
measures: Type 1 error
rates (“false positives”),
power, bias, root mean
squared error (RMSE)

11. Opioid Deaths
(per 100,000
population)
Simulate Estimate Compare
1. Real U.S. state opioid-related death rates

12. Opioid Deaths
(per 100,000
population)
Simulate Estimate Compare
1. Real U.S. state opioid-related death rates
2. Randomly select 5 states

13. Opioid Deaths
(per 100,000
population)
Simulate Estimate Compare
1. Real U.S. state opioid-related death rates
2. Randomly select 5 states
3. Randomly select policy implementation
date

14. Opioid Deaths
(per 100,000
population)
Simulate Estimate Compare
1. Real U.S. state opioid-related death rates
2. Randomly select 5 states
3. Randomly select policy implementation
date
4. Introduce policy effect after
implementation date

15. GLM
Regression
specification Weighting Standard error adj
1 Linear Fixed effects (FE) Population Weighted None, Huber, Cluster
2 Unweighted None, Huber, Cluster
3 FE + Detrended Population Weighted None, Huber, Cluster
4 Unweighted None, Huber, Cluster
5 Autoregressive Population Weighted None, Huber, Cluster
6 Unweighted None, Huber, Cluster
7 GEE model Population Weighted NA
8 Unweighted NA
9 Log-linear Fixed effects (FE) Population Weighted None, Huber, Cluster
10 Unweighted None, Huber, Cluster
11 Autoregressive Population Weighted None, Huber, Cluster
12 Unweighted None, Huber, Cluster
13
Negative
Binomial
Fixed effects (FE) Unweighted None, Huber, Cluster
14 FE + Detrended Unweighted None, Huber, Cluster
15 Autoregressive Unweighted None, Huber, Cluster
16 Poisson Fixed effects (FE) Unweighted None, Huber, Cluster
17 Autoregressive Unweighted None, Huber, Cluster

17. Simulation Project
0
0.1
0.2
0.3
0.4
0.5
No Adjustment Huber Cluster
1 5 15 30
Goal: Models
with Type 1
error = 0.05
Type 1
Error Rate
for Linear
Two-Way
Fixed Effects
Model with
Population
Weights
Number of States Implementing Policy

18. 25% effect
5% effect
15% effect
Power
Number of States Implementing the Policy

19. Simulation Project
0.00 0.05 0.10 0.15 0.20 0.25
Linear 2-way FE Wted
Linear GEE Wted
Log Y 2-way FE Wted
Linear 2-way FE Unwt
Linear Detrended Wted
Poisson 2-way FE
Linear Detrended Unwt
Log Y AR Unwt
Log Y AR Wted
Poisson AR
Linear GEE Unwt
Log Y 2-way FE Unwt
Negative Binomial 2-way FE
Negative Binomial AR
Linear AR Unwt
Negative Binomial Detrended
Linear AR Wted
Power

20. 0
0.1
0.2
0.3
0.4
0.5
Power
Instant effect 3-years till fully effective

21. • Type I error rates are unreasonably high when number of states
implementing new policy is low (< 15)
• Caution needed when such studies report “statistically significant”
findings - may be a false positive (e.g., saying a law as an effect when it
truly does not)
• Critical to use cluster adjustments to standard errors when using state
and year fixed effects in linear or log-linear models
• Recommend use of a correction factor to ensure Type I error = 0.05
• Power is very low for all models; need to find better approaches to account
for level of uncertainty
• Use of lagged outcomes as control covariates is helpful in the linear
model
• Use of negative binomial link performs better than Poisson

22. • State-policy evaluations need new methods
• Many state-of-the-art methods coming from statisticians, economists
and other methodologists
• Application of new methods typically lags their development
• Need more effective dissemination strategies to get new methods
into the hands of the broader scientific community (not just
methodologists)
• Use Shiny to help, teach courses/workshops at conferences, create
websites, actively disseminate your work

23. • Understanding how theory performs in practice is essential
• Room for methods development where applications are at the core
• Annals of Applied Statistics (AOAS) – perfect home for papers tackling
these issues – we want
• Papers that include innovative methodology brought to bear on
scientific/policy questions and relevant data
• Groundbreaking application of state-of-the-art methods

24. bethg@rand.org

26. Difference in difference (DID) method
Classic 2-way fixed effects DID specification:
𝑔(𝑌𝑖𝑡) = 𝛼 ∙ 𝐴𝑖𝑡 + 𝜷 ∙ 𝑿𝑖𝑡 + 𝜌𝒊 + 𝜎𝑡 + 𝜀𝑖𝑡
• State fixed effects (𝜌𝑖): baseline differences across states
• Time fixed effects (𝜎𝑡): temporal national trends
Detrended model
• Include state-specific linear trends

27. Autoregressive Model
• Include one-period lagged autoregressive (AR) model
Generalized Estimating Equation (GEE) Model
• Semi-parametric method that requires specification of the covariance
matrix for within-subject observations

29. Simulation Project
Bias = Total Count of Deaths by which model over or estimated effect when true effect = 5% (or 700 deaths)
-100.00 0.00 100.00 200.00 300.00 400.00 500.00 600.00 700.00
Linear AR; Population Wted
Linear AR; Unwted
Linear Detrended; Population Wted
Linear Two-Way FE; Unwted
Linear GEE; Weighhted
Linear Two-Way FE; Population Wted
Linear GEE; Unweighted
Linear Detrended; Unwted
Log-Y Two-Way FE; Unwted
Negative Binomial; Two-Way FE
Negative Binomial; Detrended
Poisson; Two-Way FE
Log-Y Two-Way FE; Population Wted
Negative Binomial; AR
Log-Y AR; Population Wted
Log-Y AR; Unwted
Poisson; AR

30. 0.00 0.20 0.40 0.60 0.80 1.00 1.20 1.40 1.60 1.80 2.00
Linear GEE; Weighhted
Linear Two-Way FE; Population Wted
Linear Detrended; Population Wted
Linear Two-Way FE; Unwted
Linear Detrended; Unwted
Linear GEE; Unweighted
Linear AR; Unwted
Linear AR; Population Wted
Poisson; AR
Log-Y AR; Population Wted
Log-Y AR; Unwted
Negative Binomial; AR
Log-Y Two-Way FE; Population Wted
Poisson; Two-Way FE
Negative Binomial; Detrended
Log-Y Two-Way FE; Unwted
Negative Binomial; Two-Way FE
Simulation Project
Root Mean Square Error in Model where True Policy Effect = 0

31. 0
0.1
0.2
0.3
0.4
0.5
Power
Instant effect
3-years till fully effective
3-years till fully effective BUT model uses an instant effect
PROBLEM:
The field is tending to
do this.

33. Inclusion criteria:
1) Must be study estimating the impact of a relevant policy on
opioid-related outcomes
2) Restricted focus to state or federal level policies
• excluded local, hospital-level initiatives
3) Published during 2005 – 2018
4) U.S. studies only

34. • Literature review entailed structured extraction regarding details of
study population, study period, analytical design, data sources, etc.
• 146 studies met inclusion criteria
• Created taxonomy to systematically classify:
• Opioid-related policies
• Opioid-related outcomes

35. Prescription Drug
Monitoring
Program
31%
Other
Prescribing
Policies
17%
Treatment
Access &
Utilization
12%
Drug Formulation or
Schedule Changes
12%
Marijuana
Laws
9%
Pill Mill/Pain
Clinic Laws
8%
Overdose
Prevention
8%
Criminalizing
Drug Use
3%

37. • Studies primarily examined proximal outcomes (e.g., PDMPs and opioid
prescribing)
• Some considered more distal outcomes (e.g., marijuana laws &
opioid-related mortality) or unintended consequences (e.g., PDMPs
& heroin use)
• Studies are increasingly accounting for policy heterogeneity: Nearly 50%
examined or controlled for policy subcomponents / dimensions
• 60% adjusted for co-occurring policies (not necessarily
comprehensively)