The field of detection and attribution has been growing for a couple of decades and has recently seen an increase in the quantity and sophistication of methods. The difficulty of comparing these methods has motivated the design of a simulation testbed. Such a testbed is difficult to develop
as it involves generating spatiotemporal fields with complex and flexible covariance structures that do not inherently favor any of the methods to be tested. The presented testbed has the ability to generate a wide class of isotropic and non-isotropic correlation matrices to simulate the climate variability. The forcing response fields are tunable, spatially correlated fields with adjustable signal-to-noise ratios. The flexibility of the simulation method allows us to replicate a
variety of climate model-like output in a testable setting. In addition to the methods used in the testbed, we present synthetic data for a few simulated climate scenarios and demonstrate the intended use of the testbed by comparing the performance of ordinary least squares (OLS) and
total least squares (TLS) on the synthetic data. This is joint work with Dorit Hammering, Alexis Hannart, and the SAMSI working group on Detection and Attribution.
5 charts on South Africa as a source country for international student recrui...
CLIM: Transition Workshop - Tunable Testbed for Detection and Attribution Methods - Nathan Lenssen, May 14, 2018
1. 1/23
Tunable Testbed for Detection and Attribution
SAMSI CLIM Workshop 2018
Nathan Lenssen 1 Dorit Hammerling 2 Alexis Hannart 3
1Columbia University, Lamont-Doherty Earth Observatory
2National Center for Atmospheric Resarch 3Ouranos
May 14, 2018
2. 2/23
Testbed Motivation
A flexible and tunable testbed will allow researchers working on detection
and attribution methods to:
Evaluate methods by comparing estimated and true parameter values
Performance scaling as a function of sample size/dimensionality
Simulate real-world scenarios to enable testbed results to represent
applications
Tunable variety of climate response patterns and climate variability
covariances
Determine robustness of methods through perturbations of testbed
parameters
Compare multiple D+A methods on variety of scenarios
3. 3/23
Roadmap of Presentation
(I) Generative Statistical Model for Detection and Attribution
Ordinary Least Squares
Error-in-Variable Formulation
Sources of Variability
(II) Major Testbed Components
Observed vs. Latent Data/Parameters
Simulation of Forced Responses
Covariances of Climate and Observational Variability
(III) Results and Applications
4. 4/23
Classical Formulation of Detection and Attribution
Ordinary Least Squares (OLS)
Observed Quantities:
y: The observed climate response of interest
X∗ The model-simulated forcing responses X∗ = (x∗
1 , . . . , x∗
m, . . . , x∗
M)
5. 4/23
Classical Formulation of Detection and Attribution
Ordinary Least Squares (OLS)
Observed Quantities:
y: The observed climate response of interest
X∗ The model-simulated forcing responses X∗ = (x∗
1 , . . . , x∗
m, . . . , x∗
M)
y = X∗
β + u
6. 4/23
Classical Formulation of Detection and Attribution
Ordinary Least Squares (OLS)
Observed Quantities:
y: The observed climate response of interest
X∗ The model-simulated forcing responses X∗ = (x∗
1 , . . . , x∗
m, . . . , x∗
M)
y = X∗
β + u
Statistical Parameter of Interest:
β Estimation of β > 0 provides detection, inference (CIs) gives us
attribution
Climate Variability:
u Error due to climate variability where u ∼ N(0, C)
C Estimated through model control runs
7. 5/23
Statistical Formulation of Detection and Attribution
Error-in-Variable (EIV)
Observed Quantities:
y The climate response of interest
X The noisy responses to forcings X = (x1, . . . , xm, . . . , xM)
8. 5/23
Statistical Formulation of Detection and Attribution
Error-in-Variable (EIV)
Observed Quantities:
y The climate response of interest
X The noisy responses to forcings X = (x1, . . . , xm, . . . , xM)
y = X∗
β + uy
X = X∗
+ U
9. 5/23
Statistical Formulation of Detection and Attribution
Error-in-Variable (EIV)
Observed Quantities:
y The climate response of interest
X The noisy responses to forcings X = (x1, . . . , xm, . . . , xM)
y = X∗
β + uy
X = X∗
+ U
Latent Quantities:
y∗ The idealized climate response where y∗ = X∗β
X∗ The idealized responses to forcings X∗ = (x∗
1 , . . . , x∗
M)
Climate Variability:
uy As OLS formulation with uy ∼ N(0, C)
U The error on the forcing responses due to climate variability
U = (u1, . . . , uM)
iid
∼ N(0, C)
10. 6/23
Statistical Formulation of Detection and Attribution
Observed Response Variability
The incomplete expression for the climate response y is
y = X∗
β + uy , uy ∼ N(0, C)
We propose three different y states to fully incorporate all of the sources
of variability.
y∗ The idealized climate response (latent) y∗ = X∗β
yrel The realized climate response (latent) yrel = y∗ + uY
yobs The observed climate response yobs = yrel + εY
With observational error εY ∼ N(0, W)
yrel = X∗
β + uy uy ∼ N(0, C)
yobs = yrel + εy εy ∼ N(0, W)
11. 7/23
Statistical Formulation of Detection and Attribution
Full Model
Full Error-in-Variable Model:
yobs = yrel + εy εy ∼ N(0, W)
yrel = X∗
β + uy uy ∼ N(0, C)
X = X∗
+ L−1/2
U um
iid
∼ N(0, C)
12. 7/23
Statistical Formulation of Detection and Attribution
Full Model
Full Error-in-Variable Model:
yobs = yrel + εy εy ∼ N(0, W)
yrel = X∗
β + uy uy ∼ N(0, C)
X = X∗
+ L−1/2
U um
iid
∼ N(0, C)
Scale-Variant Error-in-Variable Model:
yobs = yrel + εy εy ∼ N(0, W)
yrel = X∗
β + uy uy ∼ N(0, α−1
C)
X = X∗
+ L−1/2
U um
ind
∼ N(0, γ−1
m C)
13. 8/23
Roadmap of Presentation
(I) Generative Statistical Model for Detection and Attribution
Ordinary Least Squares
Error-in-Variable Formulation
Sources of Variability
(II) Major Testbed Components
Observed vs. Latent Data/Parameters
Simulation of Forced Responses
Covariances of Climate and Observational Variability
(III) Results and Applications
14. 9/23
Data and Parameters of Interest
yobs = yrel + εy εy ∼ N(0, W)
yrel = X∗
β + uy uy ∼ N(0, α−1
C)
X = X∗
+ L−1/2
U um
ind
∼ N(0, γ−1
m C)
Observed Objects:
X Observed forcing response ensembles
X = (x1, . . . , xM)
= [x
(1)
1 , . . . , x
(L1)
1 ], . . . , [x
(1)
M , . . . , x
(LM )
M ]
Yobs Observed climate response ensemble
Yobs = (y
(1)
obs, . . . , y
(Ly )
obs )
X0 Control runs from the climate model used in forcing response
experiments
X0 = (X
(1)
0 , . . . , X
(L0)
0 )
15. 10/23
Data and Parameters of Interest
yobs = yrel + εy εy ∼ N(0, W)
yrel = X∗
β + uy uy ∼ N(0, α−1
C)
X = X∗
+ L−1/2
U um
ind
∼ N(0, γ−1
m C)
Latent Objects
θ The statistical parameters in the model
θ = {β, α, γ, C, W}
X∗ True forcing response X = (x1, . . . , xM)
y∗ True climate response y∗ = Xβ
yrel Realized climate response
16. 11/23
Simulation Procedure
yobs = yrel + εy εy ∼ N(0, W)
yrel = X∗
β + uy uy ∼ N(0, α−1
C)
X = X∗
+ L−1/2
U um
ind
∼ N(0, γ−1
m C)
For a simulation of fixed dimensionality:
1) Set/Simulate fixed objects θ = {β, α, γ, C, W} and X∗
X∗
, C, and W according to simulation modules
2a) Simulate the observed forcing response ensembles xm
x
( )
m
iid
∼ N(x∗
m, C)
2b) Simulate the realized climate response yrel
yrel ∼ N(X∗
β, C)
3) Simulate the observed climate response ensemble Yobs
y
( )
obs
iid
∼ N(yrel , W)
17. 12/23
(M1) True Forcing Response X∗
Simulated Mat´ern Patterns
Fields simulated with covariance
matrices according to the
Mat´ern covariance function
Flexible statistical model that
can be fit to replicate climate
fields
Simulate land-sea interface
Random generation of X∗
allows for robustness testing of
methods
19. 14/23
(M2) Climate Variability Covariance C
Exponential covariance function
too smooth and regular for
climate applications like
precipitation
Goal: Generate non-stationary,
non-isotropic covariance matrix
Issue: Difficult to create and
guarantee invertibility!
Solution: Modify the
eigenvalues of a decomposed
exponential covariance matrix
22. 17/23
Simulation Procedure
yobs = yrel + εy εy ∼ N(0, W)
yrel = X∗
β + uy uy ∼ N(0, α−1
C)
X = X∗
+ L−1/2
U um
ind
∼ N(0, γ−1
m C)
For a simulation of fixed dimensionality:
1) Set/Simulate fixed objects θ = {β, α, γ, C, W} and X∗
X∗
, C, and W according to simulation modules
2a) Simulate the observed forcing response ensembles xm
x
( )
m
iid
∼ N(x∗
m, C)
2b) Simulate the realized climate response yrel
yrel ∼ N(X∗
β, C)
3) Simulate the observed climate response ensemble Yobs
y
( )
obs
iid
∼ N(yrel , W)
25. 20/23
Simulation Procedure
yobs = yrel + εy εy ∼ N(0, W)
yrel = X∗
β + uy uy ∼ N(0, α−1
C)
X = X∗
+ L−1/2
U um
ind
∼ N(0, γ−1
m C)
For a simulation of fixed dimensionality:
1) Set/Simulate fixed objects θ = {β, α, γ, C, W} and X∗
X∗
, C, and W according to simulation modules
2a) Simulate the observed forcing response ensembles xm
x
( )
m
iid
∼ N(x∗
m, C)
2b) Simulate the realized climate response yrel
yrel ∼ N(X∗
β, C)
3) Simulate the observed climate response ensemble Yobs
y
( )
obs
iid
∼ N(yrel , W)
27. 22/23
Discussion and Next Steps
Current Testbed
Flexible, fast generation of EIV Detection and Attribution data
Non-isotropic climate variability
First Applications
Used in testing new method from Smith, Hammerling and Johnson
Next Steps
Fit forcing responses and sources of variability to real applications
Compare traditional OLS and TLS methods as a function of testbed
parameters
Evaluate models from [Hannart, 2016] and [Katzfuss et al., 2017]
Thank you/Input/Requests
28. 23/23
References
Hannart, A. (2016).
Integrated optimal fingerprinting: Method description and illustration.
Journal of Climate, 29(6):1977–1998.
Katzfuss, M., Hammerling, D., and Smith, R. L. (2017).
A bayesian hierarchical model for climate change detection and
attribution.
Geophysical Research Letters, 44(11):5720–5728.
2017GL073688.