SlideShare a Scribd company logo
1 of 187
Download to read offline
Several perspectives and solutions on Bayesian
testing of hypotheses
Christian P. Robert
Universit´e Paris-Dauphine, Paris & University of Warwick, Coventry
Outline
Significance tests: one new parameter
Noninformative solutions
Jeffreys-Lindley paradox
Deviance (information criterion)
Testing under incomplete information
Testing via mixtures
“Significance tests: one new parameter”
Significance tests: one new parameter
Bayesian tests
Bayes factors
Improper priors for tests
Conclusion
Noninformative solutions
Jeffreys-Lindley paradox
Deviance (information criterion)
Testing under incomplete information
Testing via mixtures
Fundamental setting
Is the new parameter supported by the observations or is
any variation expressible by it better interpreted as
random? Thus we must set two hypotheses for
comparison, the more complicated having the smaller
initial probability (Jeffreys, ToP, V, §5.0)
...compare a specially suggested value of a new
parameter, often 0 [q], with the aggregate of other
possible values [q ]. We shall call q the null hypothesis
and q the alternative hypothesis [and] we must take
P(q|H) = P(q |H) = 1/2 .
Construction of Bayes tests
Definition (Test)
Given an hypothesis H0 : θ ∈ Θ0 on the parameter θ ∈ Θ0 of a
statistical model, a test is a statistical procedure that takes its
values in {0, 1}.
Type–one and type–two errors
Associated with the risk
R(θ, δ) = Eθ[aL(θ, δ(x))]
=
Pθ(δ(x) = 0) if θ ∈ Θ0,
Pθ(δ(x) = 1) otherwise,
Theorem (Bayes test)
The Bayes estimator associated with π and with the 0 − 1 loss is
δπ
(x) =
1 if P(θ ∈ Θ0|x) > P(θ ∈ Θ0|x),
0 otherwise,
Type–one and type–two errors
Associated with the risk
R(θ, δ) = Eθ[aL(θ, δ(x))]
=
Pθ(δ(x) = 0) if θ ∈ Θ0,
Pθ(δ(x) = 1) otherwise,
Theorem (Bayes test)
The Bayes estimator associated with π and with the 0 − 1 loss is
δπ
(x) =
1 if P(θ ∈ Θ0|x) > P(θ ∈ Θ0|x),
0 otherwise,
Jeffreys’ example (§5.0)
Testing whether the mean α of a normal observation is zero:
P(q|aH) ∝ exp −
a2
2s2
P(q dα|aH) ∝ exp −
(a − α)2
2s2
f (α)dα
P(q |aH) ∝ exp −
(a − α)2
2s2
f (α)dα
A (small) point of contention
Jeffreys asserts
Suppose that there is one old parameter α; the new
parameter is β and is 0 on q. In q we could replace α by
α , any function of α and β: but to make it explicit that
q reduces to q when β = 0 we shall require that α = α
when β = 0 (V, §5.0).
This amounts to assume identical parameters in both models, a
controversial principle for model choice or at the very best to make
α and β dependent a priori, a choice contradicted by the next
paragraph in ToP
A (small) point of contention
Jeffreys asserts
Suppose that there is one old parameter α; the new
parameter is β and is 0 on q. In q we could replace α by
α , any function of α and β: but to make it explicit that
q reduces to q when β = 0 we shall require that α = α
when β = 0 (V, §5.0).
This amounts to assume identical parameters in both models, a
controversial principle for model choice or at the very best to make
α and β dependent a priori, a choice contradicted by the next
paragraph in ToP
Orthogonal parameters
If
I(α, β) =
gαα 0
0 gββ
,
α and β orthogonal, but not [a posteriori] independent, contrary to
ToP assertions
...the result will be nearly independent on previous
information on old parameters (V, §5.01).
and
K =
1
f (b, a)
ngββ
2π
exp −
1
2
ngββb2
[where] h(α) is irrelevant (V, §5.01)
Orthogonal parameters
If
I(α, β) =
gαα 0
0 gββ
,
α and β orthogonal, but not [a posteriori] independent, contrary to
ToP assertions
...the result will be nearly independent on previous
information on old parameters (V, §5.01).
and
K =
1
f (b, a)
ngββ
2π
exp −
1
2
ngββb2
[where] h(α) is irrelevant (V, §5.01)
Acknowledgement in ToP
In practice it is rather unusual for a set of parameters to
arise in such a way that each can be treated as irrelevant
to the presence of any other. More usual cases are (...)
where some parameters are so closely associated that one
could hardly occur without the others (V, §5.04).
Generalisation
Theorem (Optimal Bayes decision)
Under the 0 − 1 loss function
L(θ, d) =



0 if d = IΘ0 (θ)
a0 if d = 1 and θ ∈ Θ0
a1 if d = 0 and θ ∈ Θ0
the Bayes procedure is
δπ
(x) =
1 if Prπ
(θ ∈ Θ0|x) a0/(a0 + a1)
0 otherwise
Generalisation
Theorem (Optimal Bayes decision)
Under the 0 − 1 loss function
L(θ, d) =



0 if d = IΘ0 (θ)
a0 if d = 1 and θ ∈ Θ0
a1 if d = 0 and θ ∈ Θ0
the Bayes procedure is
δπ
(x) =
1 if Prπ
(θ ∈ Θ0|x) a0/(a0 + a1)
0 otherwise
Bound comparison
Determination of a0/a1 depends on consequences of “wrong
decision” under both circumstances
Often difficult to assess in practice and replacement with “golden”
default bounds like .05, biased towards H0
Bound comparison
Determination of a0/a1 depends on consequences of “wrong
decision” under both circumstances
Often difficult to assess in practice and replacement with “golden”
default bounds like .05, biased towards H0
A function of posterior probabilities
Definition (Bayes factors)
For hypotheses H0 : θ ∈ Θ0 vs. Ha : θ ∈ Θ0
B01 =
π(Θ0|x)
π(Θc
0|x)
π(Θ0)
π(Θc
0)
=
Θ0
f (x|θ)π0(θ)dθ
Θc
0
f (x|θ)π1(θ)dθ
[Good, 1958 & ToP, V, §5.01]
Equivalent to Bayes rule: acceptance if
B01 > {(1 − π(Θ0))/a1}/{π(Θ0)/a0}
Self-contained concept
Outside decision-theoretic environment:
eliminates choice of π(Θ0)
but depends on the choice of (π0, π1)
Bayesian/marginal equivalent to the likelihood ratio
Jeffreys’ scale of evidence:
if log10(Bπ
10) between 0 and 0.5, evidence against H0 weak,
if log10(Bπ
10) 0.5 and 1, evidence substantial,
if log10(Bπ
10) 1 and 2, evidence strong and
if log10(Bπ
10) above 2, evidence decisive
Self-contained concept
Outside decision-theoretic environment:
eliminates choice of π(Θ0)
but depends on the choice of (π0, π1)
Bayesian/marginal equivalent to the likelihood ratio
Jeffreys’ scale of evidence:
if log10(Bπ
10) between 0 and 0.5, evidence against H0 weak,
if log10(Bπ
10) 0.5 and 1, evidence substantial,
if log10(Bπ
10) 1 and 2, evidence strong and
if log10(Bπ
10) above 2, evidence decisive
Self-contained concept
Outside decision-theoretic environment:
eliminates choice of π(Θ0)
but depends on the choice of (π0, π1)
Bayesian/marginal equivalent to the likelihood ratio
Jeffreys’ scale of evidence:
if log10(Bπ
10) between 0 and 0.5, evidence against H0 weak,
if log10(Bπ
10) 0.5 and 1, evidence substantial,
if log10(Bπ
10) 1 and 2, evidence strong and
if log10(Bπ
10) above 2, evidence decisive
Self-contained concept
Outside decision-theoretic environment:
eliminates choice of π(Θ0)
but depends on the choice of (π0, π1)
Bayesian/marginal equivalent to the likelihood ratio
Jeffreys’ scale of evidence:
if log10(Bπ
10) between 0 and 0.5, evidence against H0 weak,
if log10(Bπ
10) 0.5 and 1, evidence substantial,
if log10(Bπ
10) 1 and 2, evidence strong and
if log10(Bπ
10) above 2, evidence decisive
A major modification
When the null hypothesis is supported by a set of measure 0
against Lebesgue measure, π(Θ0) = 0 for an absolutely continuous
prior distribution
[End of the story?!]
Suppose we are considering whether a location parameter
α is 0. The estimation prior probability for it is uniform
and we should have to take f (α) = 0 and K[= B10]
would always be infinite (V, §5.02)
A major modification
When the null hypothesis is supported by a set of measure 0
against Lebesgue measure, π(Θ0) = 0 for an absolutely continuous
prior distribution
[End of the story?!]
Suppose we are considering whether a location parameter
α is 0. The estimation prior probability for it is uniform
and we should have to take f (α) = 0 and K[= B10]
would always be infinite (V, §5.02)
Point null refurbishment
Requirement
Defined prior distributions under both assumptions,
π0(θ) ∝ π(θ)IΘ0 (θ), π1(θ) ∝ π(θ)IΘ1 (θ),
(under the standard dominating measures on Θ0 and Θ1)
Using the prior probabilities π(Θ0) = ρ0 and π(Θ1) = ρ1,
π(θ) = ρ0π0(θ) + ρ1π1(θ).
Note If Θ0 = {θ0}, π0 is the Dirac mass in θ0
Point null refurbishment
Requirement
Defined prior distributions under both assumptions,
π0(θ) ∝ π(θ)IΘ0 (θ), π1(θ) ∝ π(θ)IΘ1 (θ),
(under the standard dominating measures on Θ0 and Θ1)
Using the prior probabilities π(Θ0) = ρ0 and π(Θ1) = ρ1,
π(θ) = ρ0π0(θ) + ρ1π1(θ).
Note If Θ0 = {θ0}, π0 is the Dirac mass in θ0
Point null hypotheses
Particular case H0 : θ = θ0
Take ρ0 = Prπ
(θ = θ0) and g1 prior density under Ha.
Posterior probability of H0
π(Θ0|x) =
f (x|θ0)ρ0
f (x|θ)π(θ) dθ
=
f (x|θ0)ρ0
f (x|θ0)ρ0 + (1 − ρ0)m1(x)
and marginal under Ha
m1(x) =
Θ1
f (x|θ)g1(θ) dθ.
Point null hypotheses
Particular case H0 : θ = θ0
Take ρ0 = Prπ
(θ = θ0) and g1 prior density under Ha.
Posterior probability of H0
π(Θ0|x) =
f (x|θ0)ρ0
f (x|θ)π(θ) dθ
=
f (x|θ0)ρ0
f (x|θ0)ρ0 + (1 − ρ0)m1(x)
and marginal under Ha
m1(x) =
Θ1
f (x|θ)g1(θ) dθ.
Point null hypotheses (cont’d)
Dual representation
π(Θ0|x) = 1 +
1 − ρ0
ρ0
m1(x)
f (x|θ0)
−1
.
and
Bπ
01(x) =
f (x|θ0)ρ0
m1(x)(1 − ρ0)
ρ0
1 − ρ0
=
f (x|θ0)
m1(x)
Connection
π(Θ0|x) = 1 +
1 − ρ0
ρ0
1
Bπ
01(x)
−1
.
Point null hypotheses (cont’d)
Dual representation
π(Θ0|x) = 1 +
1 − ρ0
ρ0
m1(x)
f (x|θ0)
−1
.
and
Bπ
01(x) =
f (x|θ0)ρ0
m1(x)(1 − ρ0)
ρ0
1 − ρ0
=
f (x|θ0)
m1(x)
Connection
π(Θ0|x) = 1 +
1 − ρ0
ρ0
1
Bπ
01(x)
−1
.
A further difficulty
Improper priors are not allowed here
If
Θ1
π1(dθ1) = ∞ or
Θ2
π2(dθ2) = ∞
then π1 or π2 cannot be coherently normalised while the
normalisation matters in the Bayes factor remember Bayes factor?
A further difficulty
Improper priors are not allowed here
If
Θ1
π1(dθ1) = ∞ or
Θ2
π2(dθ2) = ∞
then π1 or π2 cannot be coherently normalised while the
normalisation matters in the Bayes factor remember Bayes factor?
ToP unaware of the problem?
A. Not entirely, as improper priors keep being used on nuisance
parameters
Example of testing for a zero normal mean:
If σ is the standard error and λ the true value, λ is 0 on
q. We want a suitable form for its prior on q . (...) Then
we should take
P(qdσ|H) ∝ dσ/σ
P(q dσdλ|H) ∝ f
λ
σ
dσ/σdλ/λ
where f [is a true density] (V, §5.2).
Fallacy of the “same” σ!
ToP unaware of the problem?
A. Not entirely, as improper priors keep being used on nuisance
parameters
Example of testing for a zero normal mean:
If σ is the standard error and λ the true value, λ is 0 on
q. We want a suitable form for its prior on q . (...) Then
we should take
P(qdσ|H) ∝ dσ/σ
P(q dσdλ|H) ∝ f
λ
σ
dσ/σdλ/λ
where f [is a true density] (V, §5.2).
Fallacy of the “same” σ!
Not enought information
If s = 0 [!!!], then [for σ = |¯x|/τ, λ = σv]
P(q|θH) ∝
∞
0
τ
|¯x|
n
exp −
1
2
nτ2 dτ
τ
,
P(q |θH) ∝
∞
0
dτ
τ
∞
−∞
τ
|¯x|
n
f (v) exp −
1
2
n(v − τ)2
.
If n = 1 and f (v) is any even [density],
P(q |θH) ∝
1
2
√
2π
|¯x|
and P(q|θH) ∝
1
2
√
2π
|¯x|
and therefore K = 1 (V, §5.2).
Strange constraints
If n 2, the condition that K = 0 for s = 0, ¯x = 0 is
equivalent to
∞
0
f (v)vn−1
dv = ∞ .
The function satisfying this condition for [all] n is
f (v) =
1
π(1 + v2)
This is the prior recommended by Jeffreys hereafter.
But, first, many other families of densities satisfy this constraint
and a scale of 1 cannot be universal!
Second, s = 0 is a zero probability event...
Strange constraints
If n 2, the condition that K = 0 for s = 0, ¯x = 0 is
equivalent to
∞
0
f (v)vn−1
dv = ∞ .
The function satisfying this condition for [all] n is
f (v) =
1
π(1 + v2)
This is the prior recommended by Jeffreys hereafter.
But, first, many other families of densities satisfy this constraint
and a scale of 1 cannot be universal!
Second, s = 0 is a zero probability event...
Strange constraints
If n 2, the condition that K = 0 for s = 0, ¯x = 0 is
equivalent to
∞
0
f (v)vn−1
dv = ∞ .
The function satisfying this condition for [all] n is
f (v) =
1
π(1 + v2)
This is the prior recommended by Jeffreys hereafter.
But, first, many other families of densities satisfy this constraint
and a scale of 1 cannot be universal!
Second, s = 0 is a zero probability event...
Comments
ToP very imprecise about choice of priors in the setting of
tests (despite existence of Susie’s Jeffreys’ conventional partly
proper priors)
ToP misses the difficulty of improper priors [coherent with
earlier stance]
but this problem still generates debates within the B
community
Some degree of goodness-of-fit testing but against fixed
alternatives
Persistence of the form
K ≈
πn
2
1 +
t2
ν
−1/2ν+1/2
but ν not so clearly defined...
Noninformative proposals
Significance tests: one new parameter
Noninformative solutions
Pseudo-Bayes factors
Intrinsic priors
Jeffreys-Lindley paradox
Deviance (information criterion)
Testing under incomplete information
Testing via mixtures
Vague proper priors are not the solution
Taking a proper prior and take a “very large” variance (e.g.,
BUGS) will most often result in an undefined or ill-defined limit
Example (Lindley’s paradox)
If testing H0 : θ = 0 when observing x ∼ N(θ, 1), under a normal
N(0, α) prior π1(θ),
B01(x)
α−→∞
−→ 0
Vague proper priors are not the solution
Taking a proper prior and take a “very large” variance (e.g.,
BUGS) will most often result in an undefined or ill-defined limit
Example (Lindley’s paradox)
If testing H0 : θ = 0 when observing x ∼ N(θ, 1), under a normal
N(0, α) prior π1(θ),
B01(x)
α−→∞
−→ 0
Vague proper priors are not the solution
Taking a proper prior and take a “very large” variance (e.g.,
BUGS) will most often result in an undefined or ill-defined limit
Example (Lindley’s paradox)
If testing H0 : θ = 0 when observing x ∼ N(θ, 1), under a normal
N(0, α) prior π1(θ),
B01(x)
α−→∞
−→ 0
Vague proper priors are not the solution (cont’d)
Example (Poisson versus Negative binomial (4))
B12 =
1
0
λα+x−1
x!
e−λβ
dλ
1
M m
x
m − x + 1
βα
Γ(α)
if λ ∼ Ga(α, β)
=
Γ(α + x)
x! Γ(α)
β−x 1
M m
x
m − x + 1
=
(x + α − 1) · · · α
x(x − 1) · · · 1
β−x 1
M m
x
m − x + 1
depends on choice of α(β) or β(α) −→ 0
Vague proper priors are not the solution (cont’d)
Example (Poisson versus Negative binomial (4))
B12 =
1
0
λα+x−1
x!
e−λβ
dλ
1
M m
x
m − x + 1
βα
Γ(α)
if λ ∼ Ga(α, β)
=
Γ(α + x)
x! Γ(α)
β−x 1
M m
x
m − x + 1
=
(x + α − 1) · · · α
x(x − 1) · · · 1
β−x 1
M m
x
m − x + 1
depends on choice of α(β) or β(α) −→ 0
Learning from the sample
Definition (Learning sample)
Given an improper prior π, (x1, . . . , xn) is a learning sample if
π(·|x1, . . . , xn) is proper and a minimal learning sample if none of
its subsamples is a learning sample
There is just enough information in a minimal learning sample to
make inference about θ under the prior π
Learning from the sample
Definition (Learning sample)
Given an improper prior π, (x1, . . . , xn) is a learning sample if
π(·|x1, . . . , xn) is proper and a minimal learning sample if none of
its subsamples is a learning sample
There is just enough information in a minimal learning sample to
make inference about θ under the prior π
Pseudo-Bayes factors
Idea
Use one part x[i] of the data x to make the prior proper:
πi improper but πi (·|x[i]) proper
and
fi (x[n/i]|θi ) πi (θi |x[i])dθi
fj (x[n/i]|θj ) πj (θj |x[i])dθj
independent of normalizing constant
Use remaining x[n/i] to run test as if πj (θj |x[i]) is the true prior
Pseudo-Bayes factors
Idea
Use one part x[i] of the data x to make the prior proper:
πi improper but πi (·|x[i]) proper
and
fi (x[n/i]|θi ) πi (θi |x[i])dθi
fj (x[n/i]|θj ) πj (θj |x[i])dθj
independent of normalizing constant
Use remaining x[n/i] to run test as if πj (θj |x[i]) is the true prior
Pseudo-Bayes factors
Idea
Use one part x[i] of the data x to make the prior proper:
πi improper but πi (·|x[i]) proper
and
fi (x[n/i]|θi ) πi (θi |x[i])dθi
fj (x[n/i]|θj ) πj (θj |x[i])dθj
independent of normalizing constant
Use remaining x[n/i] to run test as if πj (θj |x[i]) is the true prior
Motivation
Provides a working principle for improper priors
Gather enough information from data to achieve properness
and use this properness to run the test on remaining data
does not use x twice as in Aitkin’s (1991)
Motivation
Provides a working principle for improper priors
Gather enough information from data to achieve properness
and use this properness to run the test on remaining data
does not use x twice as in Aitkin’s (1991)
Motivation
Provides a working principle for improper priors
Gather enough information from data to achieve properness
and use this properness to run the test on remaining data
does not use x twice as in Aitkin’s (1991)
Details
Since π1(θ1|x[i]) =
π1(θ1)f 1
[i](x[i]|θ1)
π1(θ1)f 1
[i](x[i]|θ1)dθ1
B12(x[n/i]) =
f 1
[n/i](x[n/i]|θ1)π1(θ1|x[i])dθ1
f 2
[n/i](x[n/i]|θ2)π2(θ2|x[i])dθ2
=
f1(x|θ1)π1(θ1)dθ1
f2(x|θ2)π2(θ2)dθ2
π2(θ2)f 2
[i](x[i]|θ2)dθ2
π1(θ1)f 1
[i](x[i]|θ1)dθ1
= BN
12(x)B21(x[i])
c Independent of scaling factor!
Unexpected problems!
depends on the choice of x[i]
many ways of combining pseudo-Bayes factors
AIBF = BN
ji
1
L
Bij (x[ ])
MIBF = BN
ji med[Bij (x[ ])]
GIBF = BN
ji exp
1
L
log Bij (x[ ])
not often an exact Bayes factor
and thus lacking inner coherence
B12 = B10B02 and B01 = 1/B10 .
[Berger & Pericchi, 1996]
Unexpected problems!
depends on the choice of x[i]
many ways of combining pseudo-Bayes factors
AIBF = BN
ji
1
L
Bij (x[ ])
MIBF = BN
ji med[Bij (x[ ])]
GIBF = BN
ji exp
1
L
log Bij (x[ ])
not often an exact Bayes factor
and thus lacking inner coherence
B12 = B10B02 and B01 = 1/B10 .
[Berger & Pericchi, 1996]
Unexpec’d problems (cont’d)
Example (Mixtures)
There is no sample size that proper-ises improper priors, except if a
training sample is allocated to each component
Reason If
x1, . . . , xn ∼
k
i=1
pi f (x|θi )
and
π(θ) =
i
πi (θi ) with πi (θi )dθi = +∞ ,
the posterior is never defined, because
Pr(“no observation from f (·|θi )”) = (1 − pi )n
Unexpec’d problems (cont’d)
Example (Mixtures)
There is no sample size that proper-ises improper priors, except if a
training sample is allocated to each component
Reason If
x1, . . . , xn ∼
k
i=1
pi f (x|θi )
and
π(θ) =
i
πi (θi ) with πi (θi )dθi = +∞ ,
the posterior is never defined, because
Pr(“no observation from f (·|θi )”) = (1 − pi )n
Intrinsic priors
There may exist a true prior that provides the same Bayes factor
Example (Normal mean)
Take x ∼ N(θ, 1) with either θ = 0 (M1) or θ = 0 (M2) and
π2(θ) = 1.
Then
BAIBF
21 = B21
1√
2π
1
n
n
i=1 e−x2
1 /2 ≈ B21 for N(0, 2)
BMIBF
21 = B21
1√
2π
e−med(x2
1 )/2 ≈ 0.93B21 for N(0, 1.2)
[Berger and Pericchi, 1998]
When such a prior exists, it is called an intrinsic prior
Intrinsic priors
There may exist a true prior that provides the same Bayes factor
Example (Normal mean)
Take x ∼ N(θ, 1) with either θ = 0 (M1) or θ = 0 (M2) and
π2(θ) = 1.
Then
BAIBF
21 = B21
1√
2π
1
n
n
i=1 e−x2
1 /2 ≈ B21 for N(0, 2)
BMIBF
21 = B21
1√
2π
e−med(x2
1 )/2 ≈ 0.93B21 for N(0, 1.2)
[Berger and Pericchi, 1998]
When such a prior exists, it is called an intrinsic prior
Intrinsic priors
There may exist a true prior that provides the same Bayes factor
Example (Normal mean)
Take x ∼ N(θ, 1) with either θ = 0 (M1) or θ = 0 (M2) and
π2(θ) = 1.
Then
BAIBF
21 = B21
1√
2π
1
n
n
i=1 e−x2
1 /2 ≈ B21 for N(0, 2)
BMIBF
21 = B21
1√
2π
e−med(x2
1 )/2 ≈ 0.93B21 for N(0, 1.2)
[Berger and Pericchi, 1998]
When such a prior exists, it is called an intrinsic prior
Intrinsic priors (cont’d)
Example (Exponential scale)
Take x1, . . . , xn
i.i.d.
∼ exp(θ − x)Ix θ
and H0 : θ = θ0, H1 : θ > θ0 , with π1(θ) = 1
Then
BA
10 = B10(x)
1
n
n
i=1
exi −θ0
− 1
−1
is the Bayes factor for
π2(θ) = eθ0−θ
1 − log 1 − eθ0−θ
Most often, however, the pseudo-Bayes factors do not correspond
to any true Bayes factor
[Berger and Pericchi, 2001]
Intrinsic priors (cont’d)
Example (Exponential scale)
Take x1, . . . , xn
i.i.d.
∼ exp(θ − x)Ix θ
and H0 : θ = θ0, H1 : θ > θ0 , with π1(θ) = 1
Then
BA
10 = B10(x)
1
n
n
i=1
exi −θ0
− 1
−1
is the Bayes factor for
π2(θ) = eθ0−θ
1 − log 1 − eθ0−θ
Most often, however, the pseudo-Bayes factors do not correspond
to any true Bayes factor
[Berger and Pericchi, 2001]
Intrinsic priors (cont’d)
Example (Exponential scale)
Take x1, . . . , xn
i.i.d.
∼ exp(θ − x)Ix θ
and H0 : θ = θ0, H1 : θ > θ0 , with π1(θ) = 1
Then
BA
10 = B10(x)
1
n
n
i=1
exi −θ0
− 1
−1
is the Bayes factor for
π2(θ) = eθ0−θ
1 − log 1 − eθ0−θ
Most often, however, the pseudo-Bayes factors do not correspond
to any true Bayes factor
[Berger and Pericchi, 2001]
Fractional Bayes factor
Idea
use directly the likelihood to separate training sample from testing
sample
BF
12 = B12(x)
Lb
2(θ2)π2(θ2)dθ2
Lb
1(θ1)π1(θ1)dθ1
[O’Hagan, 1995]
Proportion b of the sample used to gain proper-ness
Fractional Bayes factor
Idea
use directly the likelihood to separate training sample from testing
sample
BF
12 = B12(x)
Lb
2(θ2)π2(θ2)dθ2
Lb
1(θ1)π1(θ1)dθ1
[O’Hagan, 1995]
Proportion b of the sample used to gain proper-ness
Fractional Bayes factor (cont’d)
Example (Normal mean)
BF
12 =
1
√
b
en(b−1)¯x2
n /2
corresponds to exact Bayes factor for the prior N 0, 1−b
nb
If b constant, prior variance goes to 0
If b =
1
n
, prior variance stabilises around 1
If b = n−α
, α < 1, prior variance goes to 0 too.
Jeffreys–Lindley paradox
Significance tests: one new parameter
Noninformative solutions
Jeffreys-Lindley paradox
Lindley’s paradox
dual versions of the paradox
Bayesian resolutions
Deviance (information criterion)
Testing under incomplete information
Testing via mixtures
Lindley’s paradox
In a normal mean testing problem,
¯xn ∼ N(θ, σ2
/n) , H0 : θ = θ0 ,
under Jeffreys prior, θ ∼ N(θ0, σ2), the Bayes factor
B01(tn) = (1 + n)1/2
exp −nt2
n/2[1 + n] ,
where tn =
√
n|¯xn − θ0|/σ, satisfies
B01(tn)
n−→∞
−→ ∞
[assuming a fixed tn]
[Lindley, 1957]
Lindley’s paradox
Often dubbed Jeffreys–Lindley paradox...
In terms of
t =
√
n − 1¯x/s , ν = n−1
K ∼
πν
2
1 +
t2
ν
−1/2ν+1/2
.
(...) The variation of K with t
is much more important than
the variation with ν (Jeffreys,
V, §5.2).
Two versions of the paradox
“the weight of Lindley’s paradoxical result (...) burdens
proponents of the Bayesian practice”.
[Lad, 2003]
official version, opposing frequentist and Bayesian assessments
[Lindley, 1957]
intra-Bayesian version, blaming vague and improper priors for
the Bayes factor misbehaviour:
if π1(·|σ) depends on a scale parameter σ, it is often the case
that
B01(x)
σ−→∞
−→ +∞
for a given x, meaning H0 is always accepted
[Robert, 1992, 2013]
where does it matter?
In the normal case, Z ∼ N(θ, 1), θ ∼ N(0, α2), Bayes factor
B10(z) =
ez2α2/(1+α2)
√
1 + α2
=
√
1 − λ exp{λz2
/2}
Evacuation of the first version
Two paradigms [(b) versus (f)]
one (b) operates on the parameter space Θ, while the other
(f) is produced from the sample space
one (f) relies solely on the point-null hypothesis H0 and the
corresponding sampling distribution, while the other
(b) opposes H0 to a (predictive) marginal version of H1
one (f) could reject “a hypothesis that may be true (...)
because it has not predicted observable results that have not
occurred” (Jeffreys, ToP, VII, §7.2) while the other
(b) conditions upon the observed value xobs
one (f) cannot agree with the likelihood principle, while the
other (b) is almost uniformly in agreement with it
one (f) resorts to an arbitrary fixed bound α on the p-value,
while the other (b) refers to the (default) boundary probability
of 1/2
More arguments on the first version
observing a constant tn as n increases is of limited interest:
under H0 tn has limiting N(0, 1) distribution, while, under H1
tn a.s. converges to ∞
behaviour that remains entirely compatible with the
consistency of the Bayes factor, which a.s. converges either to
0 or ∞, depending on which hypothesis is true.
Consequent subsequent literature (e.g., Berger & Sellke, 1987;
Bayarri & Berger, 2004) has since then shown how divergent those
two approaches could be (to the point of being asymptotically
incompatible).
Nothing’s wrong with the second version
n, prior’s scale factor: prior variance n times larger than the
observation variance and when n goes to ∞, Bayes factor
goes to ∞ no matter what the observation is
n becomes what Lindley (1957) calls “a measure of lack of
conviction about the null hypothesis”
when prior diffuseness under H1 increases, only relevant
information becomes that θ could be equal to θ0, and this
overwhelms any evidence to the contrary contained in the data
mass of the prior distribution in the vicinity of any fixed
neighbourhood of the null hypothesis vanishes to zero under
H1
c deep coherence in the outcome: being indecisive about
the alternative hypothesis means we should not chose it
Nothing’s wrong with the second version
n, prior’s scale factor: prior variance n times larger than the
observation variance and when n goes to ∞, Bayes factor
goes to ∞ no matter what the observation is
n becomes what Lindley (1957) calls “a measure of lack of
conviction about the null hypothesis”
when prior diffuseness under H1 increases, only relevant
information becomes that θ could be equal to θ0, and this
overwhelms any evidence to the contrary contained in the data
mass of the prior distribution in the vicinity of any fixed
neighbourhood of the null hypothesis vanishes to zero under
H1
c deep coherence in the outcome: being indecisive about
the alternative hypothesis means we should not chose it
On some resolutions of the second version
use of pseudo-Bayes factors, fractional Bayes factors, &tc,
which lacks complete proper Bayesian justification
[Berger & Pericchi, 2001]
use of identical improper priors on nuisance parameters,
use of the posterior predictive distribution,
matching priors,
use of score functions extending the log score function
non-local priors correcting default priors
On some resolutions of the second version
use of pseudo-Bayes factors, fractional Bayes factors, &tc,
use of identical improper priors on nuisance parameters, a
notion already entertained by Jeffreys
[Berger et al., 1998; Marin & Robert, 2013]
use of the posterior predictive distribution,
matching priors,
use of score functions extending the log score function
non-local priors correcting default priors
On some resolutions of the second version
use of pseudo-Bayes factors, fractional Bayes factors, &tc,
use of identical improper priors on nuisance parameters,
P´ech´e de jeunesse: equating the values of the prior densities
at the point-null value θ0,
ρ0 = (1 − ρ0)π1(θ0)
[Robert, 1993]
use of the posterior predictive distribution,
matching priors,
use of score functions extending the log score function
non-local priors correcting default priors
On some resolutions of the second version
use of pseudo-Bayes factors, fractional Bayes factors, &tc,
use of identical improper priors on nuisance parameters,
use of the posterior predictive distribution, which uses the
data twice
matching priors,
use of score functions extending the log score function
non-local priors correcting default priors
On some resolutions of the second version
use of pseudo-Bayes factors, fractional Bayes factors, &tc,
use of identical improper priors on nuisance parameters,
use of the posterior predictive distribution,
matching priors, whose sole purpose is to bring frequentist
and Bayesian coverages as close as possible
[Datta & Mukerjee, 2004]
use of score functions extending the log score function
non-local priors correcting default priors
On some resolutions of the second version
use of pseudo-Bayes factors, fractional Bayes factors, &tc,
use of identical improper priors on nuisance parameters,
use of the posterior predictive distribution,
matching priors,
use of score functions extending the log score function
log B12(x) = log m1(x) − log m2(x) = S0(x, m1) − S0(x, m2) ,
that are independent of the normalising constant
[Dawid et al., 2013; Dawid & Musio, 2015]
non-local priors correcting default priors
On some resolutions of the second version
use of pseudo-Bayes factors, fractional Bayes factors, &tc,
use of identical improper priors on nuisance parameters,
use of the posterior predictive distribution,
matching priors,
use of score functions extending the log score function
non-local priors correcting default priors towards more
balanced error rates
[Johnson & Rossell, 2010; Consonni et al., 2013]
Deviance (information criterion)
Significance tests: one new parameter
Noninformative solutions
Jeffreys-Lindley paradox
Deviance (information criterion)
Testing under incomplete information
Testing via mixtures
DIC as in Dayesian?
Deviance defined by
D(θ) = −2 log(p(y|θ)) ,
Effective number of parameters computed as
pD = ¯D − D(¯θ) ,
with ¯D posterior expectation of D and ¯θ estimate of θ
Deviance information criterion (DIC) defined by
DIC = pD + ¯D
= D(¯θ) + 2pD
Models with smaller DIC better supported by the data
[Spiegelhalter et al., 2002]
“thou shalt not use the data twice”
The data is used twice in the DIC method:
1. y used once to produce the posterior π(θ|y), and the
associated estimate, ˜θ(y)
2. y used a second time to compute the posterior expectation
of the observed likelihood p(y|θ),
log p(y|θ)π(dθ|y) ∝ log p(y|θ)p(y|θ)π(dθ) ,
DIC for missing data models
Framework of missing data models
f (y|θ) = f (y, z|θ)dz ,
with observed data y = (y1, . . . , yn) and corresponding missing
data by z = (z1, . . . , zn)
How do we define DIC in such settings?
DIC for missing data models
Framework of missing data models
f (y|θ) = f (y, z|θ)dz ,
with observed data y = (y1, . . . , yn) and corresponding missing
data by z = (z1, . . . , zn)
How do we define DIC in such settings?
how many DICs can you fit in a mixture?
Q: How many giraffes can you fit in a VW bug?
A: None, the elephants are in there.
1. observed DICs
DIC1 = −4Eθ [log f (y|θ)|y] + 2 log f (y|Eθ [θ|y])
often a poor choice in case of unidentifiability
2. complete DICs based on f (y, z|θ)
3. conditional DICs based on f (y|z, θ)
[Celeux et al., BA, 2006]
how many DICs can you fit in a mixture?
Q: How many giraffes can you fit in a VW bug?
A: None, the elephants are in there.
1. observed DICs
DIC2 = −4Eθ [log f (y|θ)|y] + 2 log f (y|θ(y)) .
which uses posterior mode instead
2. complete DICs based on f (y, z|θ)
3. conditional DICs based on f (y|z, θ)
[Celeux et al., BA, 2006]
how many DICs can you fit in a mixture?
Q: How many giraffes can you fit in a VW bug?
A: None, the elephants are in there.
1. observed DICs
DIC3 = −4Eθ [log f (y|θ)|y] + 2 log f (y) ,
which instead relies on the MCMC density estimate
2. complete DICs based on f (y, z|θ)
3. conditional DICs based on f (y|z, θ)
[Celeux et al., BA, 2006]
how many DICs can you fit in a mixture?
Q: How many giraffes can you fit in a VW bug?
A: None, the elephants are in there.
1. observed DICs
2. complete DICs based on f (y, z|θ)
DIC4 = EZ [DIC(y, Z)|y]
= −4Eθ,Z [log f (y, Z|θ)|y] + 2EZ [log f (y, Z|Eθ[θ|y, Z])|y]
3. conditional DICs based on f (y|z, θ)
[Celeux et al., BA, 2006]
how many DICs can you fit in a mixture?
Q: How many giraffes can you fit in a VW bug?
A: None, the elephants are in there.
1. observed DICs
2. complete DICs based on f (y, z|θ)
DIC5 = −4Eθ,Z [log f (y, Z|θ)|y] + 2 log f (y, z(y)|θ(y)) ,
using Z as an additional parameter
3. conditional DICs based on f (y|z, θ)
[Celeux et al., BA, 2006]
how many DICs can you fit in a mixture?
Q: How many giraffes can you fit in a VW bug?
A: None, the elephants are in there.
1. observed DICs
2. complete DICs based on f (y, z|θ)
DIC6 = −4Eθ,Z [log f (y, Z|θ)|y]+2EZ[log f (y, Z|θ(y))|y, θ(y)] .
in analogy with EM, θ being an EM fixed point
3. conditional DICs based on f (y|z, θ)
[Celeux et al., BA, 2006]
how many DICs can you fit in a mixture?
Q: How many giraffes can you fit in a VW bug?
A: None, the elephants are in there.
1. observed DICs
2. complete DICs based on f (y, z|θ)
3. conditional DICs based on f (y|z, θ)
DIC7 = −4Eθ,Z [log f (y|Z, θ)|y] + 2 log f (y|z(y), θ(y)) ,
using MAP estimates
[Celeux et al., BA, 2006]
how many DICs can you fit in a mixture?
Q: How many giraffes can you fit in a VW bug?
A: None, the elephants are in there.
1. observed DICs
2. complete DICs based on f (y, z|θ)
3. conditional DICs based on f (y|z, θ)
DIC8 = −4Eθ,Z [log f (y|Z, θ)|y]+2EZ log f (y|Z, θ(y, Z))|y ,
conditioning first on Z and then integrating over Z
conditional on y
[Celeux et al., BA, 2006]
Galactic DICs
Example of the galaxy mixture dataset
DIC2 DIC3 DIC4 DIC5 DIC6 DIC7 DIC8
K (pD2) (pD3) (pD4) (pD5) (pD6) (pD7) (pD8)
2 453 451 502 705 501 417 410
(5.56) (3.66) (5.50) (207.88) (4.48) (11.07) (4.09)
3 440 436 461 622 471 378 372
(9.23) (4.94) (6.40) (167.28) (15.80) (13.59) (7.43)
4 446 439 473 649 482 388 382
(11.58) (5.41) (7.52) (183.48) (16.51) (17.47) (11.37)
5 447 442 485 658 511 395 390
(10.80) (5.48) (7.58) (180.73) (33.29) (20.00) (15.15)
6 449 444 494 676 532 407 398
(11.26) (5.49) (8.49) (191.10) (46.83) (28.23) (19.34)
7 460 446 508 700 571 425 409
(19.26) (5.83) (8.93) (200.35) (71.26) (40.51) (24.57)
questions
what is the behaviour of DIC under model mispecification?
is there an absolute scale to the DIC values, i.e. when is a
difference in DICs significant?
how can DIC handle small n’s versus p’s?
should pD be defined as var(D|y)/2 [Gelman’s suggestion]?
is WAIC (Gelman and Vehtari, 2013) making a difference for
being based on expected posterior predictive?
In an era of complex models, is DIC applicable?
[Robert, 2013]
questions
what is the behaviour of DIC under model mispecification?
is there an absolute scale to the DIC values, i.e. when is a
difference in DICs significant?
how can DIC handle small n’s versus p’s?
should pD be defined as var(D|y)/2 [Gelman’s suggestion]?
is WAIC (Gelman and Vehtari, 2013) making a difference for
being based on expected posterior predictive?
In an era of complex models, is DIC applicable?
[Robert, 2013]
Testing under incomplete information
Significance tests: one new parameter
Noninformative solutions
Jeffreys-Lindley paradox
Deviance (information criterion)
Testing under incomplete information
Testing via mixtures
Likelihood-free settings
Cases when the likelihood function f (y|θ) is unavailable (in
analytic and numerical senses) and when the completion step
f (y|θ) =
Z
f (y, z|θ) dz
is impossible or too costly because of the dimension of z
c MCMC cannot be implemented!
The ABC method
Bayesian setting: target is π(θ)f (x|θ)
When likelihood f (x|θ) not in closed form, likelihood-free rejection
technique:
ABC algorithm
For an observation y ∼ f (y|θ), under the prior π(θ), keep jointly
simulating
θ ∼ π(θ) , z ∼ f (z|θ ) ,
until the auxiliary variable z is equal to the observed value, z = y.
[Tavar´e et al., 1997]
The ABC method
Bayesian setting: target is π(θ)f (x|θ)
When likelihood f (x|θ) not in closed form, likelihood-free rejection
technique:
ABC algorithm
For an observation y ∼ f (y|θ), under the prior π(θ), keep jointly
simulating
θ ∼ π(θ) , z ∼ f (z|θ ) ,
until the auxiliary variable z is equal to the observed value, z = y.
[Tavar´e et al., 1997]
The ABC method
Bayesian setting: target is π(θ)f (x|θ)
When likelihood f (x|θ) not in closed form, likelihood-free rejection
technique:
ABC algorithm
For an observation y ∼ f (y|θ), under the prior π(θ), keep jointly
simulating
θ ∼ π(θ) , z ∼ f (z|θ ) ,
until the auxiliary variable z is equal to the observed value, z = y.
[Tavar´e et al., 1997]
A as A...pproximative
When y is a continuous random variable, strict equality z = y is
replaced with a tolerance zone
ρ(y, z)
where ρ is a distance
Output distributed from
π(θ) Pθ{ρ(y, z) < }
def
∝ π(θ|ρ(y, z) < )
[Pritchard et al., 1999]
A as A...pproximative
When y is a continuous random variable, strict equality z = y is
replaced with a tolerance zone
ρ(y, z)
where ρ is a distance
Output distributed from
π(θ) Pθ{ρ(y, z) < }
def
∝ π(θ|ρ(y, z) < )
[Pritchard et al., 1999]
ABC algorithm
In most implementations, further degree of A...pproximation:
Algorithm 1 Likelihood-free rejection sampler
for i = 1 to N do
repeat
generate θ from the prior distribution π(·)
generate z from the likelihood f (·|θ )
until ρ{η(z), η(y)}
set θi = θ
end for
where η(y) defines a (not necessarily sufficient) statistic
Which summary η(·)?
Fundamental difficulty of the choice of the summary statistic when
there is no non-trivial sufficient statistics
Loss of statistical information balanced against gain in data
roughening
Approximation error and information loss remain unknown
Choice of statistics induces choice of distance function
towards standardisation
may be imposed for external/practical reasons (e.g., LDA)
may gather several non-B point estimates
can learn about efficient combination
[Estoup et al., 2012, Genetics]
Which summary η(·)?
Fundamental difficulty of the choice of the summary statistic when
there is no non-trivial sufficient statistics
Loss of statistical information balanced against gain in data
roughening
Approximation error and information loss remain unknown
Choice of statistics induces choice of distance function
towards standardisation
may be imposed for external/practical reasons (e.g., LDA)
may gather several non-B point estimates
can learn about efficient combination
[Estoup et al., 2012, Genetics]
Which summary η(·)?
Fundamental difficulty of the choice of the summary statistic when
there is no non-trivial sufficient statistics
Loss of statistical information balanced against gain in data
roughening
Approximation error and information loss remain unknown
Choice of statistics induces choice of distance function
towards standardisation
may be imposed for external/practical reasons (e.g., LDA)
may gather several non-B point estimates
can learn about efficient combination
[Estoup et al., 2012, Genetics]
Generic ABC for model choice
Algorithm 2 Likelihood-free model choice sampler (ABC-MC)
for t = 1 to T do
repeat
Generate m from the prior π(M = m)
Generate θm from the prior πm(θm)
Generate z from the model fm(z|θm)
until ρ{η(z), η(y)} <
Set m(t) = m and θ(t)
= θm
end for
[Grelaud et al., 2009]
ABC estimates
Posterior probability π(M = m|y) approximated by the frequency
of acceptances from model m
1
T
T
t=1
Im(t)=m .
Issues with implementation:
should tolerances be the same for all models?
should summary statistics vary across models (incl. their
dimension)?
should the distance measure ρ vary as well?
ABC estimates
Posterior probability π(M = m|y) approximated by the frequency
of acceptances from model m
1
T
T
t=1
Im(t)=m .
Extension to a weighted polychotomous logistic regression estimate
of π(M = m|y), with non-parametric kernel weights
[Cornuet et al., DIYABC, 2009]
ABCµ
Idea Infer about the error as well as about the parameter:
Use of a joint density
f (θ, |y) ∝ ξ( |y, θ) × πθ(θ) × π ( )
where y is the data, and ξ( |y, θ) is the prior predictive density of
ρ(η(z), η(y)) given θ and y when z ∼ f (z|θ)
Warning! Replacement of ξ( |y, θ) with a non-parametric kernel
approximation.
[Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]
ABCµ
Idea Infer about the error as well as about the parameter:
Use of a joint density
f (θ, |y) ∝ ξ( |y, θ) × πθ(θ) × π ( )
where y is the data, and ξ( |y, θ) is the prior predictive density of
ρ(η(z), η(y)) given θ and y when z ∼ f (z|θ)
Warning! Replacement of ξ( |y, θ) with a non-parametric kernel
approximation.
[Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]
ABCµ
Idea Infer about the error as well as about the parameter:
Use of a joint density
f (θ, |y) ∝ ξ( |y, θ) × πθ(θ) × π ( )
where y is the data, and ξ( |y, θ) is the prior predictive density of
ρ(η(z), η(y)) given θ and y when z ∼ f (z|θ)
Warning! Replacement of ξ( |y, θ) with a non-parametric kernel
approximation.
[Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]
ABCµ details
Multidimensional distances ρk (k = 1, . . . , K) and errors
k = ρk(ηk(z), ηk(y)), with
k ∼ ξk( |y, θ) ≈ ^ξk( |y, θ) =
1
Bhk
b
K[{ k−ρk(ηk(zb), ηk(y))}/hk]
then used in replacing ξ( |y, θ) with mink
^ξk( |y, θ)
ABCµ involves acceptance probability
π(θ , )
π(θ, )
q(θ , θ)q( , )
q(θ, θ )q( , )
mink
^ξk( |y, θ )
mink
^ξk( |y, θ)
ABCµ details
Multidimensional distances ρk (k = 1, . . . , K) and errors
k = ρk(ηk(z), ηk(y)), with
k ∼ ξk( |y, θ) ≈ ^ξk( |y, θ) =
1
Bhk
b
K[{ k−ρk(ηk(zb), ηk(y))}/hk]
then used in replacing ξ( |y, θ) with mink
^ξk( |y, θ)
ABCµ involves acceptance probability
π(θ , )
π(θ, )
q(θ , θ)q( , )
q(θ, θ )q( , )
mink
^ξk( |y, θ )
mink
^ξk( |y, θ)
ABCµ multiple errors
[ c Ratmann et al., PNAS, 2009]
ABCµ for model choice
[ c Ratmann et al., PNAS, 2009]
Questions about ABCµ [and model choice]
For each model under comparison, marginal posterior on used to
assess the fit of the model (HPD includes 0 or not).
Is the data informative about ? [Identifiability]
How much does the prior π( ) impact the comparison?
How is using both ξ( |x0, θ) and π ( ) compatible with a
standard probability model?
Where is the penalisation for complexity in the model
comparison?
[X, Mengersen & Chen, 2010, PNAS]
Questions about ABCµ [and model choice]
For each model under comparison, marginal posterior on used to
assess the fit of the model (HPD includes 0 or not).
Is the data informative about ? [Identifiability]
How much does the prior π( ) impact the comparison?
How is using both ξ( |x0, θ) and π ( ) compatible with a
standard probability model?
Where is the penalisation for complexity in the model
comparison?
[X, Mengersen & Chen, 2010, PNAS]
Formalised framework
Central question to the validation of ABC for model choice:
When is a Bayes factor based on an insufficient statistic T(y)
consistent?
Note: c drawn on T(y) through BT
12(y) necessarily differs from
c drawn on y through B12(y)
Formalised framework
Central question to the validation of ABC for model choice:
When is a Bayes factor based on an insufficient statistic T(y)
consistent?
Note: c drawn on T(y) through BT
12(y) necessarily differs from
c drawn on y through B12(y)
A benchmark if toy example
Comparison suggested by referee of PNAS paper [thanks]:
[X, Cornuet, Marin, & Pillai, Aug. 2011]
Model M1: y ∼ N(θ1, 1) opposed to model M2: y ∼ L(θ2, 1/
√
2),
Laplace distribution with mean θ2 and scale parameter 1/
√
2
(variance one).
A benchmark if toy example
Comparison suggested by referee of PNAS paper [thanks]:
[X, Cornuet, Marin, & Pillai, Aug. 2011]
Model M1: y ∼ N(θ1, 1) opposed to model M2: y ∼ L(θ2, 1/
√
2),
Laplace distribution with mean θ2 and scale parameter 1/
√
2
(variance one).
Four possible statistics
1. sample mean y (sufficient for M1 if not M2);
2. sample median med(y) (insufficient);
3. sample variance var(y) (ancillary);
4. median absolute deviation mad(y) = med(y − med(y));
A benchmark if toy example
Comparison suggested by referee of PNAS paper [thanks]:
[X, Cornuet, Marin, & Pillai, Aug. 2011]
Model M1: y ∼ N(θ1, 1) opposed to model M2: y ∼ L(θ2, 1/
√
2),
Laplace distribution with mean θ2 and scale parameter 1/
√
2
(variance one).
0.1 0.2 0.3 0.4 0.5 0.6 0.7
0123456
posterior probability
Density
A benchmark if toy example
Comparison suggested by referee of PNAS paper [thanks]:
[X, Cornuet, Marin, & Pillai, Aug. 2011]
Model M1: y ∼ N(θ1, 1) opposed to model M2: y ∼ L(θ2, 1/
√
2),
Laplace distribution with mean θ2 and scale parameter 1/
√
2
(variance one).
q
q
q
q
q
q
q
q
q
q
q
Gauss Laplace
0.00.10.20.30.40.50.60.7
n=100
A benchmark if toy example
Comparison suggested by referee of PNAS paper [thanks]:
[X, Cornuet, Marin, & Pillai, Aug. 2011]
Model M1: y ∼ N(θ1, 1) opposed to model M2: y ∼ L(θ2, 1/
√
2),
Laplace distribution with mean θ2 and scale parameter 1/
√
2
(variance one).
0.1 0.2 0.3 0.4 0.5 0.6 0.7
0123456
posterior probability
Density
0.0 0.2 0.4 0.6 0.8 1.0
0123
probability
Density
A benchmark if toy example
Comparison suggested by referee of PNAS paper [thanks]:
[X, Cornuet, Marin, & Pillai, Aug. 2011]
Model M1: y ∼ N(θ1, 1) opposed to model M2: y ∼ L(θ2, 1/
√
2),
Laplace distribution with mean θ2 and scale parameter 1/
√
2
(variance one).
q
q
q
q
q
q
q
q
q
q
q
Gauss Laplace
0.00.10.20.30.40.50.60.7
n=100
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
Gauss Laplace
0.00.20.40.60.81.0
n=100
Consistency theorem
If Pn belongs to one of the two models and if µ0 = E[T] cannot be
attained by the other one :
0 = min (inf{|µ0 − µi (θi )|; θi ∈ Θi }, i = 1, 2)
< max (inf{|µ0 − µi (θi )|; θi ∈ Θi }, i = 1, 2) ,
then the Bayes factor BT
12 is consistent
Conclusion
Model selection feasible with ABC:
Choice of summary statistics is paramount
At best, ABC output → π(. | η(y)) which concentrates
around µ0
For estimation : {θ; µ(θ) = µ0} = θ0
For testing {µ1(θ1), θ1 ∈ Θ1} ∩ {µ2(θ2), θ2 ∈ Θ2} = ∅
[Marin et al., 2013]
Conclusion
Model selection feasible with ABC:
Choice of summary statistics is paramount
At best, ABC output → π(. | η(y)) which concentrates
around µ0
For estimation : {θ; µ(θ) = µ0} = θ0
For testing {µ1(θ1), θ1 ∈ Θ1} ∩ {µ2(θ2), θ2 ∈ Θ2} = ∅
[Marin et al., 2013]
Changing the testing perspective
Significance tests: one new parameter
Noninformative solutions
Jeffreys-Lindley paradox
Deviance (information criterion)
Testing under incomplete information
Testing via mixtures
Testing issues
Hypothesis testing
central problem of statistical inference
dramatically differentiating feature between classical and
Bayesian paradigms
wide open to controversy and divergent opinions, includ.
within the Bayesian community
non-informative Bayesian testing case mostly unresolved,
witness the Jeffreys–Lindley paradox
[Berger (2003), Mayo & Cox (2006), Gelman (2008)]
Testing hypotheses
Bayesian model selection as comparison of k potential
statistical models towards the selection of model that fits the
data “best”
mostly accepted perspective: it does not primarily seek to
identify which model is “true”, but compares fits
tools like Bayes factor naturally include a penalisation
addressing model complexity, mimicked by Bayes Information
(BIC) and Deviance Information (DIC) criteria
posterior predictive tools successfully advocated in Gelman et
al. (2013) even though they involve double use of data
Testing hypotheses
Bayesian model selection as comparison of k potential
statistical models towards the selection of model that fits the
data “best”
mostly accepted perspective: it does not primarily seek to
identify which model is “true”, but compares fits
tools like Bayes factor naturally include a penalisation
addressing model complexity, mimicked by Bayes Information
(BIC) and Deviance Information (DIC) criteria
posterior predictive tools successfully advocated in Gelman et
al. (2013) even though they involve double use of data
Some difficulties
tension between using (i) posterior probabilities justified by a
binary loss function but depending on unnatural prior weights,
and (ii) Bayes factors that eliminate this dependence but
escape direct connection with posterior distribution, unless
prior weights are integrated within the loss
subsequent and delicate interpretation (or calibration) of the
strength of the Bayes factor towards supporting a given
hypothesis or model, because it is not a Bayesian decision rule
similar difficulty with posterior probabilities, with tendency to
interpret them as p-values (rather than the opposite!) when
they only report through a marginal likelihood ratio the
respective strengths of fitting the data to both models
Some further difficulties
long-lasting impact of the prior modeling, meaning the choice
of the prior distributions on the parameter spaces of both
models under comparison, despite overall consistency proof for
Bayes factor
discontinuity in use of improper priors since they are not
justified in most testing situations, leading to many alternative
if ad hoc solutions, where data is either used twice or split in
artificial ways
binary (accept vs. reject) outcome more suited for immediate
decision (if any) than for model evaluation, in connection with
rudimentary loss function
Some additional difficulties
related impossibility to ascertain simultaneous misfit or to
detect presence of outliers
no assessment of uncertainty associated with decision itself
difficult computation of marginal likelihoods in most settings
with further controversies about which algorithm to adopt
strong dependence of posterior probabilities on conditioning
statistics, which in turn undermines their validity for model
assessment, as exhibited in ABC model choice
temptation to create pseudo-frequentist equivalents such as
q-values with even less Bayesian justifications
Paradigm shift
New proposal for a paradigm shift in the Bayesian processing of
hypothesis testing and of model selection
convergent and naturally interpretable solution
more extended use of improper priors
Simple representation of the testing problem as a
two-component mixture estimation problem where the
weights are formally equal to 0 or 1
Paradigm shift
New proposal for a paradigm shift in the Bayesian processing of
hypothesis testing and of model selection
convergent and naturally interpretable solution
more extended use of improper priors
Simple representation of the testing problem as a
two-component mixture estimation problem where the
weights are formally equal to 0 or 1
Paradigm shift
Simple representation of the testing problem as a
two-component mixture estimation problem where the
weights are formally equal to 0 or 1
Approach inspired from consistency result of Rousseau and
Mengersen (2011) on estimated overfitting mixtures
Mixture representation not directly equivalent to the use of a
posterior probability
Potential of a better approach to testing, while not expanding
number of parameters
Calibration of the posterior distribution of the weight of a
model, while moving from the artificial notion of the posterior
probability of a model
Encompassing mixture model
Idea: Given two statistical models,
M1 : x ∼ f1(x|θ1) , θ1 ∈ Θ1 and M2 : x ∼ f2(x|θ2) , θ2 ∈ Θ2 ,
embed both within an encompassing mixture
Mα : x ∼ αf1(x|θ1) + (1 − α)f2(x|θ2) , 0 α 1 (1)
Note: Both models correspond to special cases of (1), one for
α = 1 and one for α = 0
Draw inference on mixture representation (1), as if each
observation was individually and independently produced by the
mixture model
Encompassing mixture model
Idea: Given two statistical models,
M1 : x ∼ f1(x|θ1) , θ1 ∈ Θ1 and M2 : x ∼ f2(x|θ2) , θ2 ∈ Θ2 ,
embed both within an encompassing mixture
Mα : x ∼ αf1(x|θ1) + (1 − α)f2(x|θ2) , 0 α 1 (1)
Note: Both models correspond to special cases of (1), one for
α = 1 and one for α = 0
Draw inference on mixture representation (1), as if each
observation was individually and independently produced by the
mixture model
Encompassing mixture model
Idea: Given two statistical models,
M1 : x ∼ f1(x|θ1) , θ1 ∈ Θ1 and M2 : x ∼ f2(x|θ2) , θ2 ∈ Θ2 ,
embed both within an encompassing mixture
Mα : x ∼ αf1(x|θ1) + (1 − α)f2(x|θ2) , 0 α 1 (1)
Note: Both models correspond to special cases of (1), one for
α = 1 and one for α = 0
Draw inference on mixture representation (1), as if each
observation was individually and independently produced by the
mixture model
Inferential motivations
Sounds like an approximation to the real model, but several
definitive advantages to this paradigm shift:
Bayes estimate of the weight α replaces posterior probability
of model M1, equally convergent indicator of which model is
“true”, while avoiding artificial prior probabilities on model
indices, ω1 and ω2
interpretation of estimator of α at least as natural as handling
the posterior probability, while avoiding zero-one loss setting
α and its posterior distribution provide measure of proximity
to the models, while being interpretable as data propensity to
stand within one model
further allows for alternative perspectives on testing and
model choice, like predictive tools, cross-validation, and
information indices like WAIC
Computational motivations
avoids highly problematic computations of the marginal
likelihoods, since standard algorithms are available for
Bayesian mixture estimation
straightforward extension to a finite collection of models, with
a larger number of components, which considers all models at
once and eliminates least likely models by simulation
eliminates difficulty of label switching that plagues both
Bayesian estimation and Bayesian computation, since
components are no longer exchangeable
posterior distribution of α evaluates more thoroughly strength
of support for a given model than the single figure outcome of
a posterior probability
variability of posterior distribution on α allows for a more
thorough assessment of the strength of this support
Noninformative motivations
additional feature missing from traditional Bayesian answers:
a mixture model acknowledges possibility that, for a finite
dataset, both models or none could be acceptable
standard (proper and informative) prior modeling can be
reproduced in this setting, but non-informative (improper)
priors also are manageable therein, provided both models first
reparameterised towards shared parameters, e.g. location and
scale parameters
in special case when all parameters are common
Mα : x ∼ αf1(x|θ) + (1 − α)f2(x|θ) , 0 α 1
if θ is a location parameter, a flat prior π(θ) ∝ 1 is available
Weakly informative motivations
using the same parameters or some identical parameters on
both components highlights that opposition between the two
components is not an issue of enjoying different parameters
those common parameters are nuisance parameters, to be
integrated out
prior model weights ωi rarely discussed in classical Bayesian
approach, even though linear impact on posterior probabilities.
Here, prior modeling only involves selecting a prior on α, e.g.,
α ∼ B(a0, a0)
while a0 impacts posterior on α, it always leads to mass
accumulation near 1 or 0, i.e. favours most likely model
sensitivity analysis straightforward to carry
approach easily calibrated by parametric boostrap providing
reference posterior of α under each model
natural Metropolis–Hastings alternative
Poisson/Geometric
choice betwen Poisson P(λ) and Geometric Geo(p)
distribution
mixture with common parameter λ
Mα : αP(λ) + (1 − α)Geo(1/1+λ)
Allows for Jeffreys prior since resulting posterior is proper
independent Metropolis–within–Gibbs with proposal
distribution on λ equal to Poisson posterior (with acceptance
rate larger than 75%)
Poisson/Geometric
choice betwen Poisson P(λ) and Geometric Geo(p)
distribution
mixture with common parameter λ
Mα : αP(λ) + (1 − α)Geo(1/1+λ)
Allows for Jeffreys prior since resulting posterior is proper
independent Metropolis–within–Gibbs with proposal
distribution on λ equal to Poisson posterior (with acceptance
rate larger than 75%)
Poisson/Geometric
choice betwen Poisson P(λ) and Geometric Geo(p)
distribution
mixture with common parameter λ
Mα : αP(λ) + (1 − α)Geo(1/1+λ)
Allows for Jeffreys prior since resulting posterior is proper
independent Metropolis–within–Gibbs with proposal
distribution on λ equal to Poisson posterior (with acceptance
rate larger than 75%)
Beta prior
When α ∼ Be(a0, a0) prior, full conditional posterior
α ∼ Be(n1(ζ) + a0, n2(ζ) + a0)
Exact Bayes factor opposing Poisson and Geometric
B12 = nn¯xn
n
i=1
xi ! Γ n + 2 +
n
i=1
xi Γ(n + 2)
although undefined from a purely mathematical viewpoint
Parameter estimation
1e-04 0.001 0.01 0.1 0.2 0.3 0.4 0.5
3.903.954.004.054.10
Posterior means of λ and medians of α for 100 Poisson P(4) datasets of size
n = 1000, for a0 = .0001, .001, .01, .1, .2, .3, .4, .5. Each posterior approximation
is based on 104
Metropolis-Hastings iterations.
Parameter estimation
1e-04 0.001 0.01 0.1 0.2 0.3 0.4 0.5
0.9900.9920.9940.9960.9981.000
Posterior means of λ and medians of α for 100 Poisson P(4) datasets of size
n = 1000, for a0 = .0001, .001, .01, .1, .2, .3, .4, .5. Each posterior approximation
is based on 104
Metropolis-Hastings iterations.
Consistency
0 1 2 3 4 5 6 7
0.00.20.40.60.81.0
a0=.1
log(sample size)
0 1 2 3 4 5 6 7
0.00.20.40.60.81.0
a0=.3
log(sample size)
0 1 2 3 4 5 6 7
0.00.20.40.60.81.0
a0=.5
log(sample size)
Posterior means (sky-blue) and medians (grey-dotted) of α, over 100 Poisson
P(4) datasets for sample sizes from 1 to 1000.
Behaviour of Bayes factor
0 1 2 3 4 5 6 7
0.00.20.40.60.81.0
log(sample size)
0 1 2 3 4 5 6 7
0.00.20.40.60.81.0
log(sample size)
0 1 2 3 4 5 6 7
0.00.20.40.60.81.0
log(sample size)
Comparison between P(M1|x) (red dotted area) and posterior medians of α
(grey zone) for 100 Poisson P(4) datasets with sample sizes n between 1 and
1000, for a0 = .001, .1, .5
Normal-normal comparison
comparison of a normal N(θ1, 1) with a normal N(θ2, 2)
distribution
mixture with identical location parameter θ
αN(θ, 1) + (1 − α)N(θ, 2)
Jeffreys prior π(θ) = 1 can be used, since posterior is proper
Reference (improper) Bayes factor
B12 = 2
n−1/2
exp 1/4
n
i=1
(xi − ¯x)2
,
Normal-normal comparison
comparison of a normal N(θ1, 1) with a normal N(θ2, 2)
distribution
mixture with identical location parameter θ
αN(θ, 1) + (1 − α)N(θ, 2)
Jeffreys prior π(θ) = 1 can be used, since posterior is proper
Reference (improper) Bayes factor
B12 = 2
n−1/2
exp 1/4
n
i=1
(xi − ¯x)2
,
Normal-normal comparison
comparison of a normal N(θ1, 1) with a normal N(θ2, 2)
distribution
mixture with identical location parameter θ
αN(θ, 1) + (1 − α)N(θ, 2)
Jeffreys prior π(θ) = 1 can be used, since posterior is proper
Reference (improper) Bayes factor
B12 = 2
n−1/2
exp 1/4
n
i=1
(xi − ¯x)2
,
Consistency
0.1 0.2 0.3 0.4 0.5 p(M1|x)
0.00.20.40.60.81.0
n=15
0.00.20.40.60.81.0
0.1 0.2 0.3 0.4 0.5 p(M1|x)
0.650.700.750.800.850.900.951.00
n=50
0.650.700.750.800.850.900.951.00
0.1 0.2 0.3 0.4 0.5 p(M1|x)
0.750.800.850.900.951.00
n=100
0.750.800.850.900.951.00
0.1 0.2 0.3 0.4 0.5 p(M1|x)
0.930.940.950.960.970.980.991.00
n=500
0.930.940.950.960.970.980.991.00
Posterior means (wheat) and medians of α (dark wheat), compared with
posterior probabilities of M0 (gray) for a N(0, 1) sample, derived from 100
datasets for sample sizes equal to 15, 50, 100, 500. Each posterior
approximation is based on 104
MCMC iterations.
Comparison with posterior probability
0 100 200 300 400 500
-50-40-30-20-100
a0=.1
sample size
0 100 200 300 400 500
-50-40-30-20-100
a0=.3
sample size
0 100 200 300 400 500
-50-40-30-20-100
a0=.4
sample size
0 100 200 300 400 500
-50-40-30-20-100
a0=.5
sample size
Plots of ranges of log(n) log(1 − E[α|x]) (gray color) and log(1 − p(M1|x)) (red
dotted) over 100 N(0, 1) samples as sample size n grows from 1 to 500. and α
is the weight of N(0, 1) in the mixture model. The shaded areas indicate the
range of the estimations and each plot is based on a Beta prior with
a0 = .1, .2, .3, .4, .5, 1 and each posterior approximation is based on 104
iterations.
Comments
convergence to one boundary value as sample size n grows
impact of hyperarameter a0 slowly vanishes as n increases, but
present for moderate sample sizes
when simulated sample is neither from N(θ1, 1) nor from
N(θ2, 2), behaviour of posterior varies, depending on which
distribution is closest
Logit or Probit?
binary dataset, R dataset about diabetes in 200 Pima Indian
women with body mass index as explanatory variable
comparison of logit and probit fits could be suitable. We are
thus comparing both fits via our method
M1 : yi | xi
, θ1 ∼ B(1, pi ) where pi =
exp(xi θ1)
1 + exp(xi θ1)
M2 : yi | xi
, θ2 ∼ B(1, qi ) where qi = Φ(xi
θ2)
Common parameterisation
Local reparameterisation strategy that rescales parameters of the
probit model M2 so that the MLE’s of both models coincide.
[Choudhuty et al., 2007]
Φ(xi
θ2) ≈
exp(kxi θ2)
1 + exp(kxi θ2)
and use best estimate of k to bring both parameters into coherency
(k0, k1) = (θ01/θ02, θ11/θ12) ,
reparameterise M1 and M2 as
M1 :yi | xi
, θ ∼ B(1, pi ) where pi =
exp(xi θ)
1 + exp(xi θ)
M2 :yi | xi
, θ ∼ B(1, qi ) where qi = Φ(xi
(κ−1
θ)) ,
with κ−1θ = (θ0/k0, θ1/k1).
Prior modelling
Under default g-prior
θ ∼ N2(0, n(XT
X)−1
)
full conditional posterior distributions given allocations
π(θ | y, X, ζ) ∝
exp i Iζi =1yi xi θ
i;ζi =1[1 + exp(xi θ)]
exp −θT
(XT
X)θ 2n
×
i;ζi =2
Φ(xi
(κ−1
θ))yi
(1 − Φ(xi
(κ−1
θ)))(1−yi )
hence posterior distribution clearly defined
Results
Logistic Probit
a0 α θ0 θ1
θ0
k0
θ1
k1
.1 .352 -4.06 .103 -2.51 .064
.2 .427 -4.03 .103 -2.49 .064
.3 .440 -4.02 .102 -2.49 .063
.4 .456 -4.01 .102 -2.48 .063
.5 .449 -4.05 .103 -2.51 .064
Histograms of posteriors of α in favour of logistic model where a0 = .1, .2, .3,
.4, .5 for (a) Pima dataset, (b) Data from logistic model, (c) Data from probit
model
Survival analysis
Testing hypothesis that data comes from a
1. log-Normal(φ, κ2),
2. Weibull(α, λ), or
3. log-Logistic(γ, δ)
distribution
Corresponding mixture given by the density
α1 exp{−(log x − φ)2
/2κ2
}/
√
2πxκ+
α2
α
λ
exp{−(x/λ)α
}((x/λ)α−1
+
α3(δ/γ)(x/γ)δ−1
/(1 + (x/γ)δ
)2
where α1 + α2 + α3 = 1
Survival analysis
Testing hypothesis that data comes from a
1. log-Normal(φ, κ2),
2. Weibull(α, λ), or
3. log-Logistic(γ, δ)
distribution
Corresponding mixture given by the density
α1 exp{−(log x − φ)2
/2κ2
}/
√
2πxκ+
α2
α
λ
exp{−(x/λ)α
}((x/λ)α−1
+
α3(δ/γ)(x/γ)δ−1
/(1 + (x/γ)δ
)2
where α1 + α2 + α3 = 1
Reparameterisation
Looking for common parameter(s):
φ = µ + γβ = ξ
σ2
= π2
β2
/6 = ζ2
π2
/3
where γ ≈ 0.5772 is Euler-Mascheroni constant.
Allows for a noninformative prior on the common location scale
parameter,
π(φ, σ2
) = 1/σ2
Reparameterisation
Looking for common parameter(s):
φ = µ + γβ = ξ
σ2
= π2
β2
/6 = ζ2
π2
/3
where γ ≈ 0.5772 is Euler-Mascheroni constant.
Allows for a noninformative prior on the common location scale
parameter,
π(φ, σ2
) = 1/σ2
Recovery
.01 0.1 1.0 10.0
0.860.880.900.920.940.960.981.00
.01 0.1 1.0 10.00.000.020.040.060.08
Boxplots of the posterior distributions of the Normal weight α1 under the two
scenarii: truth = Normal (left panel), truth = Gumbel (right panel), a0=0.01,
0.1, 1.0, 10.0 (from left to right in each panel) and n = 10, 000 simulated
observations.
Asymptotic consistency
Posterior consistency holds for mixture testing procedure [under
minor conditions]
Two different cases
the two models, M1 and M2, are well separated
model M1 is a submodel of M2.
Asymptotic consistency
Posterior consistency holds for mixture testing procedure [under
minor conditions]
Two different cases
the two models, M1 and M2, are well separated
model M1 is a submodel of M2.
Posterior concentration rate
Let π be the prior and xn = (x1, · · · , xn) a sample with true
density f ∗
proposition
Assume that, for all c > 0, there exist Θn ⊂ Θ1 × Θ2 and B > 0 such that
π [Θc
n] n−c
, Θn ⊂ { θ1 + θ2 nB
}
and that there exist H 0 and L, δ > 0 such that, for j = 1, 2,
sup
θ,θ ∈Θn
fj,θj
− fj,θj
1 LnH
θj − θj , θ = (θ1, θ2), θ = (θ1, θ2) ,
∀ θj − θ∗
j δ; KL(fj,θj
, fj,θ∗
j
) θj − θ∗
j .
Then, when f ∗
= fθ∗,α∗ , with α∗
∈ [0, 1], there exists M > 0 such that
π (α, θ); fθ,α − f ∗
1 > M log n/n|xn
= op(1) .
Separated models
Assumption: Models are separated, i.e. identifiability holds:
∀α, α ∈ [0, 1], ∀θj , θj , j = 1, 2 Pθ,α = Pθ ,α ⇒ α = α , θ = θ
Further
inf
θ1∈Θ1
inf
θ2∈Θ2
f1,θ1
− f2,θ2 1 > 0
and, for θ∗
j ∈ Θj , if Pθj
weakly converges to Pθ∗
j
, then
θj −→ θ∗
j
in the Euclidean topology
Separated models
Assumption: Models are separated, i.e. identifiability holds:
∀α, α ∈ [0, 1], ∀θj , θj , j = 1, 2 Pθ,α = Pθ ,α ⇒ α = α , θ = θ
theorem
Under above assumptions, then for all > 0,
π [|α − α∗
| > |xn
] = op(1)
Separated models
Assumption: Models are separated, i.e. identifiability holds:
∀α, α ∈ [0, 1], ∀θj , θj , j = 1, 2 Pθ,α = Pθ ,α ⇒ α = α , θ = θ
theorem
If
θj → fj,θj
is C2 around θ∗
j , j = 1, 2,
f1,θ∗
1
− f2,θ∗
2
, f1,θ∗
1
, f2,θ∗
2
are linearly independent in y and
there exists δ > 0 such that
f1,θ∗
1
, f2,θ∗
2
, sup
|θ1−θ∗
1 |<δ
|D2
f1,θ1
|, sup
|θ2−θ∗
2 |<δ
|D2
f2,θ2
| ∈ L1
then
π |α − α∗
| > M log n/n xn
= op(1).
Separated models
Assumption: Models are separated, i.e. identifiability holds:
∀α, α ∈ [0, 1], ∀θj , θj , j = 1, 2 Pθ,α = Pθ ,α ⇒ α = α , θ = θ
theorem allows for interpretation of α under the posterior: If data
xn is generated from model M1 then posterior on α concentrates
around α = 1
Embedded case
Here M1 is a submodel of M2, i.e.
θ2 = (θ1, ψ) and θ2 = (θ1, ψ0 = 0)
corresponds to f2,θ2 ∈ M1
Same posterior concentration rate
log n/n
for estimating α when α∗ ∈ (0, 1) and ψ∗ = 0.
Null case
Case where ψ∗ = 0, i.e., f ∗ is in model M1
Two possible paths to approximate f ∗: either α goes to 1
(path 1) or ψ goes to 0 (path 2)
New identifiability condition: Pθ,α = P∗ only if
α = 1, θ1 = θ∗
1, θ2 = (θ∗
1, ψ) or α 1, θ1 = θ∗
1, θ2 = (θ∗
1, 0)
Prior
π(α, θ) = πα(α)π1(θ1)πψ(ψ), θ2 = (θ1, ψ)
with common (prior on) θ1
Null case
Case where ψ∗ = 0, i.e., f ∗ is in model M1
Two possible paths to approximate f ∗: either α goes to 1
(path 1) or ψ goes to 0 (path 2)
New identifiability condition: Pθ,α = P∗ only if
α = 1, θ1 = θ∗
1, θ2 = (θ∗
1, ψ) or α 1, θ1 = θ∗
1, θ2 = (θ∗
1, 0)
Prior
π(α, θ) = πα(α)π1(θ1)πψ(ψ), θ2 = (θ1, ψ)
with common (prior on) θ1
Assumptions
[B1] Regularity: Assume that θ1 → f1,θ1 and θ2 → f2,θ2 are 3
times continuously differentiable and that
F∗
¯f 3
1,θ∗
1
f 3
1,θ∗
1
< +∞, ¯f1,θ∗
1
= sup
|θ1−θ∗
1 |<δ
f1,θ1
, f 1,θ∗
1
= inf
|θ1−θ∗
1 |<δ
f1,θ1
F∗
sup|θ1−θ∗
1 |<δ | f1,θ∗
1
|3
f 3
1,θ∗
1
< +∞, F∗
| f1,θ∗
1
|4
f 4
1,θ∗
1
< +∞,
F∗
sup|θ1−θ∗
1 |<δ |D2
f1,θ∗
1
|2
f 2
1,θ∗
1
< +∞, F∗
sup|θ1−θ∗
1 |<δ |D3
f1,θ∗
1
|
f 1,θ∗
1
< +∞
Assumptions
[B2] Integrability: There exists
S0 ⊂ S ∩ {|ψ| > δ0}
for some positive δ0 and satisfying Leb(S0) > 0, and such that for
all ψ ∈ S0,
F∗
sup|θ1−θ∗
1 |<δ f2,θ1,ψ
f 4
1,θ∗
1
< +∞, F∗
sup|θ1−θ∗
1 |<δ f 3
2,θ1,ψ
f 3
1,θ1∗
< +∞,
Assumptions
[B3] Stronger identifiability: Set
f2,θ∗
1,ψ∗ (x) = θ1 f2,θ∗
1,ψ∗ (x)T
, ψf2,θ∗
1,ψ∗ (x)T T
.
Then for all ψ ∈ S with ψ = 0, if η0 ∈ R, η1 ∈ Rd1
η0(f1,θ∗
1
− f2,θ∗
1,ψ) + ηT
1 θ1 f1,θ∗
1
= 0 ⇔ η1 = 0, η2 = 0
Consistency
theorem
Given the mixture fθ1,ψ,α = αf1,θ1 + (1 − α)f2,θ1,ψ and a sample
xn = (x1, · · · , xn) issued from f1,θ∗
1
, under assumptions B1 − B3,
and an M > 0 such that
π (α, θ); fθ,α − f ∗
1 > M log n/n|xn
= op(1).
If α ∼ B(a1, a2), with a2 < d2, and if the prior πθ1,ψ is absolutely
continuous with positive and continuous density at (θ∗
1, 0), then for
Mn −→ ∞
π |α − α∗
| > Mn(log n)γ
/
√
n|xn
= op(1), γ = max((d1 + a2)/(d2 − a2), 1)/2,

More Related Content

What's hot

testing as a mixture estimation problem
testing as a mixture estimation problemtesting as a mixture estimation problem
testing as a mixture estimation problemChristian Robert
 
accurate ABC Oliver Ratmann
accurate ABC Oliver Ratmannaccurate ABC Oliver Ratmann
accurate ABC Oliver Ratmannolli0601
 
An overview of Bayesian testing
An overview of Bayesian testingAn overview of Bayesian testing
An overview of Bayesian testingChristian Robert
 
random forests for ABC model choice and parameter estimation
random forests for ABC model choice and parameter estimationrandom forests for ABC model choice and parameter estimation
random forests for ABC model choice and parameter estimationChristian Robert
 
Approximate Bayesian model choice via random forests
Approximate Bayesian model choice via random forestsApproximate Bayesian model choice via random forests
Approximate Bayesian model choice via random forestsChristian Robert
 
Multiple estimators for Monte Carlo approximations
Multiple estimators for Monte Carlo approximationsMultiple estimators for Monte Carlo approximations
Multiple estimators for Monte Carlo approximationsChristian Robert
 
NCE, GANs & VAEs (and maybe BAC)
NCE, GANs & VAEs (and maybe BAC)NCE, GANs & VAEs (and maybe BAC)
NCE, GANs & VAEs (and maybe BAC)Christian Robert
 
Coordinate sampler : A non-reversible Gibbs-like sampler
Coordinate sampler : A non-reversible Gibbs-like samplerCoordinate sampler : A non-reversible Gibbs-like sampler
Coordinate sampler : A non-reversible Gibbs-like samplerChristian Robert
 
ABC based on Wasserstein distances
ABC based on Wasserstein distancesABC based on Wasserstein distances
ABC based on Wasserstein distancesChristian Robert
 
Statistics (1): estimation, Chapter 2: Empirical distribution and bootstrap
Statistics (1): estimation, Chapter 2: Empirical distribution and bootstrapStatistics (1): estimation, Chapter 2: Empirical distribution and bootstrap
Statistics (1): estimation, Chapter 2: Empirical distribution and bootstrapChristian Robert
 
Laplace's Demon: seminar #1
Laplace's Demon: seminar #1Laplace's Demon: seminar #1
Laplace's Demon: seminar #1Christian Robert
 
Approximating Bayes Factors
Approximating Bayes FactorsApproximating Bayes Factors
Approximating Bayes FactorsChristian Robert
 
CISEA 2019: ABC consistency and convergence
CISEA 2019: ABC consistency and convergenceCISEA 2019: ABC consistency and convergence
CISEA 2019: ABC consistency and convergenceChristian Robert
 
ABC convergence under well- and mis-specified models
ABC convergence under well- and mis-specified modelsABC convergence under well- and mis-specified models
ABC convergence under well- and mis-specified modelsChristian Robert
 
Convergence of ABC methods
Convergence of ABC methodsConvergence of ABC methods
Convergence of ABC methodsChristian Robert
 
Inference in generative models using the Wasserstein distance [[INI]
Inference in generative models using the Wasserstein distance [[INI]Inference in generative models using the Wasserstein distance [[INI]
Inference in generative models using the Wasserstein distance [[INI]Christian Robert
 

What's hot (20)

Big model, big data
Big model, big dataBig model, big data
Big model, big data
 
testing as a mixture estimation problem
testing as a mixture estimation problemtesting as a mixture estimation problem
testing as a mixture estimation problem
 
accurate ABC Oliver Ratmann
accurate ABC Oliver Ratmannaccurate ABC Oliver Ratmann
accurate ABC Oliver Ratmann
 
An overview of Bayesian testing
An overview of Bayesian testingAn overview of Bayesian testing
An overview of Bayesian testing
 
random forests for ABC model choice and parameter estimation
random forests for ABC model choice and parameter estimationrandom forests for ABC model choice and parameter estimation
random forests for ABC model choice and parameter estimation
 
Approximate Bayesian model choice via random forests
Approximate Bayesian model choice via random forestsApproximate Bayesian model choice via random forests
Approximate Bayesian model choice via random forests
 
Multiple estimators for Monte Carlo approximations
Multiple estimators for Monte Carlo approximationsMultiple estimators for Monte Carlo approximations
Multiple estimators for Monte Carlo approximations
 
NCE, GANs & VAEs (and maybe BAC)
NCE, GANs & VAEs (and maybe BAC)NCE, GANs & VAEs (and maybe BAC)
NCE, GANs & VAEs (and maybe BAC)
 
Coordinate sampler : A non-reversible Gibbs-like sampler
Coordinate sampler : A non-reversible Gibbs-like samplerCoordinate sampler : A non-reversible Gibbs-like sampler
Coordinate sampler : A non-reversible Gibbs-like sampler
 
ABC workshop: 17w5025
ABC workshop: 17w5025ABC workshop: 17w5025
ABC workshop: 17w5025
 
ABC based on Wasserstein distances
ABC based on Wasserstein distancesABC based on Wasserstein distances
ABC based on Wasserstein distances
 
Statistics (1): estimation, Chapter 2: Empirical distribution and bootstrap
Statistics (1): estimation, Chapter 2: Empirical distribution and bootstrapStatistics (1): estimation, Chapter 2: Empirical distribution and bootstrap
Statistics (1): estimation, Chapter 2: Empirical distribution and bootstrap
 
Laplace's Demon: seminar #1
Laplace's Demon: seminar #1Laplace's Demon: seminar #1
Laplace's Demon: seminar #1
 
the ABC of ABC
the ABC of ABCthe ABC of ABC
the ABC of ABC
 
Approximating Bayes Factors
Approximating Bayes FactorsApproximating Bayes Factors
Approximating Bayes Factors
 
CISEA 2019: ABC consistency and convergence
CISEA 2019: ABC consistency and convergenceCISEA 2019: ABC consistency and convergence
CISEA 2019: ABC consistency and convergence
 
ABC convergence under well- and mis-specified models
ABC convergence under well- and mis-specified modelsABC convergence under well- and mis-specified models
ABC convergence under well- and mis-specified models
 
Convergence of ABC methods
Convergence of ABC methodsConvergence of ABC methods
Convergence of ABC methods
 
ABC-Gibbs
ABC-GibbsABC-Gibbs
ABC-Gibbs
 
Inference in generative models using the Wasserstein distance [[INI]
Inference in generative models using the Wasserstein distance [[INI]Inference in generative models using the Wasserstein distance [[INI]
Inference in generative models using the Wasserstein distance [[INI]
 

Similar to Tutorial on testing at O'Bayes 2015, Valencià, June 1, 2015

Reading Testing a point-null hypothesis, by Jiahuan Li, Feb. 25, 2013
Reading Testing a point-null hypothesis, by Jiahuan Li, Feb. 25, 2013Reading Testing a point-null hypothesis, by Jiahuan Li, Feb. 25, 2013
Reading Testing a point-null hypothesis, by Jiahuan Li, Feb. 25, 2013Christian Robert
 
Equational axioms for probability calculus and modelling of Likelihood ratio ...
Equational axioms for probability calculus and modelling of Likelihood ratio ...Equational axioms for probability calculus and modelling of Likelihood ratio ...
Equational axioms for probability calculus and modelling of Likelihood ratio ...Advanced-Concepts-Team
 
Otter 2014-12-15-01-slideshare
Otter 2014-12-15-01-slideshareOtter 2014-12-15-01-slideshare
Otter 2014-12-15-01-slideshareRuo Ando
 
Cs229 notes8
Cs229 notes8Cs229 notes8
Cs229 notes8VuTran231
 
Ian.petrow【transcendental number theory】.
Ian.petrow【transcendental number theory】.Ian.petrow【transcendental number theory】.
Ian.petrow【transcendental number theory】.Tong Leung
 
Boolean Programs and Quantified Propositional Proof System -
Boolean Programs and Quantified Propositional Proof System - Boolean Programs and Quantified Propositional Proof System -
Boolean Programs and Quantified Propositional Proof System - Michael Soltys
 
Excursion 4 Tour II: Rejection Fallacies: Whose Exaggerating What?
Excursion 4 Tour II: Rejection Fallacies: Whose Exaggerating What?Excursion 4 Tour II: Rejection Fallacies: Whose Exaggerating What?
Excursion 4 Tour II: Rejection Fallacies: Whose Exaggerating What?jemille6
 
Probability cheatsheet
Probability cheatsheetProbability cheatsheet
Probability cheatsheetSuvrat Mishra
 
Course on Bayesian computational methods
Course on Bayesian computational methodsCourse on Bayesian computational methods
Course on Bayesian computational methodsChristian Robert
 
San Antonio short course, March 2010
San Antonio short course, March 2010San Antonio short course, March 2010
San Antonio short course, March 2010Christian Robert
 
Talk at CIRM on Poisson equation and debiasing techniques
Talk at CIRM on Poisson equation and debiasing techniquesTalk at CIRM on Poisson equation and debiasing techniques
Talk at CIRM on Poisson equation and debiasing techniquesPierre Jacob
 
Probability cheatsheet
Probability cheatsheetProbability cheatsheet
Probability cheatsheetJoachim Gwoke
 
Rolle's theorem, mean value theorem
Rolle's theorem, mean value theoremRolle's theorem, mean value theorem
Rolle's theorem, mean value theoremTarun Gehlot
 
Fuzzy logic andits Applications
Fuzzy logic andits ApplicationsFuzzy logic andits Applications
Fuzzy logic andits ApplicationsDrATAMILARASIMCA
 
Probability/Statistics Lecture Notes 4: Hypothesis Testing
Probability/Statistics Lecture Notes 4: Hypothesis TestingProbability/Statistics Lecture Notes 4: Hypothesis Testing
Probability/Statistics Lecture Notes 4: Hypothesis Testingjemille6
 
Unit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VEC
Unit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VECUnit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VEC
Unit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VECsundarKanagaraj1
 

Similar to Tutorial on testing at O'Bayes 2015, Valencià, June 1, 2015 (20)

Reading Testing a point-null hypothesis, by Jiahuan Li, Feb. 25, 2013
Reading Testing a point-null hypothesis, by Jiahuan Li, Feb. 25, 2013Reading Testing a point-null hypothesis, by Jiahuan Li, Feb. 25, 2013
Reading Testing a point-null hypothesis, by Jiahuan Li, Feb. 25, 2013
 
Equational axioms for probability calculus and modelling of Likelihood ratio ...
Equational axioms for probability calculus and modelling of Likelihood ratio ...Equational axioms for probability calculus and modelling of Likelihood ratio ...
Equational axioms for probability calculus and modelling of Likelihood ratio ...
 
Otter 2014-12-15-01-slideshare
Otter 2014-12-15-01-slideshareOtter 2014-12-15-01-slideshare
Otter 2014-12-15-01-slideshare
 
Probability Cheatsheet.pdf
Probability Cheatsheet.pdfProbability Cheatsheet.pdf
Probability Cheatsheet.pdf
 
Cs229 notes8
Cs229 notes8Cs229 notes8
Cs229 notes8
 
Ian.petrow【transcendental number theory】.
Ian.petrow【transcendental number theory】.Ian.petrow【transcendental number theory】.
Ian.petrow【transcendental number theory】.
 
Boolean Programs and Quantified Propositional Proof System -
Boolean Programs and Quantified Propositional Proof System - Boolean Programs and Quantified Propositional Proof System -
Boolean Programs and Quantified Propositional Proof System -
 
Excursion 4 Tour II: Rejection Fallacies: Whose Exaggerating What?
Excursion 4 Tour II: Rejection Fallacies: Whose Exaggerating What?Excursion 4 Tour II: Rejection Fallacies: Whose Exaggerating What?
Excursion 4 Tour II: Rejection Fallacies: Whose Exaggerating What?
 
Davezies
DaveziesDavezies
Davezies
 
Probability cheatsheet
Probability cheatsheetProbability cheatsheet
Probability cheatsheet
 
Course on Bayesian computational methods
Course on Bayesian computational methodsCourse on Bayesian computational methods
Course on Bayesian computational methods
 
San Antonio short course, March 2010
San Antonio short course, March 2010San Antonio short course, March 2010
San Antonio short course, March 2010
 
JSM 2011 round table
JSM 2011 round tableJSM 2011 round table
JSM 2011 round table
 
JSM 2011 round table
JSM 2011 round tableJSM 2011 round table
JSM 2011 round table
 
Talk at CIRM on Poisson equation and debiasing techniques
Talk at CIRM on Poisson equation and debiasing techniquesTalk at CIRM on Poisson equation and debiasing techniques
Talk at CIRM on Poisson equation and debiasing techniques
 
Probability cheatsheet
Probability cheatsheetProbability cheatsheet
Probability cheatsheet
 
Rolle's theorem, mean value theorem
Rolle's theorem, mean value theoremRolle's theorem, mean value theorem
Rolle's theorem, mean value theorem
 
Fuzzy logic andits Applications
Fuzzy logic andits ApplicationsFuzzy logic andits Applications
Fuzzy logic andits Applications
 
Probability/Statistics Lecture Notes 4: Hypothesis Testing
Probability/Statistics Lecture Notes 4: Hypothesis TestingProbability/Statistics Lecture Notes 4: Hypothesis Testing
Probability/Statistics Lecture Notes 4: Hypothesis Testing
 
Unit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VEC
Unit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VECUnit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VEC
Unit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VEC
 

More from Christian Robert

Asymptotics of ABC, lecture, Collège de France
Asymptotics of ABC, lecture, Collège de FranceAsymptotics of ABC, lecture, Collège de France
Asymptotics of ABC, lecture, Collège de FranceChristian Robert
 
Workshop in honour of Don Poskitt and Gael Martin
Workshop in honour of Don Poskitt and Gael MartinWorkshop in honour of Don Poskitt and Gael Martin
Workshop in honour of Don Poskitt and Gael MartinChristian Robert
 
How many components in a mixture?
How many components in a mixture?How many components in a mixture?
How many components in a mixture?Christian Robert
 
Testing for mixtures at BNP 13
Testing for mixtures at BNP 13Testing for mixtures at BNP 13
Testing for mixtures at BNP 13Christian Robert
 
Inferring the number of components: dream or reality?
Inferring the number of components: dream or reality?Inferring the number of components: dream or reality?
Inferring the number of components: dream or reality?Christian Robert
 
Testing for mixtures by seeking components
Testing for mixtures by seeking componentsTesting for mixtures by seeking components
Testing for mixtures by seeking componentsChristian Robert
 
discussion on Bayesian restricted likelihood
discussion on Bayesian restricted likelihooddiscussion on Bayesian restricted likelihood
discussion on Bayesian restricted likelihoodChristian Robert
 
Likelihood-free Design: a discussion
Likelihood-free Design: a discussionLikelihood-free Design: a discussion
Likelihood-free Design: a discussionChristian Robert
 
a discussion of Chib, Shin, and Simoni (2017-8) Bayesian moment models
a discussion of Chib, Shin, and Simoni (2017-8) Bayesian moment modelsa discussion of Chib, Shin, and Simoni (2017-8) Bayesian moment models
a discussion of Chib, Shin, and Simoni (2017-8) Bayesian moment modelsChristian Robert
 
Poster for Bayesian Statistics in the Big Data Era conference
Poster for Bayesian Statistics in the Big Data Era conferencePoster for Bayesian Statistics in the Big Data Era conference
Poster for Bayesian Statistics in the Big Data Era conferenceChristian Robert
 
short course at CIRM, Bayesian Masterclass, October 2018
short course at CIRM, Bayesian Masterclass, October 2018short course at CIRM, Bayesian Masterclass, October 2018
short course at CIRM, Bayesian Masterclass, October 2018Christian Robert
 
ABC with Wasserstein distances
ABC with Wasserstein distancesABC with Wasserstein distances
ABC with Wasserstein distancesChristian Robert
 
prior selection for mixture estimation
prior selection for mixture estimationprior selection for mixture estimation
prior selection for mixture estimationChristian Robert
 

More from Christian Robert (20)

Asymptotics of ABC, lecture, Collège de France
Asymptotics of ABC, lecture, Collège de FranceAsymptotics of ABC, lecture, Collège de France
Asymptotics of ABC, lecture, Collège de France
 
Workshop in honour of Don Poskitt and Gael Martin
Workshop in honour of Don Poskitt and Gael MartinWorkshop in honour of Don Poskitt and Gael Martin
Workshop in honour of Don Poskitt and Gael Martin
 
discussion of ICML23.pdf
discussion of ICML23.pdfdiscussion of ICML23.pdf
discussion of ICML23.pdf
 
How many components in a mixture?
How many components in a mixture?How many components in a mixture?
How many components in a mixture?
 
restore.pdf
restore.pdfrestore.pdf
restore.pdf
 
Testing for mixtures at BNP 13
Testing for mixtures at BNP 13Testing for mixtures at BNP 13
Testing for mixtures at BNP 13
 
Inferring the number of components: dream or reality?
Inferring the number of components: dream or reality?Inferring the number of components: dream or reality?
Inferring the number of components: dream or reality?
 
CDT 22 slides.pdf
CDT 22 slides.pdfCDT 22 slides.pdf
CDT 22 slides.pdf
 
Testing for mixtures by seeking components
Testing for mixtures by seeking componentsTesting for mixtures by seeking components
Testing for mixtures by seeking components
 
discussion on Bayesian restricted likelihood
discussion on Bayesian restricted likelihooddiscussion on Bayesian restricted likelihood
discussion on Bayesian restricted likelihood
 
ABC-Gibbs
ABC-GibbsABC-Gibbs
ABC-Gibbs
 
eugenics and statistics
eugenics and statisticseugenics and statistics
eugenics and statistics
 
ABC-Gibbs
ABC-GibbsABC-Gibbs
ABC-Gibbs
 
asymptotics of ABC
asymptotics of ABCasymptotics of ABC
asymptotics of ABC
 
Likelihood-free Design: a discussion
Likelihood-free Design: a discussionLikelihood-free Design: a discussion
Likelihood-free Design: a discussion
 
a discussion of Chib, Shin, and Simoni (2017-8) Bayesian moment models
a discussion of Chib, Shin, and Simoni (2017-8) Bayesian moment modelsa discussion of Chib, Shin, and Simoni (2017-8) Bayesian moment models
a discussion of Chib, Shin, and Simoni (2017-8) Bayesian moment models
 
Poster for Bayesian Statistics in the Big Data Era conference
Poster for Bayesian Statistics in the Big Data Era conferencePoster for Bayesian Statistics in the Big Data Era conference
Poster for Bayesian Statistics in the Big Data Era conference
 
short course at CIRM, Bayesian Masterclass, October 2018
short course at CIRM, Bayesian Masterclass, October 2018short course at CIRM, Bayesian Masterclass, October 2018
short course at CIRM, Bayesian Masterclass, October 2018
 
ABC with Wasserstein distances
ABC with Wasserstein distancesABC with Wasserstein distances
ABC with Wasserstein distances
 
prior selection for mixture estimation
prior selection for mixture estimationprior selection for mixture estimation
prior selection for mixture estimation
 

Recently uploaded

RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024suelcarter1
 
Bureau of Indian Standards Specification of Shampoo.pptx
Bureau of Indian Standards Specification of Shampoo.pptxBureau of Indian Standards Specification of Shampoo.pptx
Bureau of Indian Standards Specification of Shampoo.pptxkastureyashashree
 
Application of Foraminiferal Ecology- Rahul.pptx
Application of Foraminiferal Ecology- Rahul.pptxApplication of Foraminiferal Ecology- Rahul.pptx
Application of Foraminiferal Ecology- Rahul.pptxRahulVishwakarma71547
 
Applied Biochemistry feedback_M Ahwad 2023.docx
Applied Biochemistry feedback_M Ahwad 2023.docxApplied Biochemistry feedback_M Ahwad 2023.docx
Applied Biochemistry feedback_M Ahwad 2023.docxmarwaahmad357
 
Basic Concepts in Pharmacology in molecular .pptx
Basic Concepts in Pharmacology in molecular  .pptxBasic Concepts in Pharmacology in molecular  .pptx
Basic Concepts in Pharmacology in molecular .pptxVijayaKumarR28
 
Human brain.. It's parts and function.
Human brain.. It's parts and function. Human brain.. It's parts and function.
Human brain.. It's parts and function. MUKTA MANJARI SAHOO
 
biosynthesis of the cell wall and antibiotics
biosynthesis of the cell wall and antibioticsbiosynthesis of the cell wall and antibiotics
biosynthesis of the cell wall and antibioticsSafaFallah
 
geometric quantization on coadjoint orbits
geometric quantization on coadjoint orbitsgeometric quantization on coadjoint orbits
geometric quantization on coadjoint orbitsHassan Jolany
 
Controlling Parameters of Carbonate platform Environment
Controlling Parameters of Carbonate platform EnvironmentControlling Parameters of Carbonate platform Environment
Controlling Parameters of Carbonate platform EnvironmentRahulVishwakarma71547
 
Identification of Superclusters and Their Properties in the Sloan Digital Sky...
Identification of Superclusters and Their Properties in the Sloan Digital Sky...Identification of Superclusters and Their Properties in the Sloan Digital Sky...
Identification of Superclusters and Their Properties in the Sloan Digital Sky...Sérgio Sacani
 
Role of herbs in hair care Amla and heena.pptx
Role of herbs in hair care  Amla and  heena.pptxRole of herbs in hair care  Amla and  heena.pptx
Role of herbs in hair care Amla and heena.pptxVaishnaviAware
 
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdfPests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdfPirithiRaju
 
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptxTHE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptxAkinrotimiOluwadunsi
 
Pests of Redgram_Identification, Binomics_Dr.UPR
Pests of Redgram_Identification, Binomics_Dr.UPRPests of Redgram_Identification, Binomics_Dr.UPR
Pests of Redgram_Identification, Binomics_Dr.UPRPirithiRaju
 
Role of Herbs in Cosmetics in Cosmetic Science.
Role of Herbs in Cosmetics in Cosmetic Science.Role of Herbs in Cosmetics in Cosmetic Science.
Role of Herbs in Cosmetics in Cosmetic Science.ShwetaHattimare
 
M.Pharm - Question Bank - Drug Delivery Systems
M.Pharm - Question Bank - Drug Delivery SystemsM.Pharm - Question Bank - Drug Delivery Systems
M.Pharm - Question Bank - Drug Delivery SystemsSumathi Arumugam
 
PSP3 employability assessment form .docx
PSP3 employability assessment form .docxPSP3 employability assessment form .docx
PSP3 employability assessment form .docxmarwaahmad357
 
soft skills question paper set for bba ca
soft skills question paper set for bba casoft skills question paper set for bba ca
soft skills question paper set for bba caohsadfeeling
 
Pests of ragi_Identification, Binomics_Dr.UPR
Pests of ragi_Identification, Binomics_Dr.UPRPests of ragi_Identification, Binomics_Dr.UPR
Pests of ragi_Identification, Binomics_Dr.UPRPirithiRaju
 

Recently uploaded (20)

RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024
 
Bureau of Indian Standards Specification of Shampoo.pptx
Bureau of Indian Standards Specification of Shampoo.pptxBureau of Indian Standards Specification of Shampoo.pptx
Bureau of Indian Standards Specification of Shampoo.pptx
 
Application of Foraminiferal Ecology- Rahul.pptx
Application of Foraminiferal Ecology- Rahul.pptxApplication of Foraminiferal Ecology- Rahul.pptx
Application of Foraminiferal Ecology- Rahul.pptx
 
Applied Biochemistry feedback_M Ahwad 2023.docx
Applied Biochemistry feedback_M Ahwad 2023.docxApplied Biochemistry feedback_M Ahwad 2023.docx
Applied Biochemistry feedback_M Ahwad 2023.docx
 
Basic Concepts in Pharmacology in molecular .pptx
Basic Concepts in Pharmacology in molecular  .pptxBasic Concepts in Pharmacology in molecular  .pptx
Basic Concepts in Pharmacology in molecular .pptx
 
Human brain.. It's parts and function.
Human brain.. It's parts and function. Human brain.. It's parts and function.
Human brain.. It's parts and function.
 
biosynthesis of the cell wall and antibiotics
biosynthesis of the cell wall and antibioticsbiosynthesis of the cell wall and antibiotics
biosynthesis of the cell wall and antibiotics
 
geometric quantization on coadjoint orbits
geometric quantization on coadjoint orbitsgeometric quantization on coadjoint orbits
geometric quantization on coadjoint orbits
 
Controlling Parameters of Carbonate platform Environment
Controlling Parameters of Carbonate platform EnvironmentControlling Parameters of Carbonate platform Environment
Controlling Parameters of Carbonate platform Environment
 
Identification of Superclusters and Their Properties in the Sloan Digital Sky...
Identification of Superclusters and Their Properties in the Sloan Digital Sky...Identification of Superclusters and Their Properties in the Sloan Digital Sky...
Identification of Superclusters and Their Properties in the Sloan Digital Sky...
 
Role of herbs in hair care Amla and heena.pptx
Role of herbs in hair care  Amla and  heena.pptxRole of herbs in hair care  Amla and  heena.pptx
Role of herbs in hair care Amla and heena.pptx
 
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdfPests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
 
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptxTHE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
 
Pests of Redgram_Identification, Binomics_Dr.UPR
Pests of Redgram_Identification, Binomics_Dr.UPRPests of Redgram_Identification, Binomics_Dr.UPR
Pests of Redgram_Identification, Binomics_Dr.UPR
 
Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...
Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...
Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...
 
Role of Herbs in Cosmetics in Cosmetic Science.
Role of Herbs in Cosmetics in Cosmetic Science.Role of Herbs in Cosmetics in Cosmetic Science.
Role of Herbs in Cosmetics in Cosmetic Science.
 
M.Pharm - Question Bank - Drug Delivery Systems
M.Pharm - Question Bank - Drug Delivery SystemsM.Pharm - Question Bank - Drug Delivery Systems
M.Pharm - Question Bank - Drug Delivery Systems
 
PSP3 employability assessment form .docx
PSP3 employability assessment form .docxPSP3 employability assessment form .docx
PSP3 employability assessment form .docx
 
soft skills question paper set for bba ca
soft skills question paper set for bba casoft skills question paper set for bba ca
soft skills question paper set for bba ca
 
Pests of ragi_Identification, Binomics_Dr.UPR
Pests of ragi_Identification, Binomics_Dr.UPRPests of ragi_Identification, Binomics_Dr.UPR
Pests of ragi_Identification, Binomics_Dr.UPR
 

Tutorial on testing at O'Bayes 2015, Valencià, June 1, 2015

  • 1. Several perspectives and solutions on Bayesian testing of hypotheses Christian P. Robert Universit´e Paris-Dauphine, Paris & University of Warwick, Coventry
  • 2. Outline Significance tests: one new parameter Noninformative solutions Jeffreys-Lindley paradox Deviance (information criterion) Testing under incomplete information Testing via mixtures
  • 3. “Significance tests: one new parameter” Significance tests: one new parameter Bayesian tests Bayes factors Improper priors for tests Conclusion Noninformative solutions Jeffreys-Lindley paradox Deviance (information criterion) Testing under incomplete information Testing via mixtures
  • 4. Fundamental setting Is the new parameter supported by the observations or is any variation expressible by it better interpreted as random? Thus we must set two hypotheses for comparison, the more complicated having the smaller initial probability (Jeffreys, ToP, V, §5.0) ...compare a specially suggested value of a new parameter, often 0 [q], with the aggregate of other possible values [q ]. We shall call q the null hypothesis and q the alternative hypothesis [and] we must take P(q|H) = P(q |H) = 1/2 .
  • 5. Construction of Bayes tests Definition (Test) Given an hypothesis H0 : θ ∈ Θ0 on the parameter θ ∈ Θ0 of a statistical model, a test is a statistical procedure that takes its values in {0, 1}.
  • 6. Type–one and type–two errors Associated with the risk R(θ, δ) = Eθ[aL(θ, δ(x))] = Pθ(δ(x) = 0) if θ ∈ Θ0, Pθ(δ(x) = 1) otherwise, Theorem (Bayes test) The Bayes estimator associated with π and with the 0 − 1 loss is δπ (x) = 1 if P(θ ∈ Θ0|x) > P(θ ∈ Θ0|x), 0 otherwise,
  • 7. Type–one and type–two errors Associated with the risk R(θ, δ) = Eθ[aL(θ, δ(x))] = Pθ(δ(x) = 0) if θ ∈ Θ0, Pθ(δ(x) = 1) otherwise, Theorem (Bayes test) The Bayes estimator associated with π and with the 0 − 1 loss is δπ (x) = 1 if P(θ ∈ Θ0|x) > P(θ ∈ Θ0|x), 0 otherwise,
  • 8. Jeffreys’ example (§5.0) Testing whether the mean α of a normal observation is zero: P(q|aH) ∝ exp − a2 2s2 P(q dα|aH) ∝ exp − (a − α)2 2s2 f (α)dα P(q |aH) ∝ exp − (a − α)2 2s2 f (α)dα
  • 9. A (small) point of contention Jeffreys asserts Suppose that there is one old parameter α; the new parameter is β and is 0 on q. In q we could replace α by α , any function of α and β: but to make it explicit that q reduces to q when β = 0 we shall require that α = α when β = 0 (V, §5.0). This amounts to assume identical parameters in both models, a controversial principle for model choice or at the very best to make α and β dependent a priori, a choice contradicted by the next paragraph in ToP
  • 10. A (small) point of contention Jeffreys asserts Suppose that there is one old parameter α; the new parameter is β and is 0 on q. In q we could replace α by α , any function of α and β: but to make it explicit that q reduces to q when β = 0 we shall require that α = α when β = 0 (V, §5.0). This amounts to assume identical parameters in both models, a controversial principle for model choice or at the very best to make α and β dependent a priori, a choice contradicted by the next paragraph in ToP
  • 11. Orthogonal parameters If I(α, β) = gαα 0 0 gββ , α and β orthogonal, but not [a posteriori] independent, contrary to ToP assertions ...the result will be nearly independent on previous information on old parameters (V, §5.01). and K = 1 f (b, a) ngββ 2π exp − 1 2 ngββb2 [where] h(α) is irrelevant (V, §5.01)
  • 12. Orthogonal parameters If I(α, β) = gαα 0 0 gββ , α and β orthogonal, but not [a posteriori] independent, contrary to ToP assertions ...the result will be nearly independent on previous information on old parameters (V, §5.01). and K = 1 f (b, a) ngββ 2π exp − 1 2 ngββb2 [where] h(α) is irrelevant (V, §5.01)
  • 13. Acknowledgement in ToP In practice it is rather unusual for a set of parameters to arise in such a way that each can be treated as irrelevant to the presence of any other. More usual cases are (...) where some parameters are so closely associated that one could hardly occur without the others (V, §5.04).
  • 14. Generalisation Theorem (Optimal Bayes decision) Under the 0 − 1 loss function L(θ, d) =    0 if d = IΘ0 (θ) a0 if d = 1 and θ ∈ Θ0 a1 if d = 0 and θ ∈ Θ0 the Bayes procedure is δπ (x) = 1 if Prπ (θ ∈ Θ0|x) a0/(a0 + a1) 0 otherwise
  • 15. Generalisation Theorem (Optimal Bayes decision) Under the 0 − 1 loss function L(θ, d) =    0 if d = IΘ0 (θ) a0 if d = 1 and θ ∈ Θ0 a1 if d = 0 and θ ∈ Θ0 the Bayes procedure is δπ (x) = 1 if Prπ (θ ∈ Θ0|x) a0/(a0 + a1) 0 otherwise
  • 16. Bound comparison Determination of a0/a1 depends on consequences of “wrong decision” under both circumstances Often difficult to assess in practice and replacement with “golden” default bounds like .05, biased towards H0
  • 17. Bound comparison Determination of a0/a1 depends on consequences of “wrong decision” under both circumstances Often difficult to assess in practice and replacement with “golden” default bounds like .05, biased towards H0
  • 18. A function of posterior probabilities Definition (Bayes factors) For hypotheses H0 : θ ∈ Θ0 vs. Ha : θ ∈ Θ0 B01 = π(Θ0|x) π(Θc 0|x) π(Θ0) π(Θc 0) = Θ0 f (x|θ)π0(θ)dθ Θc 0 f (x|θ)π1(θ)dθ [Good, 1958 & ToP, V, §5.01] Equivalent to Bayes rule: acceptance if B01 > {(1 − π(Θ0))/a1}/{π(Θ0)/a0}
  • 19. Self-contained concept Outside decision-theoretic environment: eliminates choice of π(Θ0) but depends on the choice of (π0, π1) Bayesian/marginal equivalent to the likelihood ratio Jeffreys’ scale of evidence: if log10(Bπ 10) between 0 and 0.5, evidence against H0 weak, if log10(Bπ 10) 0.5 and 1, evidence substantial, if log10(Bπ 10) 1 and 2, evidence strong and if log10(Bπ 10) above 2, evidence decisive
  • 20. Self-contained concept Outside decision-theoretic environment: eliminates choice of π(Θ0) but depends on the choice of (π0, π1) Bayesian/marginal equivalent to the likelihood ratio Jeffreys’ scale of evidence: if log10(Bπ 10) between 0 and 0.5, evidence against H0 weak, if log10(Bπ 10) 0.5 and 1, evidence substantial, if log10(Bπ 10) 1 and 2, evidence strong and if log10(Bπ 10) above 2, evidence decisive
  • 21. Self-contained concept Outside decision-theoretic environment: eliminates choice of π(Θ0) but depends on the choice of (π0, π1) Bayesian/marginal equivalent to the likelihood ratio Jeffreys’ scale of evidence: if log10(Bπ 10) between 0 and 0.5, evidence against H0 weak, if log10(Bπ 10) 0.5 and 1, evidence substantial, if log10(Bπ 10) 1 and 2, evidence strong and if log10(Bπ 10) above 2, evidence decisive
  • 22. Self-contained concept Outside decision-theoretic environment: eliminates choice of π(Θ0) but depends on the choice of (π0, π1) Bayesian/marginal equivalent to the likelihood ratio Jeffreys’ scale of evidence: if log10(Bπ 10) between 0 and 0.5, evidence against H0 weak, if log10(Bπ 10) 0.5 and 1, evidence substantial, if log10(Bπ 10) 1 and 2, evidence strong and if log10(Bπ 10) above 2, evidence decisive
  • 23. A major modification When the null hypothesis is supported by a set of measure 0 against Lebesgue measure, π(Θ0) = 0 for an absolutely continuous prior distribution [End of the story?!] Suppose we are considering whether a location parameter α is 0. The estimation prior probability for it is uniform and we should have to take f (α) = 0 and K[= B10] would always be infinite (V, §5.02)
  • 24. A major modification When the null hypothesis is supported by a set of measure 0 against Lebesgue measure, π(Θ0) = 0 for an absolutely continuous prior distribution [End of the story?!] Suppose we are considering whether a location parameter α is 0. The estimation prior probability for it is uniform and we should have to take f (α) = 0 and K[= B10] would always be infinite (V, §5.02)
  • 25. Point null refurbishment Requirement Defined prior distributions under both assumptions, π0(θ) ∝ π(θ)IΘ0 (θ), π1(θ) ∝ π(θ)IΘ1 (θ), (under the standard dominating measures on Θ0 and Θ1) Using the prior probabilities π(Θ0) = ρ0 and π(Θ1) = ρ1, π(θ) = ρ0π0(θ) + ρ1π1(θ). Note If Θ0 = {θ0}, π0 is the Dirac mass in θ0
  • 26. Point null refurbishment Requirement Defined prior distributions under both assumptions, π0(θ) ∝ π(θ)IΘ0 (θ), π1(θ) ∝ π(θ)IΘ1 (θ), (under the standard dominating measures on Θ0 and Θ1) Using the prior probabilities π(Θ0) = ρ0 and π(Θ1) = ρ1, π(θ) = ρ0π0(θ) + ρ1π1(θ). Note If Θ0 = {θ0}, π0 is the Dirac mass in θ0
  • 27. Point null hypotheses Particular case H0 : θ = θ0 Take ρ0 = Prπ (θ = θ0) and g1 prior density under Ha. Posterior probability of H0 π(Θ0|x) = f (x|θ0)ρ0 f (x|θ)π(θ) dθ = f (x|θ0)ρ0 f (x|θ0)ρ0 + (1 − ρ0)m1(x) and marginal under Ha m1(x) = Θ1 f (x|θ)g1(θ) dθ.
  • 28. Point null hypotheses Particular case H0 : θ = θ0 Take ρ0 = Prπ (θ = θ0) and g1 prior density under Ha. Posterior probability of H0 π(Θ0|x) = f (x|θ0)ρ0 f (x|θ)π(θ) dθ = f (x|θ0)ρ0 f (x|θ0)ρ0 + (1 − ρ0)m1(x) and marginal under Ha m1(x) = Θ1 f (x|θ)g1(θ) dθ.
  • 29. Point null hypotheses (cont’d) Dual representation π(Θ0|x) = 1 + 1 − ρ0 ρ0 m1(x) f (x|θ0) −1 . and Bπ 01(x) = f (x|θ0)ρ0 m1(x)(1 − ρ0) ρ0 1 − ρ0 = f (x|θ0) m1(x) Connection π(Θ0|x) = 1 + 1 − ρ0 ρ0 1 Bπ 01(x) −1 .
  • 30. Point null hypotheses (cont’d) Dual representation π(Θ0|x) = 1 + 1 − ρ0 ρ0 m1(x) f (x|θ0) −1 . and Bπ 01(x) = f (x|θ0)ρ0 m1(x)(1 − ρ0) ρ0 1 − ρ0 = f (x|θ0) m1(x) Connection π(Θ0|x) = 1 + 1 − ρ0 ρ0 1 Bπ 01(x) −1 .
  • 31. A further difficulty Improper priors are not allowed here If Θ1 π1(dθ1) = ∞ or Θ2 π2(dθ2) = ∞ then π1 or π2 cannot be coherently normalised while the normalisation matters in the Bayes factor remember Bayes factor?
  • 32. A further difficulty Improper priors are not allowed here If Θ1 π1(dθ1) = ∞ or Θ2 π2(dθ2) = ∞ then π1 or π2 cannot be coherently normalised while the normalisation matters in the Bayes factor remember Bayes factor?
  • 33. ToP unaware of the problem? A. Not entirely, as improper priors keep being used on nuisance parameters Example of testing for a zero normal mean: If σ is the standard error and λ the true value, λ is 0 on q. We want a suitable form for its prior on q . (...) Then we should take P(qdσ|H) ∝ dσ/σ P(q dσdλ|H) ∝ f λ σ dσ/σdλ/λ where f [is a true density] (V, §5.2). Fallacy of the “same” σ!
  • 34. ToP unaware of the problem? A. Not entirely, as improper priors keep being used on nuisance parameters Example of testing for a zero normal mean: If σ is the standard error and λ the true value, λ is 0 on q. We want a suitable form for its prior on q . (...) Then we should take P(qdσ|H) ∝ dσ/σ P(q dσdλ|H) ∝ f λ σ dσ/σdλ/λ where f [is a true density] (V, §5.2). Fallacy of the “same” σ!
  • 35. Not enought information If s = 0 [!!!], then [for σ = |¯x|/τ, λ = σv] P(q|θH) ∝ ∞ 0 τ |¯x| n exp − 1 2 nτ2 dτ τ , P(q |θH) ∝ ∞ 0 dτ τ ∞ −∞ τ |¯x| n f (v) exp − 1 2 n(v − τ)2 . If n = 1 and f (v) is any even [density], P(q |θH) ∝ 1 2 √ 2π |¯x| and P(q|θH) ∝ 1 2 √ 2π |¯x| and therefore K = 1 (V, §5.2).
  • 36. Strange constraints If n 2, the condition that K = 0 for s = 0, ¯x = 0 is equivalent to ∞ 0 f (v)vn−1 dv = ∞ . The function satisfying this condition for [all] n is f (v) = 1 π(1 + v2) This is the prior recommended by Jeffreys hereafter. But, first, many other families of densities satisfy this constraint and a scale of 1 cannot be universal! Second, s = 0 is a zero probability event...
  • 37. Strange constraints If n 2, the condition that K = 0 for s = 0, ¯x = 0 is equivalent to ∞ 0 f (v)vn−1 dv = ∞ . The function satisfying this condition for [all] n is f (v) = 1 π(1 + v2) This is the prior recommended by Jeffreys hereafter. But, first, many other families of densities satisfy this constraint and a scale of 1 cannot be universal! Second, s = 0 is a zero probability event...
  • 38. Strange constraints If n 2, the condition that K = 0 for s = 0, ¯x = 0 is equivalent to ∞ 0 f (v)vn−1 dv = ∞ . The function satisfying this condition for [all] n is f (v) = 1 π(1 + v2) This is the prior recommended by Jeffreys hereafter. But, first, many other families of densities satisfy this constraint and a scale of 1 cannot be universal! Second, s = 0 is a zero probability event...
  • 39. Comments ToP very imprecise about choice of priors in the setting of tests (despite existence of Susie’s Jeffreys’ conventional partly proper priors) ToP misses the difficulty of improper priors [coherent with earlier stance] but this problem still generates debates within the B community Some degree of goodness-of-fit testing but against fixed alternatives Persistence of the form K ≈ πn 2 1 + t2 ν −1/2ν+1/2 but ν not so clearly defined...
  • 40. Noninformative proposals Significance tests: one new parameter Noninformative solutions Pseudo-Bayes factors Intrinsic priors Jeffreys-Lindley paradox Deviance (information criterion) Testing under incomplete information Testing via mixtures
  • 41. Vague proper priors are not the solution Taking a proper prior and take a “very large” variance (e.g., BUGS) will most often result in an undefined or ill-defined limit Example (Lindley’s paradox) If testing H0 : θ = 0 when observing x ∼ N(θ, 1), under a normal N(0, α) prior π1(θ), B01(x) α−→∞ −→ 0
  • 42. Vague proper priors are not the solution Taking a proper prior and take a “very large” variance (e.g., BUGS) will most often result in an undefined or ill-defined limit Example (Lindley’s paradox) If testing H0 : θ = 0 when observing x ∼ N(θ, 1), under a normal N(0, α) prior π1(θ), B01(x) α−→∞ −→ 0
  • 43. Vague proper priors are not the solution Taking a proper prior and take a “very large” variance (e.g., BUGS) will most often result in an undefined or ill-defined limit Example (Lindley’s paradox) If testing H0 : θ = 0 when observing x ∼ N(θ, 1), under a normal N(0, α) prior π1(θ), B01(x) α−→∞ −→ 0
  • 44. Vague proper priors are not the solution (cont’d) Example (Poisson versus Negative binomial (4)) B12 = 1 0 λα+x−1 x! e−λβ dλ 1 M m x m − x + 1 βα Γ(α) if λ ∼ Ga(α, β) = Γ(α + x) x! Γ(α) β−x 1 M m x m − x + 1 = (x + α − 1) · · · α x(x − 1) · · · 1 β−x 1 M m x m − x + 1 depends on choice of α(β) or β(α) −→ 0
  • 45. Vague proper priors are not the solution (cont’d) Example (Poisson versus Negative binomial (4)) B12 = 1 0 λα+x−1 x! e−λβ dλ 1 M m x m − x + 1 βα Γ(α) if λ ∼ Ga(α, β) = Γ(α + x) x! Γ(α) β−x 1 M m x m − x + 1 = (x + α − 1) · · · α x(x − 1) · · · 1 β−x 1 M m x m − x + 1 depends on choice of α(β) or β(α) −→ 0
  • 46. Learning from the sample Definition (Learning sample) Given an improper prior π, (x1, . . . , xn) is a learning sample if π(·|x1, . . . , xn) is proper and a minimal learning sample if none of its subsamples is a learning sample There is just enough information in a minimal learning sample to make inference about θ under the prior π
  • 47. Learning from the sample Definition (Learning sample) Given an improper prior π, (x1, . . . , xn) is a learning sample if π(·|x1, . . . , xn) is proper and a minimal learning sample if none of its subsamples is a learning sample There is just enough information in a minimal learning sample to make inference about θ under the prior π
  • 48. Pseudo-Bayes factors Idea Use one part x[i] of the data x to make the prior proper: πi improper but πi (·|x[i]) proper and fi (x[n/i]|θi ) πi (θi |x[i])dθi fj (x[n/i]|θj ) πj (θj |x[i])dθj independent of normalizing constant Use remaining x[n/i] to run test as if πj (θj |x[i]) is the true prior
  • 49. Pseudo-Bayes factors Idea Use one part x[i] of the data x to make the prior proper: πi improper but πi (·|x[i]) proper and fi (x[n/i]|θi ) πi (θi |x[i])dθi fj (x[n/i]|θj ) πj (θj |x[i])dθj independent of normalizing constant Use remaining x[n/i] to run test as if πj (θj |x[i]) is the true prior
  • 50. Pseudo-Bayes factors Idea Use one part x[i] of the data x to make the prior proper: πi improper but πi (·|x[i]) proper and fi (x[n/i]|θi ) πi (θi |x[i])dθi fj (x[n/i]|θj ) πj (θj |x[i])dθj independent of normalizing constant Use remaining x[n/i] to run test as if πj (θj |x[i]) is the true prior
  • 51. Motivation Provides a working principle for improper priors Gather enough information from data to achieve properness and use this properness to run the test on remaining data does not use x twice as in Aitkin’s (1991)
  • 52. Motivation Provides a working principle for improper priors Gather enough information from data to achieve properness and use this properness to run the test on remaining data does not use x twice as in Aitkin’s (1991)
  • 53. Motivation Provides a working principle for improper priors Gather enough information from data to achieve properness and use this properness to run the test on remaining data does not use x twice as in Aitkin’s (1991)
  • 54. Details Since π1(θ1|x[i]) = π1(θ1)f 1 [i](x[i]|θ1) π1(θ1)f 1 [i](x[i]|θ1)dθ1 B12(x[n/i]) = f 1 [n/i](x[n/i]|θ1)π1(θ1|x[i])dθ1 f 2 [n/i](x[n/i]|θ2)π2(θ2|x[i])dθ2 = f1(x|θ1)π1(θ1)dθ1 f2(x|θ2)π2(θ2)dθ2 π2(θ2)f 2 [i](x[i]|θ2)dθ2 π1(θ1)f 1 [i](x[i]|θ1)dθ1 = BN 12(x)B21(x[i]) c Independent of scaling factor!
  • 55. Unexpected problems! depends on the choice of x[i] many ways of combining pseudo-Bayes factors AIBF = BN ji 1 L Bij (x[ ]) MIBF = BN ji med[Bij (x[ ])] GIBF = BN ji exp 1 L log Bij (x[ ]) not often an exact Bayes factor and thus lacking inner coherence B12 = B10B02 and B01 = 1/B10 . [Berger & Pericchi, 1996]
  • 56. Unexpected problems! depends on the choice of x[i] many ways of combining pseudo-Bayes factors AIBF = BN ji 1 L Bij (x[ ]) MIBF = BN ji med[Bij (x[ ])] GIBF = BN ji exp 1 L log Bij (x[ ]) not often an exact Bayes factor and thus lacking inner coherence B12 = B10B02 and B01 = 1/B10 . [Berger & Pericchi, 1996]
  • 57. Unexpec’d problems (cont’d) Example (Mixtures) There is no sample size that proper-ises improper priors, except if a training sample is allocated to each component Reason If x1, . . . , xn ∼ k i=1 pi f (x|θi ) and π(θ) = i πi (θi ) with πi (θi )dθi = +∞ , the posterior is never defined, because Pr(“no observation from f (·|θi )”) = (1 − pi )n
  • 58. Unexpec’d problems (cont’d) Example (Mixtures) There is no sample size that proper-ises improper priors, except if a training sample is allocated to each component Reason If x1, . . . , xn ∼ k i=1 pi f (x|θi ) and π(θ) = i πi (θi ) with πi (θi )dθi = +∞ , the posterior is never defined, because Pr(“no observation from f (·|θi )”) = (1 − pi )n
  • 59. Intrinsic priors There may exist a true prior that provides the same Bayes factor Example (Normal mean) Take x ∼ N(θ, 1) with either θ = 0 (M1) or θ = 0 (M2) and π2(θ) = 1. Then BAIBF 21 = B21 1√ 2π 1 n n i=1 e−x2 1 /2 ≈ B21 for N(0, 2) BMIBF 21 = B21 1√ 2π e−med(x2 1 )/2 ≈ 0.93B21 for N(0, 1.2) [Berger and Pericchi, 1998] When such a prior exists, it is called an intrinsic prior
  • 60. Intrinsic priors There may exist a true prior that provides the same Bayes factor Example (Normal mean) Take x ∼ N(θ, 1) with either θ = 0 (M1) or θ = 0 (M2) and π2(θ) = 1. Then BAIBF 21 = B21 1√ 2π 1 n n i=1 e−x2 1 /2 ≈ B21 for N(0, 2) BMIBF 21 = B21 1√ 2π e−med(x2 1 )/2 ≈ 0.93B21 for N(0, 1.2) [Berger and Pericchi, 1998] When such a prior exists, it is called an intrinsic prior
  • 61. Intrinsic priors There may exist a true prior that provides the same Bayes factor Example (Normal mean) Take x ∼ N(θ, 1) with either θ = 0 (M1) or θ = 0 (M2) and π2(θ) = 1. Then BAIBF 21 = B21 1√ 2π 1 n n i=1 e−x2 1 /2 ≈ B21 for N(0, 2) BMIBF 21 = B21 1√ 2π e−med(x2 1 )/2 ≈ 0.93B21 for N(0, 1.2) [Berger and Pericchi, 1998] When such a prior exists, it is called an intrinsic prior
  • 62. Intrinsic priors (cont’d) Example (Exponential scale) Take x1, . . . , xn i.i.d. ∼ exp(θ − x)Ix θ and H0 : θ = θ0, H1 : θ > θ0 , with π1(θ) = 1 Then BA 10 = B10(x) 1 n n i=1 exi −θ0 − 1 −1 is the Bayes factor for π2(θ) = eθ0−θ 1 − log 1 − eθ0−θ Most often, however, the pseudo-Bayes factors do not correspond to any true Bayes factor [Berger and Pericchi, 2001]
  • 63. Intrinsic priors (cont’d) Example (Exponential scale) Take x1, . . . , xn i.i.d. ∼ exp(θ − x)Ix θ and H0 : θ = θ0, H1 : θ > θ0 , with π1(θ) = 1 Then BA 10 = B10(x) 1 n n i=1 exi −θ0 − 1 −1 is the Bayes factor for π2(θ) = eθ0−θ 1 − log 1 − eθ0−θ Most often, however, the pseudo-Bayes factors do not correspond to any true Bayes factor [Berger and Pericchi, 2001]
  • 64. Intrinsic priors (cont’d) Example (Exponential scale) Take x1, . . . , xn i.i.d. ∼ exp(θ − x)Ix θ and H0 : θ = θ0, H1 : θ > θ0 , with π1(θ) = 1 Then BA 10 = B10(x) 1 n n i=1 exi −θ0 − 1 −1 is the Bayes factor for π2(θ) = eθ0−θ 1 − log 1 − eθ0−θ Most often, however, the pseudo-Bayes factors do not correspond to any true Bayes factor [Berger and Pericchi, 2001]
  • 65. Fractional Bayes factor Idea use directly the likelihood to separate training sample from testing sample BF 12 = B12(x) Lb 2(θ2)π2(θ2)dθ2 Lb 1(θ1)π1(θ1)dθ1 [O’Hagan, 1995] Proportion b of the sample used to gain proper-ness
  • 66. Fractional Bayes factor Idea use directly the likelihood to separate training sample from testing sample BF 12 = B12(x) Lb 2(θ2)π2(θ2)dθ2 Lb 1(θ1)π1(θ1)dθ1 [O’Hagan, 1995] Proportion b of the sample used to gain proper-ness
  • 67. Fractional Bayes factor (cont’d) Example (Normal mean) BF 12 = 1 √ b en(b−1)¯x2 n /2 corresponds to exact Bayes factor for the prior N 0, 1−b nb If b constant, prior variance goes to 0 If b = 1 n , prior variance stabilises around 1 If b = n−α , α < 1, prior variance goes to 0 too.
  • 68. Jeffreys–Lindley paradox Significance tests: one new parameter Noninformative solutions Jeffreys-Lindley paradox Lindley’s paradox dual versions of the paradox Bayesian resolutions Deviance (information criterion) Testing under incomplete information Testing via mixtures
  • 69. Lindley’s paradox In a normal mean testing problem, ¯xn ∼ N(θ, σ2 /n) , H0 : θ = θ0 , under Jeffreys prior, θ ∼ N(θ0, σ2), the Bayes factor B01(tn) = (1 + n)1/2 exp −nt2 n/2[1 + n] , where tn = √ n|¯xn − θ0|/σ, satisfies B01(tn) n−→∞ −→ ∞ [assuming a fixed tn] [Lindley, 1957]
  • 70. Lindley’s paradox Often dubbed Jeffreys–Lindley paradox... In terms of t = √ n − 1¯x/s , ν = n−1 K ∼ πν 2 1 + t2 ν −1/2ν+1/2 . (...) The variation of K with t is much more important than the variation with ν (Jeffreys, V, §5.2).
  • 71. Two versions of the paradox “the weight of Lindley’s paradoxical result (...) burdens proponents of the Bayesian practice”. [Lad, 2003] official version, opposing frequentist and Bayesian assessments [Lindley, 1957] intra-Bayesian version, blaming vague and improper priors for the Bayes factor misbehaviour: if π1(·|σ) depends on a scale parameter σ, it is often the case that B01(x) σ−→∞ −→ +∞ for a given x, meaning H0 is always accepted [Robert, 1992, 2013]
  • 72. where does it matter? In the normal case, Z ∼ N(θ, 1), θ ∼ N(0, α2), Bayes factor B10(z) = ez2α2/(1+α2) √ 1 + α2 = √ 1 − λ exp{λz2 /2}
  • 73. Evacuation of the first version Two paradigms [(b) versus (f)] one (b) operates on the parameter space Θ, while the other (f) is produced from the sample space one (f) relies solely on the point-null hypothesis H0 and the corresponding sampling distribution, while the other (b) opposes H0 to a (predictive) marginal version of H1 one (f) could reject “a hypothesis that may be true (...) because it has not predicted observable results that have not occurred” (Jeffreys, ToP, VII, §7.2) while the other (b) conditions upon the observed value xobs one (f) cannot agree with the likelihood principle, while the other (b) is almost uniformly in agreement with it one (f) resorts to an arbitrary fixed bound α on the p-value, while the other (b) refers to the (default) boundary probability of 1/2
  • 74. More arguments on the first version observing a constant tn as n increases is of limited interest: under H0 tn has limiting N(0, 1) distribution, while, under H1 tn a.s. converges to ∞ behaviour that remains entirely compatible with the consistency of the Bayes factor, which a.s. converges either to 0 or ∞, depending on which hypothesis is true. Consequent subsequent literature (e.g., Berger & Sellke, 1987; Bayarri & Berger, 2004) has since then shown how divergent those two approaches could be (to the point of being asymptotically incompatible).
  • 75. Nothing’s wrong with the second version n, prior’s scale factor: prior variance n times larger than the observation variance and when n goes to ∞, Bayes factor goes to ∞ no matter what the observation is n becomes what Lindley (1957) calls “a measure of lack of conviction about the null hypothesis” when prior diffuseness under H1 increases, only relevant information becomes that θ could be equal to θ0, and this overwhelms any evidence to the contrary contained in the data mass of the prior distribution in the vicinity of any fixed neighbourhood of the null hypothesis vanishes to zero under H1 c deep coherence in the outcome: being indecisive about the alternative hypothesis means we should not chose it
  • 76. Nothing’s wrong with the second version n, prior’s scale factor: prior variance n times larger than the observation variance and when n goes to ∞, Bayes factor goes to ∞ no matter what the observation is n becomes what Lindley (1957) calls “a measure of lack of conviction about the null hypothesis” when prior diffuseness under H1 increases, only relevant information becomes that θ could be equal to θ0, and this overwhelms any evidence to the contrary contained in the data mass of the prior distribution in the vicinity of any fixed neighbourhood of the null hypothesis vanishes to zero under H1 c deep coherence in the outcome: being indecisive about the alternative hypothesis means we should not chose it
  • 77. On some resolutions of the second version use of pseudo-Bayes factors, fractional Bayes factors, &tc, which lacks complete proper Bayesian justification [Berger & Pericchi, 2001] use of identical improper priors on nuisance parameters, use of the posterior predictive distribution, matching priors, use of score functions extending the log score function non-local priors correcting default priors
  • 78. On some resolutions of the second version use of pseudo-Bayes factors, fractional Bayes factors, &tc, use of identical improper priors on nuisance parameters, a notion already entertained by Jeffreys [Berger et al., 1998; Marin & Robert, 2013] use of the posterior predictive distribution, matching priors, use of score functions extending the log score function non-local priors correcting default priors
  • 79. On some resolutions of the second version use of pseudo-Bayes factors, fractional Bayes factors, &tc, use of identical improper priors on nuisance parameters, P´ech´e de jeunesse: equating the values of the prior densities at the point-null value θ0, ρ0 = (1 − ρ0)π1(θ0) [Robert, 1993] use of the posterior predictive distribution, matching priors, use of score functions extending the log score function non-local priors correcting default priors
  • 80. On some resolutions of the second version use of pseudo-Bayes factors, fractional Bayes factors, &tc, use of identical improper priors on nuisance parameters, use of the posterior predictive distribution, which uses the data twice matching priors, use of score functions extending the log score function non-local priors correcting default priors
  • 81. On some resolutions of the second version use of pseudo-Bayes factors, fractional Bayes factors, &tc, use of identical improper priors on nuisance parameters, use of the posterior predictive distribution, matching priors, whose sole purpose is to bring frequentist and Bayesian coverages as close as possible [Datta & Mukerjee, 2004] use of score functions extending the log score function non-local priors correcting default priors
  • 82. On some resolutions of the second version use of pseudo-Bayes factors, fractional Bayes factors, &tc, use of identical improper priors on nuisance parameters, use of the posterior predictive distribution, matching priors, use of score functions extending the log score function log B12(x) = log m1(x) − log m2(x) = S0(x, m1) − S0(x, m2) , that are independent of the normalising constant [Dawid et al., 2013; Dawid & Musio, 2015] non-local priors correcting default priors
  • 83. On some resolutions of the second version use of pseudo-Bayes factors, fractional Bayes factors, &tc, use of identical improper priors on nuisance parameters, use of the posterior predictive distribution, matching priors, use of score functions extending the log score function non-local priors correcting default priors towards more balanced error rates [Johnson & Rossell, 2010; Consonni et al., 2013]
  • 84. Deviance (information criterion) Significance tests: one new parameter Noninformative solutions Jeffreys-Lindley paradox Deviance (information criterion) Testing under incomplete information Testing via mixtures
  • 85. DIC as in Dayesian? Deviance defined by D(θ) = −2 log(p(y|θ)) , Effective number of parameters computed as pD = ¯D − D(¯θ) , with ¯D posterior expectation of D and ¯θ estimate of θ Deviance information criterion (DIC) defined by DIC = pD + ¯D = D(¯θ) + 2pD Models with smaller DIC better supported by the data [Spiegelhalter et al., 2002]
  • 86. “thou shalt not use the data twice” The data is used twice in the DIC method: 1. y used once to produce the posterior π(θ|y), and the associated estimate, ˜θ(y) 2. y used a second time to compute the posterior expectation of the observed likelihood p(y|θ), log p(y|θ)π(dθ|y) ∝ log p(y|θ)p(y|θ)π(dθ) ,
  • 87. DIC for missing data models Framework of missing data models f (y|θ) = f (y, z|θ)dz , with observed data y = (y1, . . . , yn) and corresponding missing data by z = (z1, . . . , zn) How do we define DIC in such settings?
  • 88. DIC for missing data models Framework of missing data models f (y|θ) = f (y, z|θ)dz , with observed data y = (y1, . . . , yn) and corresponding missing data by z = (z1, . . . , zn) How do we define DIC in such settings?
  • 89. how many DICs can you fit in a mixture? Q: How many giraffes can you fit in a VW bug? A: None, the elephants are in there. 1. observed DICs DIC1 = −4Eθ [log f (y|θ)|y] + 2 log f (y|Eθ [θ|y]) often a poor choice in case of unidentifiability 2. complete DICs based on f (y, z|θ) 3. conditional DICs based on f (y|z, θ) [Celeux et al., BA, 2006]
  • 90. how many DICs can you fit in a mixture? Q: How many giraffes can you fit in a VW bug? A: None, the elephants are in there. 1. observed DICs DIC2 = −4Eθ [log f (y|θ)|y] + 2 log f (y|θ(y)) . which uses posterior mode instead 2. complete DICs based on f (y, z|θ) 3. conditional DICs based on f (y|z, θ) [Celeux et al., BA, 2006]
  • 91. how many DICs can you fit in a mixture? Q: How many giraffes can you fit in a VW bug? A: None, the elephants are in there. 1. observed DICs DIC3 = −4Eθ [log f (y|θ)|y] + 2 log f (y) , which instead relies on the MCMC density estimate 2. complete DICs based on f (y, z|θ) 3. conditional DICs based on f (y|z, θ) [Celeux et al., BA, 2006]
  • 92. how many DICs can you fit in a mixture? Q: How many giraffes can you fit in a VW bug? A: None, the elephants are in there. 1. observed DICs 2. complete DICs based on f (y, z|θ) DIC4 = EZ [DIC(y, Z)|y] = −4Eθ,Z [log f (y, Z|θ)|y] + 2EZ [log f (y, Z|Eθ[θ|y, Z])|y] 3. conditional DICs based on f (y|z, θ) [Celeux et al., BA, 2006]
  • 93. how many DICs can you fit in a mixture? Q: How many giraffes can you fit in a VW bug? A: None, the elephants are in there. 1. observed DICs 2. complete DICs based on f (y, z|θ) DIC5 = −4Eθ,Z [log f (y, Z|θ)|y] + 2 log f (y, z(y)|θ(y)) , using Z as an additional parameter 3. conditional DICs based on f (y|z, θ) [Celeux et al., BA, 2006]
  • 94. how many DICs can you fit in a mixture? Q: How many giraffes can you fit in a VW bug? A: None, the elephants are in there. 1. observed DICs 2. complete DICs based on f (y, z|θ) DIC6 = −4Eθ,Z [log f (y, Z|θ)|y]+2EZ[log f (y, Z|θ(y))|y, θ(y)] . in analogy with EM, θ being an EM fixed point 3. conditional DICs based on f (y|z, θ) [Celeux et al., BA, 2006]
  • 95. how many DICs can you fit in a mixture? Q: How many giraffes can you fit in a VW bug? A: None, the elephants are in there. 1. observed DICs 2. complete DICs based on f (y, z|θ) 3. conditional DICs based on f (y|z, θ) DIC7 = −4Eθ,Z [log f (y|Z, θ)|y] + 2 log f (y|z(y), θ(y)) , using MAP estimates [Celeux et al., BA, 2006]
  • 96. how many DICs can you fit in a mixture? Q: How many giraffes can you fit in a VW bug? A: None, the elephants are in there. 1. observed DICs 2. complete DICs based on f (y, z|θ) 3. conditional DICs based on f (y|z, θ) DIC8 = −4Eθ,Z [log f (y|Z, θ)|y]+2EZ log f (y|Z, θ(y, Z))|y , conditioning first on Z and then integrating over Z conditional on y [Celeux et al., BA, 2006]
  • 97. Galactic DICs Example of the galaxy mixture dataset DIC2 DIC3 DIC4 DIC5 DIC6 DIC7 DIC8 K (pD2) (pD3) (pD4) (pD5) (pD6) (pD7) (pD8) 2 453 451 502 705 501 417 410 (5.56) (3.66) (5.50) (207.88) (4.48) (11.07) (4.09) 3 440 436 461 622 471 378 372 (9.23) (4.94) (6.40) (167.28) (15.80) (13.59) (7.43) 4 446 439 473 649 482 388 382 (11.58) (5.41) (7.52) (183.48) (16.51) (17.47) (11.37) 5 447 442 485 658 511 395 390 (10.80) (5.48) (7.58) (180.73) (33.29) (20.00) (15.15) 6 449 444 494 676 532 407 398 (11.26) (5.49) (8.49) (191.10) (46.83) (28.23) (19.34) 7 460 446 508 700 571 425 409 (19.26) (5.83) (8.93) (200.35) (71.26) (40.51) (24.57)
  • 98. questions what is the behaviour of DIC under model mispecification? is there an absolute scale to the DIC values, i.e. when is a difference in DICs significant? how can DIC handle small n’s versus p’s? should pD be defined as var(D|y)/2 [Gelman’s suggestion]? is WAIC (Gelman and Vehtari, 2013) making a difference for being based on expected posterior predictive? In an era of complex models, is DIC applicable? [Robert, 2013]
  • 99. questions what is the behaviour of DIC under model mispecification? is there an absolute scale to the DIC values, i.e. when is a difference in DICs significant? how can DIC handle small n’s versus p’s? should pD be defined as var(D|y)/2 [Gelman’s suggestion]? is WAIC (Gelman and Vehtari, 2013) making a difference for being based on expected posterior predictive? In an era of complex models, is DIC applicable? [Robert, 2013]
  • 100. Testing under incomplete information Significance tests: one new parameter Noninformative solutions Jeffreys-Lindley paradox Deviance (information criterion) Testing under incomplete information Testing via mixtures
  • 101. Likelihood-free settings Cases when the likelihood function f (y|θ) is unavailable (in analytic and numerical senses) and when the completion step f (y|θ) = Z f (y, z|θ) dz is impossible or too costly because of the dimension of z c MCMC cannot be implemented!
  • 102. The ABC method Bayesian setting: target is π(θ)f (x|θ) When likelihood f (x|θ) not in closed form, likelihood-free rejection technique: ABC algorithm For an observation y ∼ f (y|θ), under the prior π(θ), keep jointly simulating θ ∼ π(θ) , z ∼ f (z|θ ) , until the auxiliary variable z is equal to the observed value, z = y. [Tavar´e et al., 1997]
  • 103. The ABC method Bayesian setting: target is π(θ)f (x|θ) When likelihood f (x|θ) not in closed form, likelihood-free rejection technique: ABC algorithm For an observation y ∼ f (y|θ), under the prior π(θ), keep jointly simulating θ ∼ π(θ) , z ∼ f (z|θ ) , until the auxiliary variable z is equal to the observed value, z = y. [Tavar´e et al., 1997]
  • 104. The ABC method Bayesian setting: target is π(θ)f (x|θ) When likelihood f (x|θ) not in closed form, likelihood-free rejection technique: ABC algorithm For an observation y ∼ f (y|θ), under the prior π(θ), keep jointly simulating θ ∼ π(θ) , z ∼ f (z|θ ) , until the auxiliary variable z is equal to the observed value, z = y. [Tavar´e et al., 1997]
  • 105. A as A...pproximative When y is a continuous random variable, strict equality z = y is replaced with a tolerance zone ρ(y, z) where ρ is a distance Output distributed from π(θ) Pθ{ρ(y, z) < } def ∝ π(θ|ρ(y, z) < ) [Pritchard et al., 1999]
  • 106. A as A...pproximative When y is a continuous random variable, strict equality z = y is replaced with a tolerance zone ρ(y, z) where ρ is a distance Output distributed from π(θ) Pθ{ρ(y, z) < } def ∝ π(θ|ρ(y, z) < ) [Pritchard et al., 1999]
  • 107. ABC algorithm In most implementations, further degree of A...pproximation: Algorithm 1 Likelihood-free rejection sampler for i = 1 to N do repeat generate θ from the prior distribution π(·) generate z from the likelihood f (·|θ ) until ρ{η(z), η(y)} set θi = θ end for where η(y) defines a (not necessarily sufficient) statistic
  • 108. Which summary η(·)? Fundamental difficulty of the choice of the summary statistic when there is no non-trivial sufficient statistics Loss of statistical information balanced against gain in data roughening Approximation error and information loss remain unknown Choice of statistics induces choice of distance function towards standardisation may be imposed for external/practical reasons (e.g., LDA) may gather several non-B point estimates can learn about efficient combination [Estoup et al., 2012, Genetics]
  • 109. Which summary η(·)? Fundamental difficulty of the choice of the summary statistic when there is no non-trivial sufficient statistics Loss of statistical information balanced against gain in data roughening Approximation error and information loss remain unknown Choice of statistics induces choice of distance function towards standardisation may be imposed for external/practical reasons (e.g., LDA) may gather several non-B point estimates can learn about efficient combination [Estoup et al., 2012, Genetics]
  • 110. Which summary η(·)? Fundamental difficulty of the choice of the summary statistic when there is no non-trivial sufficient statistics Loss of statistical information balanced against gain in data roughening Approximation error and information loss remain unknown Choice of statistics induces choice of distance function towards standardisation may be imposed for external/practical reasons (e.g., LDA) may gather several non-B point estimates can learn about efficient combination [Estoup et al., 2012, Genetics]
  • 111. Generic ABC for model choice Algorithm 2 Likelihood-free model choice sampler (ABC-MC) for t = 1 to T do repeat Generate m from the prior π(M = m) Generate θm from the prior πm(θm) Generate z from the model fm(z|θm) until ρ{η(z), η(y)} < Set m(t) = m and θ(t) = θm end for [Grelaud et al., 2009]
  • 112. ABC estimates Posterior probability π(M = m|y) approximated by the frequency of acceptances from model m 1 T T t=1 Im(t)=m . Issues with implementation: should tolerances be the same for all models? should summary statistics vary across models (incl. their dimension)? should the distance measure ρ vary as well?
  • 113. ABC estimates Posterior probability π(M = m|y) approximated by the frequency of acceptances from model m 1 T T t=1 Im(t)=m . Extension to a weighted polychotomous logistic regression estimate of π(M = m|y), with non-parametric kernel weights [Cornuet et al., DIYABC, 2009]
  • 114. ABCµ Idea Infer about the error as well as about the parameter: Use of a joint density f (θ, |y) ∝ ξ( |y, θ) × πθ(θ) × π ( ) where y is the data, and ξ( |y, θ) is the prior predictive density of ρ(η(z), η(y)) given θ and y when z ∼ f (z|θ) Warning! Replacement of ξ( |y, θ) with a non-parametric kernel approximation. [Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]
  • 115. ABCµ Idea Infer about the error as well as about the parameter: Use of a joint density f (θ, |y) ∝ ξ( |y, θ) × πθ(θ) × π ( ) where y is the data, and ξ( |y, θ) is the prior predictive density of ρ(η(z), η(y)) given θ and y when z ∼ f (z|θ) Warning! Replacement of ξ( |y, θ) with a non-parametric kernel approximation. [Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]
  • 116. ABCµ Idea Infer about the error as well as about the parameter: Use of a joint density f (θ, |y) ∝ ξ( |y, θ) × πθ(θ) × π ( ) where y is the data, and ξ( |y, θ) is the prior predictive density of ρ(η(z), η(y)) given θ and y when z ∼ f (z|θ) Warning! Replacement of ξ( |y, θ) with a non-parametric kernel approximation. [Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]
  • 117. ABCµ details Multidimensional distances ρk (k = 1, . . . , K) and errors k = ρk(ηk(z), ηk(y)), with k ∼ ξk( |y, θ) ≈ ^ξk( |y, θ) = 1 Bhk b K[{ k−ρk(ηk(zb), ηk(y))}/hk] then used in replacing ξ( |y, θ) with mink ^ξk( |y, θ) ABCµ involves acceptance probability π(θ , ) π(θ, ) q(θ , θ)q( , ) q(θ, θ )q( , ) mink ^ξk( |y, θ ) mink ^ξk( |y, θ)
  • 118. ABCµ details Multidimensional distances ρk (k = 1, . . . , K) and errors k = ρk(ηk(z), ηk(y)), with k ∼ ξk( |y, θ) ≈ ^ξk( |y, θ) = 1 Bhk b K[{ k−ρk(ηk(zb), ηk(y))}/hk] then used in replacing ξ( |y, θ) with mink ^ξk( |y, θ) ABCµ involves acceptance probability π(θ , ) π(θ, ) q(θ , θ)q( , ) q(θ, θ )q( , ) mink ^ξk( |y, θ ) mink ^ξk( |y, θ)
  • 119. ABCµ multiple errors [ c Ratmann et al., PNAS, 2009]
  • 120. ABCµ for model choice [ c Ratmann et al., PNAS, 2009]
  • 121. Questions about ABCµ [and model choice] For each model under comparison, marginal posterior on used to assess the fit of the model (HPD includes 0 or not). Is the data informative about ? [Identifiability] How much does the prior π( ) impact the comparison? How is using both ξ( |x0, θ) and π ( ) compatible with a standard probability model? Where is the penalisation for complexity in the model comparison? [X, Mengersen & Chen, 2010, PNAS]
  • 122. Questions about ABCµ [and model choice] For each model under comparison, marginal posterior on used to assess the fit of the model (HPD includes 0 or not). Is the data informative about ? [Identifiability] How much does the prior π( ) impact the comparison? How is using both ξ( |x0, θ) and π ( ) compatible with a standard probability model? Where is the penalisation for complexity in the model comparison? [X, Mengersen & Chen, 2010, PNAS]
  • 123. Formalised framework Central question to the validation of ABC for model choice: When is a Bayes factor based on an insufficient statistic T(y) consistent? Note: c drawn on T(y) through BT 12(y) necessarily differs from c drawn on y through B12(y)
  • 124. Formalised framework Central question to the validation of ABC for model choice: When is a Bayes factor based on an insufficient statistic T(y) consistent? Note: c drawn on T(y) through BT 12(y) necessarily differs from c drawn on y through B12(y)
  • 125. A benchmark if toy example Comparison suggested by referee of PNAS paper [thanks]: [X, Cornuet, Marin, & Pillai, Aug. 2011] Model M1: y ∼ N(θ1, 1) opposed to model M2: y ∼ L(θ2, 1/ √ 2), Laplace distribution with mean θ2 and scale parameter 1/ √ 2 (variance one).
  • 126. A benchmark if toy example Comparison suggested by referee of PNAS paper [thanks]: [X, Cornuet, Marin, & Pillai, Aug. 2011] Model M1: y ∼ N(θ1, 1) opposed to model M2: y ∼ L(θ2, 1/ √ 2), Laplace distribution with mean θ2 and scale parameter 1/ √ 2 (variance one). Four possible statistics 1. sample mean y (sufficient for M1 if not M2); 2. sample median med(y) (insufficient); 3. sample variance var(y) (ancillary); 4. median absolute deviation mad(y) = med(y − med(y));
  • 127. A benchmark if toy example Comparison suggested by referee of PNAS paper [thanks]: [X, Cornuet, Marin, & Pillai, Aug. 2011] Model M1: y ∼ N(θ1, 1) opposed to model M2: y ∼ L(θ2, 1/ √ 2), Laplace distribution with mean θ2 and scale parameter 1/ √ 2 (variance one). 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0123456 posterior probability Density
  • 128. A benchmark if toy example Comparison suggested by referee of PNAS paper [thanks]: [X, Cornuet, Marin, & Pillai, Aug. 2011] Model M1: y ∼ N(θ1, 1) opposed to model M2: y ∼ L(θ2, 1/ √ 2), Laplace distribution with mean θ2 and scale parameter 1/ √ 2 (variance one). q q q q q q q q q q q Gauss Laplace 0.00.10.20.30.40.50.60.7 n=100
  • 129. A benchmark if toy example Comparison suggested by referee of PNAS paper [thanks]: [X, Cornuet, Marin, & Pillai, Aug. 2011] Model M1: y ∼ N(θ1, 1) opposed to model M2: y ∼ L(θ2, 1/ √ 2), Laplace distribution with mean θ2 and scale parameter 1/ √ 2 (variance one). 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0123456 posterior probability Density 0.0 0.2 0.4 0.6 0.8 1.0 0123 probability Density
  • 130. A benchmark if toy example Comparison suggested by referee of PNAS paper [thanks]: [X, Cornuet, Marin, & Pillai, Aug. 2011] Model M1: y ∼ N(θ1, 1) opposed to model M2: y ∼ L(θ2, 1/ √ 2), Laplace distribution with mean θ2 and scale parameter 1/ √ 2 (variance one). q q q q q q q q q q q Gauss Laplace 0.00.10.20.30.40.50.60.7 n=100 q q q q q q q q q q q q q q q q q q Gauss Laplace 0.00.20.40.60.81.0 n=100
  • 131. Consistency theorem If Pn belongs to one of the two models and if µ0 = E[T] cannot be attained by the other one : 0 = min (inf{|µ0 − µi (θi )|; θi ∈ Θi }, i = 1, 2) < max (inf{|µ0 − µi (θi )|; θi ∈ Θi }, i = 1, 2) , then the Bayes factor BT 12 is consistent
  • 132. Conclusion Model selection feasible with ABC: Choice of summary statistics is paramount At best, ABC output → π(. | η(y)) which concentrates around µ0 For estimation : {θ; µ(θ) = µ0} = θ0 For testing {µ1(θ1), θ1 ∈ Θ1} ∩ {µ2(θ2), θ2 ∈ Θ2} = ∅ [Marin et al., 2013]
  • 133. Conclusion Model selection feasible with ABC: Choice of summary statistics is paramount At best, ABC output → π(. | η(y)) which concentrates around µ0 For estimation : {θ; µ(θ) = µ0} = θ0 For testing {µ1(θ1), θ1 ∈ Θ1} ∩ {µ2(θ2), θ2 ∈ Θ2} = ∅ [Marin et al., 2013]
  • 134. Changing the testing perspective Significance tests: one new parameter Noninformative solutions Jeffreys-Lindley paradox Deviance (information criterion) Testing under incomplete information Testing via mixtures
  • 135. Testing issues Hypothesis testing central problem of statistical inference dramatically differentiating feature between classical and Bayesian paradigms wide open to controversy and divergent opinions, includ. within the Bayesian community non-informative Bayesian testing case mostly unresolved, witness the Jeffreys–Lindley paradox [Berger (2003), Mayo & Cox (2006), Gelman (2008)]
  • 136. Testing hypotheses Bayesian model selection as comparison of k potential statistical models towards the selection of model that fits the data “best” mostly accepted perspective: it does not primarily seek to identify which model is “true”, but compares fits tools like Bayes factor naturally include a penalisation addressing model complexity, mimicked by Bayes Information (BIC) and Deviance Information (DIC) criteria posterior predictive tools successfully advocated in Gelman et al. (2013) even though they involve double use of data
  • 137. Testing hypotheses Bayesian model selection as comparison of k potential statistical models towards the selection of model that fits the data “best” mostly accepted perspective: it does not primarily seek to identify which model is “true”, but compares fits tools like Bayes factor naturally include a penalisation addressing model complexity, mimicked by Bayes Information (BIC) and Deviance Information (DIC) criteria posterior predictive tools successfully advocated in Gelman et al. (2013) even though they involve double use of data
  • 138. Some difficulties tension between using (i) posterior probabilities justified by a binary loss function but depending on unnatural prior weights, and (ii) Bayes factors that eliminate this dependence but escape direct connection with posterior distribution, unless prior weights are integrated within the loss subsequent and delicate interpretation (or calibration) of the strength of the Bayes factor towards supporting a given hypothesis or model, because it is not a Bayesian decision rule similar difficulty with posterior probabilities, with tendency to interpret them as p-values (rather than the opposite!) when they only report through a marginal likelihood ratio the respective strengths of fitting the data to both models
  • 139. Some further difficulties long-lasting impact of the prior modeling, meaning the choice of the prior distributions on the parameter spaces of both models under comparison, despite overall consistency proof for Bayes factor discontinuity in use of improper priors since they are not justified in most testing situations, leading to many alternative if ad hoc solutions, where data is either used twice or split in artificial ways binary (accept vs. reject) outcome more suited for immediate decision (if any) than for model evaluation, in connection with rudimentary loss function
  • 140. Some additional difficulties related impossibility to ascertain simultaneous misfit or to detect presence of outliers no assessment of uncertainty associated with decision itself difficult computation of marginal likelihoods in most settings with further controversies about which algorithm to adopt strong dependence of posterior probabilities on conditioning statistics, which in turn undermines their validity for model assessment, as exhibited in ABC model choice temptation to create pseudo-frequentist equivalents such as q-values with even less Bayesian justifications
  • 141. Paradigm shift New proposal for a paradigm shift in the Bayesian processing of hypothesis testing and of model selection convergent and naturally interpretable solution more extended use of improper priors Simple representation of the testing problem as a two-component mixture estimation problem where the weights are formally equal to 0 or 1
  • 142. Paradigm shift New proposal for a paradigm shift in the Bayesian processing of hypothesis testing and of model selection convergent and naturally interpretable solution more extended use of improper priors Simple representation of the testing problem as a two-component mixture estimation problem where the weights are formally equal to 0 or 1
  • 143. Paradigm shift Simple representation of the testing problem as a two-component mixture estimation problem where the weights are formally equal to 0 or 1 Approach inspired from consistency result of Rousseau and Mengersen (2011) on estimated overfitting mixtures Mixture representation not directly equivalent to the use of a posterior probability Potential of a better approach to testing, while not expanding number of parameters Calibration of the posterior distribution of the weight of a model, while moving from the artificial notion of the posterior probability of a model
  • 144. Encompassing mixture model Idea: Given two statistical models, M1 : x ∼ f1(x|θ1) , θ1 ∈ Θ1 and M2 : x ∼ f2(x|θ2) , θ2 ∈ Θ2 , embed both within an encompassing mixture Mα : x ∼ αf1(x|θ1) + (1 − α)f2(x|θ2) , 0 α 1 (1) Note: Both models correspond to special cases of (1), one for α = 1 and one for α = 0 Draw inference on mixture representation (1), as if each observation was individually and independently produced by the mixture model
  • 145. Encompassing mixture model Idea: Given two statistical models, M1 : x ∼ f1(x|θ1) , θ1 ∈ Θ1 and M2 : x ∼ f2(x|θ2) , θ2 ∈ Θ2 , embed both within an encompassing mixture Mα : x ∼ αf1(x|θ1) + (1 − α)f2(x|θ2) , 0 α 1 (1) Note: Both models correspond to special cases of (1), one for α = 1 and one for α = 0 Draw inference on mixture representation (1), as if each observation was individually and independently produced by the mixture model
  • 146. Encompassing mixture model Idea: Given two statistical models, M1 : x ∼ f1(x|θ1) , θ1 ∈ Θ1 and M2 : x ∼ f2(x|θ2) , θ2 ∈ Θ2 , embed both within an encompassing mixture Mα : x ∼ αf1(x|θ1) + (1 − α)f2(x|θ2) , 0 α 1 (1) Note: Both models correspond to special cases of (1), one for α = 1 and one for α = 0 Draw inference on mixture representation (1), as if each observation was individually and independently produced by the mixture model
  • 147. Inferential motivations Sounds like an approximation to the real model, but several definitive advantages to this paradigm shift: Bayes estimate of the weight α replaces posterior probability of model M1, equally convergent indicator of which model is “true”, while avoiding artificial prior probabilities on model indices, ω1 and ω2 interpretation of estimator of α at least as natural as handling the posterior probability, while avoiding zero-one loss setting α and its posterior distribution provide measure of proximity to the models, while being interpretable as data propensity to stand within one model further allows for alternative perspectives on testing and model choice, like predictive tools, cross-validation, and information indices like WAIC
  • 148. Computational motivations avoids highly problematic computations of the marginal likelihoods, since standard algorithms are available for Bayesian mixture estimation straightforward extension to a finite collection of models, with a larger number of components, which considers all models at once and eliminates least likely models by simulation eliminates difficulty of label switching that plagues both Bayesian estimation and Bayesian computation, since components are no longer exchangeable posterior distribution of α evaluates more thoroughly strength of support for a given model than the single figure outcome of a posterior probability variability of posterior distribution on α allows for a more thorough assessment of the strength of this support
  • 149. Noninformative motivations additional feature missing from traditional Bayesian answers: a mixture model acknowledges possibility that, for a finite dataset, both models or none could be acceptable standard (proper and informative) prior modeling can be reproduced in this setting, but non-informative (improper) priors also are manageable therein, provided both models first reparameterised towards shared parameters, e.g. location and scale parameters in special case when all parameters are common Mα : x ∼ αf1(x|θ) + (1 − α)f2(x|θ) , 0 α 1 if θ is a location parameter, a flat prior π(θ) ∝ 1 is available
  • 150. Weakly informative motivations using the same parameters or some identical parameters on both components highlights that opposition between the two components is not an issue of enjoying different parameters those common parameters are nuisance parameters, to be integrated out prior model weights ωi rarely discussed in classical Bayesian approach, even though linear impact on posterior probabilities. Here, prior modeling only involves selecting a prior on α, e.g., α ∼ B(a0, a0) while a0 impacts posterior on α, it always leads to mass accumulation near 1 or 0, i.e. favours most likely model sensitivity analysis straightforward to carry approach easily calibrated by parametric boostrap providing reference posterior of α under each model natural Metropolis–Hastings alternative
  • 151. Poisson/Geometric choice betwen Poisson P(λ) and Geometric Geo(p) distribution mixture with common parameter λ Mα : αP(λ) + (1 − α)Geo(1/1+λ) Allows for Jeffreys prior since resulting posterior is proper independent Metropolis–within–Gibbs with proposal distribution on λ equal to Poisson posterior (with acceptance rate larger than 75%)
  • 152. Poisson/Geometric choice betwen Poisson P(λ) and Geometric Geo(p) distribution mixture with common parameter λ Mα : αP(λ) + (1 − α)Geo(1/1+λ) Allows for Jeffreys prior since resulting posterior is proper independent Metropolis–within–Gibbs with proposal distribution on λ equal to Poisson posterior (with acceptance rate larger than 75%)
  • 153. Poisson/Geometric choice betwen Poisson P(λ) and Geometric Geo(p) distribution mixture with common parameter λ Mα : αP(λ) + (1 − α)Geo(1/1+λ) Allows for Jeffreys prior since resulting posterior is proper independent Metropolis–within–Gibbs with proposal distribution on λ equal to Poisson posterior (with acceptance rate larger than 75%)
  • 154. Beta prior When α ∼ Be(a0, a0) prior, full conditional posterior α ∼ Be(n1(ζ) + a0, n2(ζ) + a0) Exact Bayes factor opposing Poisson and Geometric B12 = nn¯xn n i=1 xi ! Γ n + 2 + n i=1 xi Γ(n + 2) although undefined from a purely mathematical viewpoint
  • 155. Parameter estimation 1e-04 0.001 0.01 0.1 0.2 0.3 0.4 0.5 3.903.954.004.054.10 Posterior means of λ and medians of α for 100 Poisson P(4) datasets of size n = 1000, for a0 = .0001, .001, .01, .1, .2, .3, .4, .5. Each posterior approximation is based on 104 Metropolis-Hastings iterations.
  • 156. Parameter estimation 1e-04 0.001 0.01 0.1 0.2 0.3 0.4 0.5 0.9900.9920.9940.9960.9981.000 Posterior means of λ and medians of α for 100 Poisson P(4) datasets of size n = 1000, for a0 = .0001, .001, .01, .1, .2, .3, .4, .5. Each posterior approximation is based on 104 Metropolis-Hastings iterations.
  • 157. Consistency 0 1 2 3 4 5 6 7 0.00.20.40.60.81.0 a0=.1 log(sample size) 0 1 2 3 4 5 6 7 0.00.20.40.60.81.0 a0=.3 log(sample size) 0 1 2 3 4 5 6 7 0.00.20.40.60.81.0 a0=.5 log(sample size) Posterior means (sky-blue) and medians (grey-dotted) of α, over 100 Poisson P(4) datasets for sample sizes from 1 to 1000.
  • 158. Behaviour of Bayes factor 0 1 2 3 4 5 6 7 0.00.20.40.60.81.0 log(sample size) 0 1 2 3 4 5 6 7 0.00.20.40.60.81.0 log(sample size) 0 1 2 3 4 5 6 7 0.00.20.40.60.81.0 log(sample size) Comparison between P(M1|x) (red dotted area) and posterior medians of α (grey zone) for 100 Poisson P(4) datasets with sample sizes n between 1 and 1000, for a0 = .001, .1, .5
  • 159. Normal-normal comparison comparison of a normal N(θ1, 1) with a normal N(θ2, 2) distribution mixture with identical location parameter θ αN(θ, 1) + (1 − α)N(θ, 2) Jeffreys prior π(θ) = 1 can be used, since posterior is proper Reference (improper) Bayes factor B12 = 2 n−1/2 exp 1/4 n i=1 (xi − ¯x)2 ,
  • 160. Normal-normal comparison comparison of a normal N(θ1, 1) with a normal N(θ2, 2) distribution mixture with identical location parameter θ αN(θ, 1) + (1 − α)N(θ, 2) Jeffreys prior π(θ) = 1 can be used, since posterior is proper Reference (improper) Bayes factor B12 = 2 n−1/2 exp 1/4 n i=1 (xi − ¯x)2 ,
  • 161. Normal-normal comparison comparison of a normal N(θ1, 1) with a normal N(θ2, 2) distribution mixture with identical location parameter θ αN(θ, 1) + (1 − α)N(θ, 2) Jeffreys prior π(θ) = 1 can be used, since posterior is proper Reference (improper) Bayes factor B12 = 2 n−1/2 exp 1/4 n i=1 (xi − ¯x)2 ,
  • 162. Consistency 0.1 0.2 0.3 0.4 0.5 p(M1|x) 0.00.20.40.60.81.0 n=15 0.00.20.40.60.81.0 0.1 0.2 0.3 0.4 0.5 p(M1|x) 0.650.700.750.800.850.900.951.00 n=50 0.650.700.750.800.850.900.951.00 0.1 0.2 0.3 0.4 0.5 p(M1|x) 0.750.800.850.900.951.00 n=100 0.750.800.850.900.951.00 0.1 0.2 0.3 0.4 0.5 p(M1|x) 0.930.940.950.960.970.980.991.00 n=500 0.930.940.950.960.970.980.991.00 Posterior means (wheat) and medians of α (dark wheat), compared with posterior probabilities of M0 (gray) for a N(0, 1) sample, derived from 100 datasets for sample sizes equal to 15, 50, 100, 500. Each posterior approximation is based on 104 MCMC iterations.
  • 163. Comparison with posterior probability 0 100 200 300 400 500 -50-40-30-20-100 a0=.1 sample size 0 100 200 300 400 500 -50-40-30-20-100 a0=.3 sample size 0 100 200 300 400 500 -50-40-30-20-100 a0=.4 sample size 0 100 200 300 400 500 -50-40-30-20-100 a0=.5 sample size Plots of ranges of log(n) log(1 − E[α|x]) (gray color) and log(1 − p(M1|x)) (red dotted) over 100 N(0, 1) samples as sample size n grows from 1 to 500. and α is the weight of N(0, 1) in the mixture model. The shaded areas indicate the range of the estimations and each plot is based on a Beta prior with a0 = .1, .2, .3, .4, .5, 1 and each posterior approximation is based on 104 iterations.
  • 164. Comments convergence to one boundary value as sample size n grows impact of hyperarameter a0 slowly vanishes as n increases, but present for moderate sample sizes when simulated sample is neither from N(θ1, 1) nor from N(θ2, 2), behaviour of posterior varies, depending on which distribution is closest
  • 165. Logit or Probit? binary dataset, R dataset about diabetes in 200 Pima Indian women with body mass index as explanatory variable comparison of logit and probit fits could be suitable. We are thus comparing both fits via our method M1 : yi | xi , θ1 ∼ B(1, pi ) where pi = exp(xi θ1) 1 + exp(xi θ1) M2 : yi | xi , θ2 ∼ B(1, qi ) where qi = Φ(xi θ2)
  • 166. Common parameterisation Local reparameterisation strategy that rescales parameters of the probit model M2 so that the MLE’s of both models coincide. [Choudhuty et al., 2007] Φ(xi θ2) ≈ exp(kxi θ2) 1 + exp(kxi θ2) and use best estimate of k to bring both parameters into coherency (k0, k1) = (θ01/θ02, θ11/θ12) , reparameterise M1 and M2 as M1 :yi | xi , θ ∼ B(1, pi ) where pi = exp(xi θ) 1 + exp(xi θ) M2 :yi | xi , θ ∼ B(1, qi ) where qi = Φ(xi (κ−1 θ)) , with κ−1θ = (θ0/k0, θ1/k1).
  • 167. Prior modelling Under default g-prior θ ∼ N2(0, n(XT X)−1 ) full conditional posterior distributions given allocations π(θ | y, X, ζ) ∝ exp i Iζi =1yi xi θ i;ζi =1[1 + exp(xi θ)] exp −θT (XT X)θ 2n × i;ζi =2 Φ(xi (κ−1 θ))yi (1 − Φ(xi (κ−1 θ)))(1−yi ) hence posterior distribution clearly defined
  • 168. Results Logistic Probit a0 α θ0 θ1 θ0 k0 θ1 k1 .1 .352 -4.06 .103 -2.51 .064 .2 .427 -4.03 .103 -2.49 .064 .3 .440 -4.02 .102 -2.49 .063 .4 .456 -4.01 .102 -2.48 .063 .5 .449 -4.05 .103 -2.51 .064 Histograms of posteriors of α in favour of logistic model where a0 = .1, .2, .3, .4, .5 for (a) Pima dataset, (b) Data from logistic model, (c) Data from probit model
  • 169. Survival analysis Testing hypothesis that data comes from a 1. log-Normal(φ, κ2), 2. Weibull(α, λ), or 3. log-Logistic(γ, δ) distribution Corresponding mixture given by the density α1 exp{−(log x − φ)2 /2κ2 }/ √ 2πxκ+ α2 α λ exp{−(x/λ)α }((x/λ)α−1 + α3(δ/γ)(x/γ)δ−1 /(1 + (x/γ)δ )2 where α1 + α2 + α3 = 1
  • 170. Survival analysis Testing hypothesis that data comes from a 1. log-Normal(φ, κ2), 2. Weibull(α, λ), or 3. log-Logistic(γ, δ) distribution Corresponding mixture given by the density α1 exp{−(log x − φ)2 /2κ2 }/ √ 2πxκ+ α2 α λ exp{−(x/λ)α }((x/λ)α−1 + α3(δ/γ)(x/γ)δ−1 /(1 + (x/γ)δ )2 where α1 + α2 + α3 = 1
  • 171. Reparameterisation Looking for common parameter(s): φ = µ + γβ = ξ σ2 = π2 β2 /6 = ζ2 π2 /3 where γ ≈ 0.5772 is Euler-Mascheroni constant. Allows for a noninformative prior on the common location scale parameter, π(φ, σ2 ) = 1/σ2
  • 172. Reparameterisation Looking for common parameter(s): φ = µ + γβ = ξ σ2 = π2 β2 /6 = ζ2 π2 /3 where γ ≈ 0.5772 is Euler-Mascheroni constant. Allows for a noninformative prior on the common location scale parameter, π(φ, σ2 ) = 1/σ2
  • 173. Recovery .01 0.1 1.0 10.0 0.860.880.900.920.940.960.981.00 .01 0.1 1.0 10.00.000.020.040.060.08 Boxplots of the posterior distributions of the Normal weight α1 under the two scenarii: truth = Normal (left panel), truth = Gumbel (right panel), a0=0.01, 0.1, 1.0, 10.0 (from left to right in each panel) and n = 10, 000 simulated observations.
  • 174. Asymptotic consistency Posterior consistency holds for mixture testing procedure [under minor conditions] Two different cases the two models, M1 and M2, are well separated model M1 is a submodel of M2.
  • 175. Asymptotic consistency Posterior consistency holds for mixture testing procedure [under minor conditions] Two different cases the two models, M1 and M2, are well separated model M1 is a submodel of M2.
  • 176. Posterior concentration rate Let π be the prior and xn = (x1, · · · , xn) a sample with true density f ∗ proposition Assume that, for all c > 0, there exist Θn ⊂ Θ1 × Θ2 and B > 0 such that π [Θc n] n−c , Θn ⊂ { θ1 + θ2 nB } and that there exist H 0 and L, δ > 0 such that, for j = 1, 2, sup θ,θ ∈Θn fj,θj − fj,θj 1 LnH θj − θj , θ = (θ1, θ2), θ = (θ1, θ2) , ∀ θj − θ∗ j δ; KL(fj,θj , fj,θ∗ j ) θj − θ∗ j . Then, when f ∗ = fθ∗,α∗ , with α∗ ∈ [0, 1], there exists M > 0 such that π (α, θ); fθ,α − f ∗ 1 > M log n/n|xn = op(1) .
  • 177. Separated models Assumption: Models are separated, i.e. identifiability holds: ∀α, α ∈ [0, 1], ∀θj , θj , j = 1, 2 Pθ,α = Pθ ,α ⇒ α = α , θ = θ Further inf θ1∈Θ1 inf θ2∈Θ2 f1,θ1 − f2,θ2 1 > 0 and, for θ∗ j ∈ Θj , if Pθj weakly converges to Pθ∗ j , then θj −→ θ∗ j in the Euclidean topology
  • 178. Separated models Assumption: Models are separated, i.e. identifiability holds: ∀α, α ∈ [0, 1], ∀θj , θj , j = 1, 2 Pθ,α = Pθ ,α ⇒ α = α , θ = θ theorem Under above assumptions, then for all > 0, π [|α − α∗ | > |xn ] = op(1)
  • 179. Separated models Assumption: Models are separated, i.e. identifiability holds: ∀α, α ∈ [0, 1], ∀θj , θj , j = 1, 2 Pθ,α = Pθ ,α ⇒ α = α , θ = θ theorem If θj → fj,θj is C2 around θ∗ j , j = 1, 2, f1,θ∗ 1 − f2,θ∗ 2 , f1,θ∗ 1 , f2,θ∗ 2 are linearly independent in y and there exists δ > 0 such that f1,θ∗ 1 , f2,θ∗ 2 , sup |θ1−θ∗ 1 |<δ |D2 f1,θ1 |, sup |θ2−θ∗ 2 |<δ |D2 f2,θ2 | ∈ L1 then π |α − α∗ | > M log n/n xn = op(1).
  • 180. Separated models Assumption: Models are separated, i.e. identifiability holds: ∀α, α ∈ [0, 1], ∀θj , θj , j = 1, 2 Pθ,α = Pθ ,α ⇒ α = α , θ = θ theorem allows for interpretation of α under the posterior: If data xn is generated from model M1 then posterior on α concentrates around α = 1
  • 181. Embedded case Here M1 is a submodel of M2, i.e. θ2 = (θ1, ψ) and θ2 = (θ1, ψ0 = 0) corresponds to f2,θ2 ∈ M1 Same posterior concentration rate log n/n for estimating α when α∗ ∈ (0, 1) and ψ∗ = 0.
  • 182. Null case Case where ψ∗ = 0, i.e., f ∗ is in model M1 Two possible paths to approximate f ∗: either α goes to 1 (path 1) or ψ goes to 0 (path 2) New identifiability condition: Pθ,α = P∗ only if α = 1, θ1 = θ∗ 1, θ2 = (θ∗ 1, ψ) or α 1, θ1 = θ∗ 1, θ2 = (θ∗ 1, 0) Prior π(α, θ) = πα(α)π1(θ1)πψ(ψ), θ2 = (θ1, ψ) with common (prior on) θ1
  • 183. Null case Case where ψ∗ = 0, i.e., f ∗ is in model M1 Two possible paths to approximate f ∗: either α goes to 1 (path 1) or ψ goes to 0 (path 2) New identifiability condition: Pθ,α = P∗ only if α = 1, θ1 = θ∗ 1, θ2 = (θ∗ 1, ψ) or α 1, θ1 = θ∗ 1, θ2 = (θ∗ 1, 0) Prior π(α, θ) = πα(α)π1(θ1)πψ(ψ), θ2 = (θ1, ψ) with common (prior on) θ1
  • 184. Assumptions [B1] Regularity: Assume that θ1 → f1,θ1 and θ2 → f2,θ2 are 3 times continuously differentiable and that F∗ ¯f 3 1,θ∗ 1 f 3 1,θ∗ 1 < +∞, ¯f1,θ∗ 1 = sup |θ1−θ∗ 1 |<δ f1,θ1 , f 1,θ∗ 1 = inf |θ1−θ∗ 1 |<δ f1,θ1 F∗ sup|θ1−θ∗ 1 |<δ | f1,θ∗ 1 |3 f 3 1,θ∗ 1 < +∞, F∗ | f1,θ∗ 1 |4 f 4 1,θ∗ 1 < +∞, F∗ sup|θ1−θ∗ 1 |<δ |D2 f1,θ∗ 1 |2 f 2 1,θ∗ 1 < +∞, F∗ sup|θ1−θ∗ 1 |<δ |D3 f1,θ∗ 1 | f 1,θ∗ 1 < +∞
  • 185. Assumptions [B2] Integrability: There exists S0 ⊂ S ∩ {|ψ| > δ0} for some positive δ0 and satisfying Leb(S0) > 0, and such that for all ψ ∈ S0, F∗ sup|θ1−θ∗ 1 |<δ f2,θ1,ψ f 4 1,θ∗ 1 < +∞, F∗ sup|θ1−θ∗ 1 |<δ f 3 2,θ1,ψ f 3 1,θ1∗ < +∞,
  • 186. Assumptions [B3] Stronger identifiability: Set f2,θ∗ 1,ψ∗ (x) = θ1 f2,θ∗ 1,ψ∗ (x)T , ψf2,θ∗ 1,ψ∗ (x)T T . Then for all ψ ∈ S with ψ = 0, if η0 ∈ R, η1 ∈ Rd1 η0(f1,θ∗ 1 − f2,θ∗ 1,ψ) + ηT 1 θ1 f1,θ∗ 1 = 0 ⇔ η1 = 0, η2 = 0
  • 187. Consistency theorem Given the mixture fθ1,ψ,α = αf1,θ1 + (1 − α)f2,θ1,ψ and a sample xn = (x1, · · · , xn) issued from f1,θ∗ 1 , under assumptions B1 − B3, and an M > 0 such that π (α, θ); fθ,α − f ∗ 1 > M log n/n|xn = op(1). If α ∼ B(a1, a2), with a2 < d2, and if the prior πθ1,ψ is absolutely continuous with positive and continuous density at (θ∗ 1, 0), then for Mn −→ ∞ π |α − α∗ | > Mn(log n)γ / √ n|xn = op(1), γ = max((d1 + a2)/(d2 − a2), 1)/2,