MCMSki IV (the 5th IMS-ISBA joint meeting) January 2014 Chamonix Mont-Blanc, France The associated journal article has now been uploaded to arXiv: http://arxiv.org/abs/1403.4359

1. Background Pre-computation Simulation Study Conclusion
Pre-computation for ABC in image analysis
Matt Moores1,2 Kerrie Mengersen1,2 Christian Robert3,4
1Mathematical Sciences School, Queensland University of Technology,
Brisbane, Australia
2Institute for Health and Biomedical Innovation, QUT Kelvin Grove
3CEREMADE, Universit´e Paris Dauphine, France
4CREST, INSEE, France
MCMSki IV, Chamonix 2014

2. Background Pre-computation Simulation Study Conclusion
Outline
1 Background
Approximate Bayesian Computation (ABC)
Sequential Monte Carlo (SMC-ABC)
Hidden Potts model
2 Pre-computation
3 Simulation Study

3. Background Pre-computation Simulation Study Conclusion
Background
Image analysis often involves:
Large datasets, with millions of pixels
Multiple images with similar characteristics
For example: satellite remote sensing (Landsat), computed
tomography (CT)
Table : Scale of common types of images
Number Landsat CT slices
of pixels (90m2/px) (512×512)
26 0.06km2
. . .
56 14.06km2
0.1
106 900.00km2
3.8
156 10251.56km2
43.5

4. Background Pre-computation Simulation Study Conclusion
Approximate Bayesian Computation (ABC)
Algorithm 1 ABC rejection sampler
1: for all iterations t ∈ 1 . . . T do
2: Draw independent proposal θ ∼ π(θ)
3: Generate x ∼ f(·|θ )
4: if |ρ(x) − ρ(y)| < then
5: set θt ← θ
6: else
7: set θt ← θt−1
8: end if
9: end for
Pritchard, Seielstad, Perez-Lezaun & Feldman (1999) Mol. Biol. Evol. 16(12)
Marin, Pudlo, Robert & Ryder (2012) Stat. Comput. 22(6)

5. Background Pre-computation Simulation Study Conclusion
Adaptive ABC using Sequential Monte Carlo (SMC-ABC)
Algorithm 2 SMC-ABC
1: Draw N particles θi ∼ π(θ)
2: Generate pseudo-data xi,m ∼ f(·|θi)
3: repeat
4: Adaptively select ABC tolerance t
5: Update importance weights ωi for each particle
6: if effective sample size (ESS) < Nmin then
7: Resample particles according to their weights
8: end if
9: Update particles using random walk proposal
(with adaptive RWMH bandwidth σ2
t )
10: until
naccept
N < 0.015 or t = 0
Del Moral, Doucet, & Jasra (2012) Stat. Comput. 22(5)
Liu (2001) Monte Carlo Strategies in Scientific Computing New York: Springer

6. Background Pre-computation Simulation Study Conclusion
Motivation
Computational cost is dominated by simulation of pseudo-data
e.g. Hidden Potts model in image analysis
(Grelaud et al. 2009, Everitt 2012)
Model fitting with ABC can be separated into:
Learning about the summary statistic, given the parameter
ρ(x) | θ
Choosing parameter values, given a summary statistic
θ | ρ(y)
For latent models, an additional step of learning about the
summary statistic, given the data: ρ(z) | y, θ
Grelaud, Robert, Marin, Rodolphe & Taly (2009) Bayesian Analysis 4(2)
Everitt (2012) JCGS 21(4)

7. Background Pre-computation Simulation Study Conclusion
hidden Markov random field
Joint distribution of observed pixel intensities yi ∈ y
and latent labels zi ∈ z:
Pr(y, z|µ, σ2
) ∝ L(y|µ, σ2
, z)π(µ|σ2
)π(σ2
)π(z|β)π(β) (1)
Additive Gaussian noise:
yi|zi =j
iid
∼ N µj, σ2
j (2)
Potts model:
π(zi|zi∼ , β) =
exp {β i∼ δ(zi, z )}
k
j=1 exp {β i∼ δ(j, z )}
(3)
Potts (1952) Proceedings of the Cambridge Philosophical Society 48(1)

8. Background Pre-computation Simulation Study Conclusion
Inverse Temperature

9. Background Pre-computation Simulation Study Conclusion
Doubly-intractable likelihood
p(β|z) = C(β)−1
π(β) exp {β S(z)} (4)
The normalising constant of the Potts model has computational
complexity of O(n2kn), since it involves a sum over all possible
combinations of the labels z ∈ Z:
C(β) =
z∈Z
exp {β S(z)} (5)
S(z) is the sufficient statistic of the Potts model:
S(z) =
i∼ ∈L
δ(zi, z ) (6)
where L is the set of all unique neighbour pairs.

10. Background Pre-computation Simulation Study Conclusion
Pre-computation
The distribution of ρ(x) | θ is independent of the data
By simulating pseudo-data for values of θ, we can create a
mapping function ˆf(θ) to approximate E[ρ(x)|θ]
This mapping function can be reused across multiple datasets,
amortising its computational cost
By mapping directly from θ → ρ(x), we avoid the need to simulate
pseudo-data during model fitting

11. Background Pre-computation Simulation Study Conclusion
Sufficient statistic of the Potts model
0.0 0.5 1.0 1.5 2.0 2.5 3.0
1000015000200002500030000
β
S(z)
(a) E(S(z)|β)
0.0 0.5 1.0 1.5 2.0 2.5 3.0050100150200250
β
σ(S(z))
(b) σ(S(z)|β)
Figure : Distribution of S(z) | β for n = 56
, k = 3

12. Background Pre-computation Simulation Study Conclusion
Scalable SMC-ABC for the hidden Potts model
Algorithm 3 SMC-ABC using precomputed ˆf(β)
1: Draw N particles βi ∼ π0(β)
2: Approximate sufficient statistics S(xi,m) ≈ ˆf(βi)
3: repeat
4: Update S(zt)|y, πt(β)
5: Adaptively select ABC tolerance t
6: Update importance weights ωi for each particle
7: if effective sample size (ESS) < Nmin then
8: Resample particles according to their weights
9: end if
10: Update particles using random walk proposal
(with adaptive RWMH bandwidth σ2
t )
11: until
naccept
N < 0.015 or t < 10−9 or t ≥ 100

13. Background Pre-computation Simulation Study Conclusion
Simulation Study
20 images, n = 125 × 125, k = 3:
β ∼ U(0, 1.005)
z ∼ f(·|β) using 2000 iterations of Swendsen-Wang
µj ∼ N 0, 1002
1
σ2
j
∼ Γ (1, 100)
Comparison of 2 ABC algorithms:
Scalable SMC-ABC using precomputed ˆf(β)
Standard SMC-ABC using 500 iterations of Gibbs sampling
Swendsen & Wang (1987) Physical Review Letters 58

14. Background Pre-computation Simulation Study Conclusion
Accuracy of posterior estimates for β
0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
β
posteriordistribution
(a) pseudo-data
0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
β
posteriordistribution
(b) pre-computed

15. Background Pre-computation Simulation Study Conclusion
Distribution of posterior sampling error for β
algorithm
error
0.0
0.2
0.4
0.6
Pseudo−data Pre−computed

16. Background Pre-computation Simulation Study Conclusion
Improvement in runtime
Pseudo−data Pre−computed
0.51.02.05.010.020.050.0100.0
algorithm
elapsedtime(hours)
(a) elapsed (wall clock) time
Pseudo−data Pre−computed
51020501002005001000
algorithm
CPUtime(hours)
(b) CPU time

17. Background Pre-computation Simulation Study Conclusion
Summary
Scalability of SMC-ABC can be improved by pre-computing an
approximate mapping θ → ρ(x)
Pre-computation took 8 minutes on a 16 core Xeon server
Average runtime for SMC-ABC improved from 74.4 hours to
39 minutes
The mapping function represents the nonlinear, heteroskedastic
relationship between the parameter and the summary statistic.
This method could be extended to multivariate applications, such
as estimating both β and k for the hidden Potts model.

18. Appendix
Acknowledgements
I gratefully acknowledge the financial support received from:
Mathematical Sciences School,
Queensland University of Technology, Brisbane, Australia
Institute for Health and Biomedical Innovation, QUT
Bayesian section of the American Statistical Association
International Society for Bayesian Analysis
BayesComp section of ISBA
CEREMADE, Universit´e Paris Dauphine, France
Department of Economics, University of Warwick, UK
Computational resources and services used in this work were
provided by the HPC and Research Support Group, QUT.

19. Appendix
For Further Reading I
Jun S. Liu
Monte Carlo Strategies in Scientific Computing
Springer-Verlag, 2001.
Pierre Del Moral, Arnaud Doucet & Ajay Jasra
An adaptive sequential Monte Carlo method for approximate Bayesian
computation.
Statistics & Computing, 22(5): 1009–20, 2012.
Richard Everitt
Bayesian Parameter Estimation for Latent Markov Random Fields and
Social Networks.
J. Comput. Graph. Stat., 21(4): 940–60, 2012.
A. Grelaud, C. P. Robert, J.-M. Marin, F. Rodolphe & J.-F. Taly
ABC likelihood-free methods for model choice in Gibbs random fields.
Bayesian Analysis, 4(2): 317–36, 2009.

20. Appendix
For Further Reading II
J.-M. Marin, P. Pudlo, C. P. Robert & R. J. Ryder
Approximate Bayesian computational methods.
Statistics & Computing, 22(6): 1167–80, 2012.
Renfrey B. Potts
Some generalized order-disorder transformations.
Proc. Cambridge Philosophical Society, 48(1): 106–9, 1952.
J. K. Pritchard, M. T. Seielstad, A. Perez-Lezaun & M. W. Feldman
Population growth of human Y chromosomes: a study of Y chromosome
microsatellites
Mol. Biol. Evol., 16(12): 1791–8, 1999.
R. H. Swendsen & J.-S. Wang
Nonuniversal critical dynamics in Monte Carlo simulations.
Physical Review Letters, 58: 86–8, 1987.

21. Appendix
ABC tolerance levels
Iteration
0 20 40 60 80 100
0200040006000
ε
σ
(a)
Iteration
0 20 40 60 80 100
0200040006000800010000
ε
σ
(b)

22. Appendix
Sufficient Statistic
Iteration
S(z)
0 20 40 60 80 100
15000155001600016500170001750018000
(a)
Iteration
S(z)
0 20 40 60 80 100
100001200014000160001800020000
(b)

23. Appendix
RWMH acceptance rate
Iteration
proportionaccepted
0 20 40 60 80 100
0.00.20.40.60.81.0
(a)
Iteration
proportionaccepted
0 20 40 60 80 100
0.00.20.40.60.81.0
(b)

24. Appendix
Effective Sample Size
Iteration
ESS
0 20 40 60 80 100
0200040006000800010000
(a)
Iteration
ESS
0 20 40 60 80 100
0200040006000800010000
(b)