1. Deterministic Sampling
for
quantification of modeling uncertainty
– with application to
Dynamic Metrology
Jan Peter Hessling
Kapernicus AB
E-mail: peter(at)kapernicus.com
Tel.: +46 72 50 407 55
KAPERNICUS

2. My story…1991 – present
KAPERNICUS
●
M.Sc. (Swe) Applied Physics, Linköping Univ., Sweden
●
M.Sc. (Am.) Physics, Univ. of Massachusetts, USA:
Quantum Physics – Applied Math.
●
Ph.D. Theoretical Physics, Chalmers Univ. Sweden:
Superconductivity – Current fluct. in heterostructures (–1996)
●
STRI – ABB: Electrical Power Systems
– Voltage fluctuations from e-arc furnaces
●
ABB Corporate Research: Electrical power generation,
Windpower – Windformer park,…
●
Allgon Systems: Telecommunication – Microwave filters,
Tuning support microwave filters (software)
●
SP: Analysis of Dynamic Measurements
– Dynamic Metrology (2004–2011)
●
SP: Uncertain Scientific Modeling
– Deterministic Sampling Methods (2010-2015)

3. KAPERNICUS
“…And what is good, Phaedrus,
And what is not good –
Need we ask anyone to tell us these
things”
Robert M. Pirsig (1974), Zen and the art of motorcycle maintenance

4. KAPERNICUS
Applications

5. Ex. Meteorology – Weather forecasting
SMHI Ensemble forecasting http://www.smhi.se/en/publications/hydrological-
ensemble-forecasts-hydrologiska-ensembleprognoser-1.6763
KAPERNICUS

6. Ex. Medicine – Differential diagnosis of neurological disorders
Identification of dynamic models of error correction
KAPERNICUS
Normal,
Alzheimer, etc

7. Ex. Deterministic Sampling – Dynamic Metrology
'Spaghetti'-plot step response uncertain [sensor] model
0 2 0 4 0 6 0 8 0 1 0 0
0
0 .0 1
0 .0 2
0 .0 3
0 .0 4
0 .0 5
0 .0 6
S a m p le
Stepresponse
S t d M E
0 .0 5 M E
S t d M E - S t d M C - U
KAPERNICUS

8. 0 .8 1
- 0 . 3
- 0 . 2
- 0 . 1
0
0 . 1
0 . 2
0 . 3
R e ( z )
Im(z)
Ex. Spaghettiplot Dynamic Metrology...
– A Selection of Deterministic Ensembles
Sigma-points (UKF):
Sampled poles of digital filter
KAPERNICUS
Hessling J.P., Deterministic Sampling for propagating model covariance,
SIAM/ASA J. Uncertainty Quantification, 1(1), 297–318, (2013):

9. Applications Deterministic Sampling
KAPERNICUS
•
Dynamic Metrology*
•
Signal Processing*
•
Nuclear power*
•
Medicine (models of timing)*
•
High speed electronic devices*
•
Automotive industry*
•
Fire safety simulations*
•
Meteorology*
•
Robust control systems MPC (Model Predictive Control)
•
Electrical Power Supply, ”Smart Grids” etc.
•
Aerospace industry
•
Safety of large infrastructures, bridges, towers, etc.
•
Econometrics
•
Financial budgets…
•
Stock markets
•
…

10. KAPERNICUS
Why?

11. Why sample deterministically and not randomly? (“like
everybody else...”)
Random sampling:
● Not efficient (large ensembles)
● Not reproducible (sampling variance)
KAPERNICUS
1 0
0
1 0
1
1 0
2
1 0
3
0
0 .5
1
1 .5
2
2 .5
A n ta l s a m p e l
P a r a m e t e r
D S
D S
M e a n M C
S td M C
M e a n
S td
1 0
0
1 0
1
1 0
2
1 0
3
6
8
1 0
1 2
1 4
1 6
1 8
2 0
2 2
2 4
2 6
A n ta l s a m p e l
M o d e l
D S
D S
M e a n M C
S td M C
M e a n L IN
S td L IN
 
 
 
 4.2,6.1
,4,3,2,,~
4.0,2,
2,1
)(
4



qqDS
qq
k
MC
qq
nkNq
qqh




DS: Deterministic Sampling
MC: Monte Carlo /
Random Sampling

12. Stratification, LHS, orthogonal sampling etc
effectively add determinism to random sampling...
KAPERNICUS

13. KAPERNICUS
Some methods of
deterministic
sampling

14. The BIG picture…
KAPERNICUS
1
3
2

15. [Direct] Uncertainty Quantification
Synthesis of deterministic ensembles
KAPERNICUS
1
● Match modeled and known mean and covariance
Consistent to second order
● Excitation matrix <=> Type of ensemble
 
ImVV
VImVV
VSUUh
USUhh
T
mT
Tm
T






ˆˆ
01ˆ,ˆˆ
ˆ1
,cov
1
1
2
21

16. Some excitation matrices / DS ensembles
̂V HAD =
(H H
H −H )
=
(1 1
1 −1)∗
(1 1
1 −1)...
 
  












































2221
1211
1
ˆˆ
ˆˆ
ˆ
1111
1111
1111
ˆ
11ˆ
ˆ
VV
VV
NV
V
InV
IInV
CMB
BIN
nnnSPX
nnnnSTD
KAPERNICUS

17. Sample Annealing
Numerical synthesis of deterministic ensembles
Match arbitrary set of statistical moments of ensemble
and parameters (known!):
● Parameter Ensemble
– Samples (Strongly non-linear)
– Weights (Linear!)
● Annealing idea:
– Iterate
– Generate trial sample displacements
– Evaluate weights against parameter statistics
– Penalize non-uniformity of weights – reject/accept trial
sample displacement
KAPERNICUS

18. KAPERNICUS
Ex. Annealing
Univariate lognormal distribution
Result
Mom Prop Direct Correct
M1 1.1288 1.1330 1.1330
M2 0.2911 0.3648 0.3648
M3 0.0564 0.3866 0.3866
Weights and ensemble
Propagated
0.2500 1.3151
0.2500 0.7599
0.2500 1.9190
0.2500 0.5212
Direct
0.3348 0.6874
0.3346 0.6698
0.3319 2.0276
-0.0013 -4.2718
y≡log(x)∼ N (μ=0,σ=0.5)

19. Inverse Uncertainty Quantification
– Identification of models
“All models are wrong, and to suppose that
inputs should always be set to their ‘true’
values when these are ‘known’ is to invest the
model with too much credibility in practice.
Treating a model more pragmatically, as
having inputs that we can ‘tweak’ empirically,
can increase its value and predictive power.”
KAPERNICUS
Kennedy, M. C., & O'Hagan, A. (2001). Bayesian calibration of computer models.
Journal of the Royal Statistical Society. Series B, Statistical Methodology, 425-464:

20. Sample Annealing revisited
– for inverse uncertainty quantification
Match set of statistical moments of
model prediction and calibration data:
● Model Ensemble (hypothetical model prediction)
– Fields of evaluated samples (Strongly non-linear)
– Weights (Linear!)
● Annealing idea:
– Iterate
– Generate trial sample displacements
– Evaluate model for each displaced sample
– Evaluate weights against calibration data
– Penalize non-uniformity of weights – reject/accept trial
sample displacementKAPERNICUS

21. Inverse Uncertainty Quantification /
Identification of model ensembles
2
Hessling J.P. (2014). Identification of complex models,
SIAM/ASA J. Uncertainty Quantification, 2(1), 717–744:
● Identification of parameter ensembles
(not best estimates)
● Fix point iteration of identification
=> match regression and sample points
● Bayesian inclusion of prior ensemble
● Complete information with maximum entropy
●
Consistent sampling to 2nd
order
● Ex. Wiener filter (predecessor to Kalman Filter)
KAPERNICUS
3

22. 3
Hessling J.P. (2014). Identification of complex models,
SIAM/ASA J. Uncertainty Quantification, 2(1), 717–744:
Bayes rule for deterministic ensembles
● Bayes rule given for prob. distr. – not(!) deterministic ensembles:
● Solution (indirect)
– Complete information with principle of maximum entropy
(=> Gaussian pdf for 1st
and 2nd
order moments)
– Change representation from ensembles to pdf
– Apply Bayes rule
– Synthesize ensemble, i.e. change representation back
● NOTE: Prior information and derived information (Max Likelihood)
must be independent!
KAPERNICUS
P(Model∣Data)∝P(Data∣Model)P(Model )

23. - 0 .1 - 0 .0 5 0 0 .0 5 0 .1 0 .1 5 0 .2 0 .2 5
0 .7
0 .7 5
0 .8
0 .8 5
0 .9
0 .9 5
1
1 .0 5

( 1 )
1
= P R IO R
( : ,1 )

( 1 )
2
= 
P R IO R
( : ,2 ) 
( 1 )
3
= 
P R I O R
( : ,3 )

( 1 )
4
= 
P R IO R
( : ,4 )

P O S T
= 
( N )
1
2
T r u e
0 0 .5 1 1 .5
0
0 .2
0 .4
0 .6
0 .8
1
1 .2
1 .4
'F i e ld ' x
'Measured'/Priorpredictionh(x,)
C a l d a ta
+ 2 
- 2 
P o s t M o d e l E n s
0 0 .5 1 1 .5
- 0 .2
0
0 .2
0 .4
0 .6
0 .8
1
1 .2
'F i e ld ' x
'Measured'/Priorpredictionh(x,)
C a l d a ta
+ 2 
- 2 
P r i o r M o d e l E n s
   
  ?cov?,
sin,, 2121



 xxh
Hessling J.P. (2014). Identification of complex models,
SIAM/ASA J. Uncertainty Quantification, 2(1), 717–744:
Iterated identificaion – Toy model (Prelude)
ΣPRIOR →Ψ[1] →Σ[1] →Ψ[2]…→ΣPOST =ΨPOST

24. Hessling J.P. (2014). Identification of complex models, SIAM/ASA J.
Uncertainty Quantification, 2(1), 717–744:
– dramatic(!) effect of matching covariance for toy model
KAPERNICUS
Anti-correlate...
Even larger effect (std) for larger number of parameters!
Why?

25. Hessling J.P. (2014). Identification of complex models, SIAM/ASA J.
Uncertainty Quantification, 2(1), 717–744:
Example Inertial Navigator – Dynamic Metrology
KAPERNICUS
Ensemble of
Wiener filters

26. KAPERNICUS
Hessling J.P. (2014). Identification of complex models, SIAM/ASA J.
Uncertainty Quantification, 2(1), 717–744:
Example Inertial Navigator – Identified WF ensemble

27. KAPERNICUS
Hessling J.P. (2014). Identification of complex models, SIAM/ASA J.
Uncertainty Quantification, 2(1), 717–744:
Example Inertial Navigator – Convergence WF ensemble

28. KAPERNICUS
Hessling J.P. (2014). Identification of complex models, SIAM/ASA J.
Uncertainty Quantification, 2(1), 717–744:
Example Inertial Navigator – Identifiability
- 0 .5 - 0 .4 - 0 .3 - 0 .2 - 0 .1 0 0 .1 0 .2 0 .3 0 .4 0 .5
- 0 .5
- 0 .4
- 0 .3
- 0 .2
- 0 .1
0
0 .1
0 .2
0 .3
0 .4
0 .5
R e ( s )
Im(s)
p
r
p
x
p
x


2
/v
2
= 0 .1


2
/v
2
= 1 0 0
- 0 .5 - 0 .4 - 0 .3 - 0 .2 - 0 .1 0 0 .1 0 .2 0 .3 0 .4 0 .5
- 0 .5
- 0 .4
- 0 .3
- 0 .2
- 0 .1
0
0 .1
0 .2
0 .3
0 .4
0 .5
R e ( s )
Im(s)
p
r
p
x
p
x
w x
0 .1
w x
Nearly degenerate – fix one of them!

29. KAPERNICUS
Hessling J.P. (2014). Identification of complex models, SIAM/ASA J.
Uncertainty Quantification, 2(1), 717–744:
Example Inertial Navigator – Residual analysis
0 5 1 0 1 5 2 0 2 5 3 0
- 1
- 0 .8
- 0 .6
- 0 .4
- 0 .2
0
0 .2
0 .4
0 .6
0 .8
1
L a g ( s a m p le s )
1  s t d ( c
u
( 
[ 0 ]
) ) - 1
1 0 0  s t d ( c
u
( [ 2 2 ]
) ) - 1
m e a n ( c y
( [2 2 ]
) )
m e a n ( c u
( [0 ]
) )
m e a n ( c u
( [2 2 ]
) )
0 5 1 0 1 5 2 0 2 5 3 0
- 2
- 1
0
1
2
3
4
x 1 0
- 3
L a g ( s a m p le s )
m e a n ( c
x
( [2 2 ]
) )
s td ( c
x
( [2 2 ]
) )  1 0 0
Autocorrelation of residual Autocorrelation of innovation
NOTE: Uncertain model=Ensemble of Wiener filters => Mean and Std

30. Advanced application of deterministic sampling
Medicine: Model of error correction (revisited)
Metronome

31. Challenge: Medicine Model of error correction
Deterministic sampling of sampling covariance!
KAPERNICUS
Std > Mean !
R.A. Fisher says:
“Acov insignificant!”

32. KAPERNICUS
Foundations

33. Relevance of modeling
KAPERNICUS
● Deterministic sampling
– Approximates(?) less well known statistical information
– Respects reliable structure of models
with minimal use of surrogates
● Scalable
– How many samples can be afforded?
● High efficiency enables
– Identification of complex models
– Iterative identification
– Illustrate credibility of uncertainty, i.e.
“the uncertainty of the uncertainty”
(using alternative ensembles)
– ???

34. Fact: Statistical information often incomplete
● Statistical moment – hierarchy of info
● 1: Mean – 'center'
● 2: Variance – 'variation'
● 1-1: Covariance – 'low order dependency'
● 3: Skew – 'asymmetry'
● 1-2,1-1-1: Skew – 'asymmetry of dependency'
● 4: Kurtosis – 'shape' [of distr]
● 1-2-1,1-1-1-1,...: …
● Not knowing is not knowing – not zero!!!
KAPERNICUS
Asym pdf
Sym pdf
Common...
But inconsistent!
Consistent statistical modeling

35. Representations of information
KAPERNICUS
● Parametric uncertainty
– Probability density / distribution functions
– Statistical moments (mean, std, cov, skew, kurt,..)
– Finite ensembles (approx.)
● Random (not reprod. - sampling variance)
● Deterministic
● Structural information
– Model equations
– Uncertainty difficult – alt. models?
Concept of representation – compare Fourier series...
Monte
Carlo

36. Uncertain models
KAPERNICUS
● Conventional
– Best estimate
– Associate uncertainty to best estimate
● Uncertain models:
– Best (aspects of interest) representation of
● Prior information
● Observations/measurement
– Not knowing [covariance] = not of interest, not zero!
● Propagation of uncertainty (Symmetry! => annealing)
– DUQ: Model => Prediction
– IUQ: Observation => Model

37. Identifiability of deterministic ensembles
● Global identifiability of certain model
● Identifiability of model ensemble
● is expressed in testable information and entails
identifiability of parameter ensemble
● which requires a given excitation matrix
KAPERNICUS

38. Optimal uncertain modeling
● “The more constraints – the worse model fit”
● Model represents known info
do not describe the 'truth'
● Conflict of assigned correlations:
– Uncorrelated measurement noise
– Correlated model output
=> Exceedingly low model uncertainty
=> Unwarranted confidence in prediction
● Only match covariance if known!
KAPERNICUS

39. Publications
Book chapters (freely downloadable from www.intechopen.com)
1. Deterministic sampling
Hessling J P, Deterministic sampling for Quantification of Modeling Uncertainty of
Signals in Digital Filters and Signal Processing, ISBN 978-953-51-0871-9 (INTECH,
2013)
2. Digital Filters and deterministic sampling
Hessling J P, Integration of digital filters and measurements in Digital Filters, ISBN
978-953-307-190-9 (INTECH, 2011)
3. Dynamic Metrology
Hessling J P, Metrology for non-stationary dynamic measurements in Advances in
Measurement systems, ISBN 978-953-307-061-2 (INTECH, 2010)
Journal articles, see author (Jan Peter Hessling) page on Google scholar
KAPERNICUS

40. KAPERNICUS
!

41. Summary
●
Deterministic sampling
– Uncertainty Quantification/Ident. of [complex] models
– Motivation, comparison to random sampling
– Applications
– Selection of methods and 'tricks'
– Foundations
● Dynamic Metrology
– Analysis of dynamic measurements
– Application of deterministic sampling
KAPERNICUS