Classical inference in/for physics

Classical Inference
in Physics
Thiago Mosqueiro
Institute of Physics of S˜ao Carlos
University of S˜ao Paulo
July 31 2012
Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 1 / 38

Equilibrium criticality – Ising model
• Model of ferromagnetism in statistical
mechanics
• Contact with thermal reservoir
• Lattice of N binary elements – spins
• Each site of this lattice: or
Energy of a conﬁguragion S = (S1, S2, . . .),
E(S) = −J
p,j
SjSp
Probability of S = (S1, S2, . . .),
P(S) =
exp − J
kT
p,j
SjSp
∀S
exp − J
kT
p,j
SjSp Monte Carlo step: each time step means N itera-
tions of the algorithm.

The Market for Lemons: Quality Uncertainty and the Market Mechanism, G. Akerlof.
The quarterly journal of economics, 1970

Objectivies
• What’s the diﬀerence between Probability and Statistics?
• Inference
• Statistics – models, hypotehsis and estimation
• Main focus: Maximum likelyhood estimators
• Glimpse of hypothesis testing
“What! you have solved it already?”
“Well, that would be too much to say.
I have discovered a suggestive fact, that is all. It is, however, very suggestive.”
Sign of Four (II), Sir Arthur Conan Doyle

Summary
1 Probability and random variables
2 Estimators
3 Maximum likelihood
4 Non-trivial example
5 One last example
6 Hypothesis testing
7 Conclusions

Where are we?
2 Estimators
5 One last example
7 Conclusions

Random variable
• Let |ψ be the state of a particle
• Suppose we know for every n the solutions H |n = n |n
What is the probability the particle is in state |n ?
• It can happen that |ψ = |1 . . . or |ψ = |2 . . . or . . .
• In this sense, |ψ is a random variable
• Suppose you have n dice Side j of each die have probability pj
• The result of tossing the j-th die is Xj
What is the probability of Xj > x?
• Xj is another random variable

Random variables
• Events:
X = x, X ≤ x, X = x and Y < y
• Probability of an event:
P (X = x), P (X ≤ x), P (X = x, Y < y)
• Moments:
X :=
x
P(X = x)x, X2
:=
x
P(X = x)x2
, . . .
• Surprisal:
I (X = x) = − log [P (X = x)]
• Entropy:
I (X = x) = −
x
P (X = x) log [P (X = x)]

Example: gaussian distributed variable
X ∼ N(µ, σ)
P(x < X ≤ x + dx) = ρ(x)dx = A exp −
(x − µ)2
σ2
dx
- 3 - 2 - 1
φμ,σ2(
0.8
0.6
0.4
0.2
0.0
−5 −3 1 3 5
x
1.0
−1 0 2 4−2−4
x)
0,μ=
0,μ=
0,μ=
−2,μ=
2
0.2,σ =
2
1.0,σ =
2
5.0,σ =
2
0.5,σ =

Example: gaussian distributed variable
X ∼ N(µ, σ)
P(x < X ≤ x + dx) = ρ(x)dx = A exp −
(x − µ)2
σ2
dx
0
1
2
3
4
5
6
0 100 200 300 400 500 600 700
EventXj
Realization j

Now the big problem...
What if I don’t have a model...?
How can one obtain information from observations?
How can I know my model ﬁts the reality?

Where are we?
2 Estimators
5 One last example
7 Conclusions

Statistically infering...
• Statistics validates and ﬁts Probabilistic models
• build a statistical model that should describe the process
• interpret the data as realizations of your model
• Inference gives you a statistical proposition
• Models may be parametric, non-parametric or semi-parametric
• Of course let’s focus on parametric models.
Let’s guess our model:
Xj ∼ N(µ, σ)
What’s your best guess
about µ and σ?
Usual way of doing this estimate is
by means of an estimator 0
1
2
3
4
5
6
0 100 200 300 400 500 600 700
EventXj
Realization j

Estimator
• Data: set of i.i.d. X1, X2, X3, . . . Xm
• Given a data set and a statistical model, we have a probability with
some parameter θ
• An estimator is a function of the data set to some sample estimates
Examples of estimators
• X =
1
m
m
j=1
Xj
• ˆσ =
1
m
m
j=1
Xj − X
2
• ˆxmin = min (X1, X2, X3, . . . Xm)

Mean Square Error
• EQM ˆθ := ˆθ − θ
2
• EQM ˆθ = ˆθ2
− 2θˆθ + θ2
= ˆθ2
− 2θ ˆθ + θ2
= ˆθ2
− ˆθ
2
+ ˆθ
2
− 2θ ˆθ + θ2
EQM ˆθ = Var ˆθ − B2
(ˆθ)
• Bias: B(ˆθ) = ˆθ − θ
• Non-biased estimator: B(ˆθ) = 0 ⇐⇒ ˆθ = θ

Example to gaussian mean and variance
• Random sample: X1, X2, X3, . . . , Xm
• Starting with the mean value: ˆµ = X :=
1
m
m
j=1
Xj
• ˆµ =
1
m
m
j=1
Xj =
1
m
m
j=1
Xj =
1
m
m
j=1
µ = µ
Thus, X is a non-biased estimator for µ
• Now let’s take a look at the
• EQM [ˆµ] = Var X = Var
1
m
m
j=1
Xj =
1
m2
m
j=1
Var [Xj] =
1
m2
m
j=1
σ2
=
σ2
m
Moreover, X → µ when m → ∞

Example to gaussian mean and variance
• Let’s now do it with the variance: ˆσ2
b =
1
m
m
j=1
Xj − X
2
• ˆσ2
b =
1
m
m
j=1
Xj − X
2
=
1
m
m
j=1
Xj − X
2
=
m − 1
m
σ2
Thus, ˆσb is asymptotically non-biased estimator for σ
• Conversely, let’s deﬁne ˆσ2
=
1
m − 1
m
j=1
Xj − X
2
• EQM [ˆµ] = σ2
ˆσ is an unbiased estimator for σ

Back to Gaussian random variables
0
1
2
3
4
5
6
0 100 200 300 400 500 600 700
EventXj
Realization j
• Estimation: ˆσb = 0.49944735873
• Actual value used to generate the data: ˆσ = 0.49980448952

Where are we?
2 Estimators
5 One last example
7 Conclusions

Maximum Likelihood
• Maximize the ”likelihood“: get an estimator for te parameter of a
given statistical model
• p(x1, x2, x3, . . . |θ) :: given the value θ for the parameter,
this is the probability that X1 = x1, X2 = x2, . . .
• p(x1, x2, x3, . . . |θ) = p(x1|θ)p(x2|θ)p(x3|θ) . . . =
m
j=1
p(xj|θ)
• Let x = x1, x2, x3, . . .
L(θ, x) =
m
j=1
p(xj|θ)
• To maximize it, we can use ln (L(θ, x)) =
m
j=1
p(xj|θ)
•
∂
∂θ
ln (L(θ, x)) = 0 – solve it for θ

Example for gaussian
• Suppose Xj ∼ N(µ, σ) is our statistical model
• L(ˆµ, x) =
1
√
2πσ2
m
exp
1
2σ2
m
j=1
(xj − ˆµ)2
• By calculating
∂
∂ˆµ
ln (L(ˆµ, x)) = 0, we get to
m
j=1
(xj − ˆµ) =
m
j=1
xj − mˆµ = 0
• Finally, we get ˆµ =
1
m
m
j=1
xj
• On the other hand,
∂
∂ˆσ
ln (L(ˆσ, x)) = 0 gives ........
• This is what is usually done to derive such an estimator

Example for gaussian
• Suppose Xj ∼ N(µ, σ) is our statistical model
• L(ˆµ, x) =
1
√
2πσ2
m
exp
1
2σ2
m
j=1
(xj − ˆµ)2
• By calculating
∂
∂ˆµ
ln (L(ˆµ, x)) = 0, we get to
m
j=1
(xj − ˆµ) =
m
j=1
xj − mˆµ = 0
• Finally, we get ˆµ =
1
m
m
j=1
xj
• On the other hand,
∂
∂ˆσ
ln (L(ˆσ, x)) = 0 gives ˆσ =
1
m
m
j=1
Xj − X
2
,
which we have already discovered to be biased!
• This is what is usually done to derive such an estimator

Deutsch Tank Problem
• Suppose you have a box with (unkown) n tickets, labled from 1 to n.
• You take one ticket, it’s label is x. What’s your best guess for n?

Estimation of the maximum
• Statistical model: p(x|ˆn) =
1
ˆn
, x ≤ ˆn – a uniform distribution
• Remember p(x|ˆn) = 0, x > ˆn. This is important to this maximization.
• L(ˆn, x) =
1
ˆn
, which is maximized when ˆn is the largest!
• Therefore, our best guess at this moment would be ˆn = x.
• If we had conversely made several observations X1, X2, X3, . . ., then
ˆn = max (X1, X2, X3, . . .)
• ”However, this is awful“ – it is a poor estimation!
• Other methods are far more accurate and, indeed, were succesfully used in WWII.

Where are we?
2 Estimators
5 One last example
7 Conclusions

What is this?
Large values signiﬁcant?
Is the mean informative? Probably not.
0
10
20
30
40
50
60
70
0 20 40 60 80 100 120 140 160
Xj
Realization j

Empirical PDF
• We don’t know p(x)∆x, but we have a lot data...
• In this experiment, I will use 107
points.
• We can then calculate the following estimator:
ˆp(x) =
1
m
m
j=1
δ (Xj ∈ [x, x + ∆x])
10-8
10-7
10-6
10-5
10-4
10-3
10-2
10-1
100
100
101
102
103
104
P(X=x)
Event X = x
In fact, we can show that
ˆp(x) is an unbiased
estimator for p(x).
Let’s propose then a model:
p(x) ∼ x−α
But... α =?

Empirical CDF
If you want to ﬁt, use the CDF
10-7
10-6
10-5
10-4
10-3
10-2
10-1
100
100
101
102
103
104
105
P(X>=x)
Event X = x
However, how about a maximum likelihood estimator?

Exponent estimator
• General power-law distribution: p(x)dx =
α − 1
xmin
x
xmin
−α
dx
• We have to maximize the likelihood: L (ˆα|X1, . . . Xm) =
m
j=1
ˆα − 1
xmin
Xj
xmin
−ˆα
• ln [L (α|X1, . . . Xm)] =
m
j=1
ln(α − 1) − ln(xmin) − α ln
Xj
xmin
• Maximizing it...
ˆα = 1 +
m
m
j=1
Xj/xmin

PDF
In our case, ˆα ∼ 2.65
10-8
10-7
10-6
10-5
10-4
10-3
10-2
10-1
100
100
101
102
103
104
P(X=x)
Event X = x
Testing data
Fitting curve

Where are we?
2 Estimators
5 One last example
7 Conclusions

What about this...
-250
-200
-150
-100
-50
0
50
100
150
0 50 100 150 200 250 300
EventXj
Realization j
Let’s start by the empirical pdf!

Gaussian!!! :)
Again, we can propose a model: A Gaussian!
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
-4 -2 0 2 4
P(X=x)
Event X = x
Estimating the variance: ˆσ ≈ 200

Gaussian!!! :(
Again, we can propose a model: A Gaussian!
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
-4 -2 0 2 4
P(X=x)
Event X = x
Empirical PDF
Fitting
Estimating the variance: ˆσ ≈ 200 – Something is not right!

Okay, not a gaussian
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
-10 -5 0 5 10
P(X=x)
Event X = x

Okay, not a gaussian
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
-20 -15 -10 -5 0 5 10 15 20
P(X=x)
Event X = x

Power-law decay – Cauchy distributions
1e-06
1e-05
0.0001
0.001
0.01
0.1
1
1 10 100 1000 10000 100000
P(X=x)
Event X = x
Empirical CDF
Very suggestive indeed.

Where are we?
2 Estimators
5 One last example
7 Conclusions

Hypothesis testing
• What we have been doing till now is ∼ exploratory analysis
• We now want to confirm a prediction or hypothesis – confirmatory analysis
• Is this last data set gaussian or cauchy distributed?
• The general recipe:
- An initial guess, possibly true
- State an relevant null and its alternative hypothesis
- Formulate an appropriate test – T and a significance level τ
- Estimate the distribution of T under your null hypothesis
- Compute the observed quantity t and verify your null hypothesis

Let’s try this shit!
• H0: the data is normally distributed – N(0, σ)
• We have the estimated pdf of the data sample – these are our
m observations Oj
• The test will be
T =
m
j=1
(Oj − Ej)2
Ej
• Ej = A exp −
(x)2
σ2
are the expected frequencies!
• It is easy to derive that T ∼ χ2
m−1 (Bolfarine)
• In our case, the observed t ≈ 400 and P (T = t) → 0.
• This rejects H0.

Suggestions
• Other hypothesis testing techniques:
* τ-Student test
* minimax
* Lagrange multiplier
* Union-intersection
* Fisher test
* ...
• Non-parametric testins, such as Kolmogorov-Smirnov

Where are we?
2 Estimators
5 One last example
7 Conclusions

Classical inference in/for physics

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (15)

Ähnlich wie Classical inference in/for physics

Ähnlich wie Classical inference in/for physics (20)

Mehr von Thiago Mosqueiro

Mehr von Thiago Mosqueiro (12)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Classical inference in/for physics