In this presentation, we see some topics about Least Squares method.

1. Introduction to Least Squares
minimization method
Kamal Aghazade

2. How can you describe your problem(s)?
2

3. Now suppose we observed data like
the inverse problem in the concept of Bayesian theory
is to find the most probable model vector that produce
d
m d
ˆWe need to find a model like :
ˆ arg max P( | , ) :some parameters like variance
m
m
m m d  
1 2
ˆ argmin { ln P( | , ) ln P( , )}
Cost Function
 
  
m
m m d m
1 44 2 4 43 1 4 2 4 3
1 4 4 4 44 2 4 4 4 4 43
 
3

4. 2
/2 2
1
In the concept of Linear problem we have:
:
1 1
( | , ) exp [ ]
(2 ) 2
N
iN N
i
We have
P 
   
 
 
   
 

e d Gm
d m d Gm
1 2
ˆ argmin { ln P( | , ) ln P( , )}
Cost Function
 
  
m
m m d m
1 44 2 4 43 1 4 2 4 3
1 4 4 4 44 2 4 4 4 4 43
 
4

5. 1
2
2 2
2
2
In the concept of Linear problem we have:
:
ln ( | , )
1
2
ˆ arg minLS
We have
P
const


 
  
  
 
e d Gm
d m
d Gm
m d Gm Least Squares Optimization
5

6. Reference:
https://www.encyclopediaofmath.org/index.php/Norm
6

7. Least Squares
• Least Squares go back to Gauss and Legendre in the late
1970s.
We have two different interpretations of Least Squares
 Best Approximation Estimate (BAE) in terms of Quadratic norms
 Maximum Likelihood Estimate (MLE) in terms of distributions
Least Squares is applicable to both under-determined and over-determined problems
7

8. Least Squares Solution to Linear Problem
8

9. 1 1
1) (donot usedirectinverseapprocah for )
2) rewrite the problem as ( is the error)
thismeans that there may not exist one value that would satisfy
one obvious strategy is to m
M N
M N
 

  
   

Ax f
x A f A
Ax f e e
x Ax f
   
1/21/2 2 2 2 22 2 2 2
1 2 2 1 2 32
inimize so norm of
In practical terms this norm of can be its Euclidean length:
... ...M Me e e e e e e e         
e
e
e
1/
1
In general form for p-norms we have:
| |
pM
p
ip
i
e

 
  
 
e
9

10. 1 1
3)
singular
ˆ( ) ( )
T
T
T T T
N M
assume NO rank deficiency and is non
 
  

  
x A y
AA
y AA f x A AA f
10

11. Geometric interpretation of LS
• In the case of LS approximation we are minimizing the Euclidean
distance.
 Ax f eˆ
ˆ :best Least Squares approximation
:
ˆ is orthogonal to column space of matrix
it must be orthogonal to each column in
ˆ ˆ( ) 0 ( )T T T
f X e
X
e orthogonal error
f X
f X X f
  


 
   
 

   
A
A
A A
A
A A A A A
11

12. Least Squares in practice
•
1
1/ 1/2
2
1 1
1/
1
Lp-norms
1 | |
| | 2 | |
| | max
M
i
i
pM M
p
i ip
i i
M
i i
ii
p e
e p e
P e e

 




  
   
 
   
     
   

         

 

e
e
12

13. 1/
1
| |
pM
p
ip
i
e

 
 
 
e
What is the response
of each norm?
Look at
this
13

14. 2L norm
Implies that data obey Gaussian Statistics.
The choice of norm depends on the particular type
of Statistics of the observations.
1
2
many scattered data Long-tailed distribution L norm is the proper choice
few scattered data Short-tailed distribution L norm is the proper choice
 
 
14

15. Before start to use LS, check the distribution of the data set.
Normal distribution investigation, which direction(s)?
1
2
2 2
2
2
In the concept of Linear problem we have:
:
ln ( | , )
1
2
ˆ arg minLS
We have
P
const


 
  
  
 
e d Gm
d m
d Gm
m d Gm
Just a friendly
reminder 
15

16. Methods for checking normality of the data:
 Histogram
 Q-Q plot
 Kolmogorov-Smirnov test
 Anderson-Darling test
 Lilliefors test
YES we have Normal
distribution.
Go on with LS solution 
16

17. Least Squares method is subject to some issues!
• Set Some constraint on the model (e.g. BVLS).
• Set Regularization to the LS problem.
• Set priory information to the LS problem.
• LS goal is to find a model that best fit the data.
• LS solution is subject to ill-conditioning.
• Evaluation of the solution depends on other information.
solution
17

18. Least Squares applications
 Model fitting
 Physics
 Control
 Estimation
 Statistics
 Image processing
Geophysics
 Medicine
18

19.  Seismic tomography
 Full waveform inversion
 Radon transform
 Least Squares Migration
 model based inversion
 deconvolution
Least Squares applications in Seismic
19

20. Have a normal
distributed day
--