1. Mahalanobis Kernel based on Probabilistic
Principal Component
M. Fauvel1 , A. Villa2,3 , J. Chanussot2 and J. A. Benediktsson3
1
DYNAFOR, INRA & ENSAT, INPT, Université de Toulouse - France
2
GIPSA-Lab, Grenoble Institute of Technology - France
3
University of Iceland, Reykjavik - Iceland
2011 IEEE International Geoscience and Remote Sensing Symposium
24-29 July, Vancouver, Canada

2. Hyperspectral images
Fully regularized Mahalanobis kernel
Experimental results
Conclusions and perspectives

3. Hyperspectral images
Fully regularized Mahalanobis kernel
Experimental results
Conclusions and perspectives

4. Hyperspectral images
High number of spectral measurements but
limited number of training samples.
xi ∈ Rd with d 10
i ∈ {1, . . . , n} and n ≈ d
√
Quality and quantity of information
× Curse of dimensionality:
→ Statistical
→ Geometrical
→ Computational
ns = 208010, n = 3921,
d = 103, nc ≈ 430

5. Properties of HD space
Statistical properties :
High number of parameters
Slow rate of convergence
Conventional Gaussian models fail
Geometrical properties :
Empty space
Minkowski norms ineffective
Local kernel methods fail
Conventionally: dimension reduction or regularization

6. Properties of HD space
Statistical properties :
High number of parameters
Slow rate of convergence
Conventional Gaussian models fail
Geometrical properties :
Empty space
Minkowski norms ineffective
Local kernel methods fail
Conventionally: dimension reduction or regularization
In this work parsimonious model + kernel method

7. Proposed approach
Probabilistic PCA and kernel methods
Use emptiness property to construct the kernel
How:
Mahalanobis distance for class c:
dΣc (x, z) = (x − z)t Σ−1 (x − z)
c
Gaussian Radial kernel:
d(x, z)2
kg (x, z) = exp −
2σ 2
Mahalanobis kernel:
(x − z)t Σ−1 (x − z)
c
km (x, z|c) = exp −
2σ 2

8. Hyperspectral images
Fully regularized Mahalanobis kernel
Experimental results
Conclusions and perspectives

9. The PPCA model 1/3
PPCA: parsimonious probabilistic model [Tipping and Bishop, 1999]
Cluster assumption: each class c lives in a specific subspace
Covariance matrix of class c:
d
Σc = Q c Λ c Q t =
c
t
λci qci qci
i=1
PPCA: diag(Λc ) = (λc1 + bc ) . . . (λcpc + bc ) bc . . . . . . bc with pc d
pc d−pc
Covariance matrix of class c under PPCA:
pc d
t t
Σc = (λci + bc )qci qci + bc qci qci
i=1 i=pc +1
Ac ¯
Ac
¯
Ac is the signal subspace and Ac is the noise subspace (Rd = Ac ¯
Ac )

10. The PPCA model 2/3
In R3 :
z z
¯
A
e3
x−z Qn x
A
e2
x−z Qs
e1
The inverse can be computed explicitly:
pc d
1 1
Σ−1 =
c
t
qci qci + t
qci qci
i=1
λci + bc bc i=pc +1
d t
Using I = i=1 qci qci ,
pc
−λci 1
Σ−1 =
c
t
qci qci + I
i=1
(λci + bc )bc bc

11. The PPCA model 3/3
So what?
Less parameters have to be estimated (d = 100 and pc = 10)
Full Σ: d(d + 3)/2 parameters → 5150
PPCA: d(pc + 1) + 2 − pc (pc − 1)/2 parameters → 1057
Better than PCA
x and z may be artificially close in Ac
An accurate estimation of pc is necessary
Estimation: from the sample covariance matrix
nc
ˆ 1 t
Σc = xi − xc xi − xc , xi ∈ c
¯ ¯
nc i=1
pc
ˆ
λci ˆ
are estimated by the first pc eigenvalues of Σc
i=1
pc
ˆ
qci ˆ
are estimated by the first pc eigenvectors of Σc
i=1
ˆc is estimated by trace(Σc ) −
ˆ ˆc
p ˆ
b i=1
λci /(d − pc )
ˆ
pc is estimated with the BIC
ˆ
max {BIC(pc ) = −2l + (d(pc + 1) + 2 − pc (pc − 1)/2) log(nc )}
pc

12. Mahalanobis kernel 1/2
ˆ pc pc +1
λci i=1
and ˆc are used to initialize the kernel hyperparameters σi
b i=1
The kernel:
pc
ˆ
1 (x − z)t qci qci (x − z)
ˆ ˆt x−z 2
km (x, z|c) = exp − 2 + 2
2 i=1
σi σpc +1
ˆ
Another formulation: product of Gaussian kernels
pc
ˆ
km (x, z|c) = kg (x, z) × qt ˆt
kg (ˆ ci x, qci z)
i=1
The Mahalanobis kernel builds with the PPCA model is a mixture of a
Gaussian kernel on the original data and a Gaussian kernel on the pc first
principal components

13. Mahalanobis kernel 2/2
km (0, x|c) with 0 = [0, 0] and x ∈ [−1, 1]2
1 1 1
0.9
0.9 0.8
0.8
0.5 0.5 0.5
0.8 0.7
0.6
0 0 0 0.6
0.7
0.4 0.5
−0.5 0.6 −0.5 −0.5 0.4
0.2
0.3
−1 0.5 −1 −1
−1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1
(a) (b) (c)
Σc = [0.6 − 0.2; −0.2 0.6] and pc = 1
Red contour line → km = 0.75
(a): Gaussian kernel
2 2
(b): Mahalanobis kernel with σ1 = σ2 = 0.5
2 2
(c): Mahalanobis kernel with σ1 = 1.5 and σ2 = 0.5

14. Hyperspectral images
Fully regularized Mahalanobis kernel
Experimental results
Conclusions and perspectives

15. Learning framework
{(xi , yi )|yi = c}nc
i=1 f (x|c)
PPCA Estimation SVM
Gradient Step
L2-SVM and radius-margin bound optimization
Multiclass: one classifier per class (but SVMci vs cj = SVMcj vs ci )

16. Simulated data
Linear mixture of Gaussian following PPCA model
c
x= αi si + b, y = j such as αj = max αi and si ∼ PPCA
i
i=1
d = 207, p = 10, c = 2, n = 1000 and SNR = 1
Mean values were extracted from spectral library
50 tries
0.98
0.97
0.96
0.95
0.94
0.93
0.92
1 2 3
Gaussian PCA PPCA

17. Hyperspectral image 1/2
ROSIS-03, University of Pavia
d = 103, n = 3921 and c = 9
BIC estimation:
c 1 2 3 4 5 6 7 8 9
p
ˆ 20 21 14 26 13 17 12 19 14
Classification accuracies:
Kernel Gaussian Mahalanobis PCA-Mahalanobis PPCA-Mahalanobis
σ s m m m
Asphalt (548, 6631) 94,8 93,2 96,0 96,0
Meadow (540, 18649) 79,4 73,2 82,0 80,6
Gravel (392, 2099) 97,2 95,9 97,7 97,2
Tree (524, 3064) 94,3 97,9 98,3 98,1
Metal Sheet (265, 1345) 99,8 99,9 99,9 99,9
Bare Soil (532, 5029) 87,8 92,7 91,4 90,7
Bitumen (375, 1330) 98,8 98,7 98,7 99,0
Brick (514, 3682) 96,7 95,6 97,5 97,2
Shadow (231, 947) 99,9 98,7 99,9 99,9
Average class accuracy 94,3 94,0 95,7 95,4

18. Hyperspectral image 2/2

19. ˆ
Influence of the parameter pc
Precision of the estimation (on the simulated data):
Estimation of pc for x : 21
ˆ
Estimation of pc for si : 10
ˆ
OA vs pc (hyperspectral image, class Meadow):
ˆ
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0 5 10 15 20 25 30

20. Hyperspectral images
Fully regularized Mahalanobis kernel
Experimental results
Conclusions and perspectives

21. Conclusions
Classification of hyperspectral image
A Mahalanobis kernel based on PPCA was proposed:
Cluster assumption
Multiple hyperparameters
Link with mixture kernels
SVM Classification framework
Good classification results on two data sets
Better than the conventional RBF
As good as PCA + RBF

22. Perspectives
Implementation: Optimization of the hyperparameters
Estimation of pc
ˆ
Construction of other kernels:
p
k(x, z) = xt Σ−1 z + 1

23. Mahalanobis Kernel based on Probabilistic
Principal Component
M. Fauvel1 , A. Villa2,3 , J. Chanussot2 and J. A. Benediktsson3
1
DYNAFOR, INRA & ENSAT, INPT, Université de Toulouse - France
2
GIPSA-Lab, Grenoble Institute of Technology - France
3
University of Iceland, Reykjavik - Iceland
2011 IEEE International Geoscience and Remote Sensing Symposium
24-29 July, Vancouver, Canada