Spectral methods are used in computer graphics, machine learning, and computer vision, where many important problems boil down to constructing a Laplacian operator and finding its eigenvalues and eigenfunctions. We show how to generalize spectral geometry to multiple data spaces. Our construction is based on the idea of simultaneous diagonalization of Laplacian operators. We describe this problem and discuss numerical methods for its solution. We provide several synthetic and real examples of manifold learning, object classification, and clustering, showing that the joint spectral geometry better captures the inherent structure of multi-modal data.
Talk at SIAM-IS 2014 (http://www.math.hkbu.edu.hk/SIAM-IS14/). A big thanks to Michael Bronstein for providing a great set of slides this presentation is a mere extension of.
Feature-aligned N-BEATS with Sinkhorn divergence (ICLR '24)
Building Compatible Bases on Graphs, Images, and Manifolds
1. Building Compatible Bases on Graphs,
Images, and Manifolds
Davide Eynard
Institute of Computational Science, Faculty of Informatics
University of Lugano, Switzerland
SIAM-IS, 14 May 2014
Based on joint works with Artiom Kovnatsky, Michael M. Bronstein,
Klaus Glashoff, and Alexander M. Bronstein
1 / 85
16. Spectral geometry
Laplacian eigenmap: m-dimensional embedding of X using the
first eigenvectors of the Laplacian
U = (φ1, . . . , φm)
Belkin, Niyogi 2001
16 / 85
17. Heat equation
Heat diffusion on X is governed
by the heat equation
Lf(t) +
∂
∂t
f(t) = 0, f(0) = u,
where f(t) is the amount of heat
at time t
17 / 85
18. Heat equation
Heat diffusion on X is governed
by the heat equation
Lf(t) +
∂
∂t
f(t) = 0, f(0) = u,
where f(t) is the amount of heat
at time t
18 / 85
19. Heat equation
Heat diffusion on X is governed
by the heat equation
Lf(t) +
∂
∂t
f(t) = 0, f(0) = u,
where f(t) is the amount of heat
at time t
Heat operator (or heat kernel)
Ht
= e−tL
= Φe−tΛ
ΦT
provides the solution of the heat equation f(t) = Ht
f(0)
19 / 85
20. Heat equation
Heat diffusion on X is governed
by the heat equation
Lf(t) +
∂
∂t
f(t) = 0, f(0) = u,
where f(t) is the amount of heat
at time t
Heat operator (or heat kernel)
Ht
= e−tL
= Φe−tΛ
ΦT
provides the solution of the heat equation f(t) = Ht
f(0)
‘How much heat is transferred from point xi to point xj in time t’
20 / 85
21. Spectral geometry
Diffusion map: m-dimensional embedding of X using the heat
kernel
U = (e−tλ1
φ1, . . . , e−tλm
φm)
B´erard et al. 1994; Coifman, Lafon 2006
21 / 85
22. Spectral geometry
Diffusion distance: crosstalk between heat kernels
d2
t (xp, xq) =
n
i=1
((Ht
)pi − (Ht
)qi)2
B´erard et al. 1994; Coifman, Lafon 2006
22 / 85
23. Spectral geometry
Diffusion distance: crosstalk between heat kernels
d2
t (xp, xq) =
n
i=1
((Ht
)pi − (Ht
)qi)2
=
n
i=1
e−2tλi
(φpi − φqi)2
B´erard et al. 1994; Coifman, Lafon 2006
23 / 85
24. Spectral geometry
Diffusion distance: Euclidean distance in the diffusion map
space
dt(xp, xq) = Up − Uq 2
B´erard et al. 1994; Coifman, Lafon 2006
24 / 85
25. Spectral geometry
Diffusion distance: Euclidean distance in the diffusion map
space
dt(xp, xq) = Up − Uq 2
B´erard et al. 1994; Coifman, Lafon 2006
25 / 85
31. Diagonalization of the Laplacian
Eigendecomposition can be posed as the minimization problem
min
ΦT
Φ=I
off(ΦT
LΦ)
with off-diagonality penalty off(X) = i=j x2
ij.
31 / 85
32. Diagonalization of the Laplacian
Eigendecomposition can be posed as the minimization problem
min
ΦT
Φ=I
off(ΦT
LΦ)
with off-diagonality penalty off(X) = i=j x2
ij.
Jacobi iteration: compose Φ = · · · R3R2R1 as a sequence of
Givens rotations, where each new rotation tries to reduce the
off-diagonal terms
Jacobi 1846
32 / 85
33. Diagonalization of the Laplacian
Eigendecomposition can be posed as the minimization problem
min
ΦT
Φ=I
off(ΦT
LΦ)
with off-diagonality penalty off(X) = i=j x2
ij.
Jacobi iteration: compose Φ = · · · R3R2R1 as a sequence of
Givens rotations, where each new rotation tries to reduce the
off-diagonal terms
Analytic expression for optimal rotation for given pivot
Jacobi 1846
33 / 85
34. Diagonalization of the Laplacian
Eigendecomposition can be posed as the minimization problem
min
ΦT
Φ=I
off(ΦT
LΦ)
with off-diagonality penalty off(X) = i=j x2
ij.
Jacobi iteration: compose Φ = · · · R3R2R1 as a sequence of
Givens rotations, where each new rotation tries to reduce the
off-diagonal terms
Analytic expression for optimal rotation for given pivot
Rotation applied in place – no matrix multiplication
Jacobi 1846
34 / 85
35. Diagonalization of the Laplacian
Eigendecomposition can be posed as the minimization problem
min
ΦT
Φ=I
off(ΦT
LΦ)
with off-diagonality penalty off(X) = i=j x2
ij.
Jacobi iteration: compose Φ = · · · R3R2R1 as a sequence of
Givens rotations, where each new rotation tries to reduce the
off-diagonal terms
Analytic expression for optimal rotation for given pivot
Rotation applied in place – no matrix multiplication
Guaranteed decrease of the off-diagonal terms
Jacobi 1846
35 / 85
36. Diagonalization of the Laplacian
Eigendecomposition can be posed as the minimization problem
min
ΦT
Φ=I
off(ΦT
LΦ)
with off-diagonality penalty off(X) = i=j x2
ij.
Jacobi iteration: compose Φ = · · · R3R2R1 as a sequence of
Givens rotations, where each new rotation tries to reduce the
off-diagonal terms
Analytic expression for optimal rotation for given pivot
Rotation applied in place – no matrix multiplication
Guaranteed decrease of the off-diagonal terms
Orthonormality guaranteed by construction
Jacobi 1846
36 / 85
37. Joint approximate diagonalization
Laplacians of X and Y are diagonalized independently:
min
ΦT
Φ=I,ΨT
Ψ=I
off(ΦT
LXΦ) + off(ΨT
LY Ψ)
φ2 φ3 φ4 φ5
ψ2 ψ3 ψ4 ψ5
Cardoso 1995; Eynard, Bronstein2
, Glashoff 2012
37 / 85
38. Joint approximate diagonalization
Diagonalize Laplacians of X and Y simultaneously:
min
ˆΦ
T
ˆΦ=I
off( ˆΦ
T
LX
ˆΦ) + off( ˆΦ
T
LY
ˆΦ)
ˆφ2
ˆφ3
ˆφ4
ˆφ5
ˆφ2
ˆφ3
ˆφ4
ˆφ5
Cardoso 1995; Eynard, Bronstein2
, Glashoff 2012
38 / 85
39. Joint approximate diagonalization
Diagonalize Laplacians of X and Y simultaneously:
min
ˆΦ
T
ˆΦ=I
off( ˆΦ
T
LX
ˆΦ) + off( ˆΦ
T
LY
ˆΦ)
In most cases, ˆΦ is only an approximate eigenbasis
Cardoso 1995; Eynard, Bronstein2
, Glashoff 2012
39 / 85
40. Joint approximate diagonalization
Diagonalize Laplacians of X and Y simultaneously:
min
ˆΦ
T
ˆΦ=I
off( ˆΦ
T
LX
ˆΦ) + off( ˆΦ
T
LY
ˆΦ)
In most cases, ˆΦ is only an approximate eigenbasis
Modified Jacobi iteration (JADE): compose ˆΦ = · · · R3R2R1
as a sequence of Givens rotations, where each new rotation
tries to reduce the off-diagonal terms
Cardoso 1995; Eynard, Bronstein2
, Glashoff 2012
40 / 85
41. Joint approximate diagonalization
Diagonalize Laplacians of X and Y simultaneously:
min
ˆΦ
T
ˆΦ=I
off( ˆΦ
T
LX
ˆΦ) + off( ˆΦ
T
LY
ˆΦ)
In most cases, ˆΦ is only an approximate eigenbasis
Modified Jacobi iteration (JADE): compose ˆΦ = · · · R3R2R1
as a sequence of Givens rotations, where each new rotation
tries to reduce the off-diagonal terms
Overall complexity akin to the standard Jacobi iteration
Cardoso 1995; Eynard, Bronstein2
, Glashoff 2012
41 / 85
48. Drawbacks of JADE
In many applications, we do not need the whole basis, just the
first k n eigenvectors
48 / 85
49. Drawbacks of JADE
In many applications, we do not need the whole basis, just the
first k n eigenvectors
Explicit assumption of orthonormality of the joint basis
restricts Laplacian discretization to symmetric matrices only
49 / 85
50. Drawbacks of JADE
In many applications, we do not need the whole basis, just the
first k n eigenvectors
Explicit assumption of orthonormality of the joint basis
restricts Laplacian discretization to symmetric matrices only
Requires bijective known correspondence between X and Y
50 / 85
53. Partial correspondence
Chili, food
San Francisco,
USA
Landrover, SUV
Cayenne, city
Cayenne, Porsche
Cayenne, pepper
Marijuana,
cannabis
Alligator
Crocodile
Bear
Apple
MacBook
Orange
Image space Tag space
53 / 85
55. Partial correspondence
Two discrete manifolds with different number of vertices,
X = {x1, . . . , xn} and Y = {x1, . . . , xm}
Laplacians LX of size n × n and LY of size m × m
55 / 85
56. Partial correspondence
Two discrete manifolds with different number of vertices,
X = {x1, . . . , xn} and Y = {x1, . . . , xm}
Laplacians LX of size n × n and LY of size m × m
Set of corresponding functions F = (f1, . . . , fq) and
G = (g1, . . . , gq)
56 / 85
57. Partial correspondence
Two discrete manifolds with different number of vertices,
X = {x1, . . . , xn} and Y = {x1, . . . , xm}
Laplacians LX of size n × n and LY of size m × m
Set of corresponding functions F = (f1, . . . , fq) and
G = (g1, . . . , gq)
We cannot find a common eigenbasis ˆΦ of Laplacians LX
and LY , because they now have different dimensions
57 / 85
58. Coupled diagonalization
Find two sets of coupled approximate eigenvectors ˆΦ, ˆΨ
min
ˆΦ, ˆΨ
off( ˆΦ
T
LX
ˆΦ) + off( ˆΨ
T
LY
ˆΨ) + µ FT ˆΦ − GT ˆΨ 2
F
s.t. ˆΦ
T
ˆΦ = I, ˆΨ
T
ˆΨ = I
Kovnatsky, Bronstein2
, Glashoff, Kimmel 2013
58 / 85
59. Perturbation of joint eigenbasis
Theorem (Cardoso 1994) Let A = ΦΛΦT
be a symmetric
matrix with simple δ-separated spectrum (|λi − λj| ≥ δ) and
B = ΦΛΦT
+ E. Then, the joint approximate eigenvectors of
A, B satisfy
ˆφi = φi +
j=i
αijφj + O( 2
)
where αij = φT
i Eφj/2(λj − λi) ≤ E 2/2δ
Cardoso 1994
59 / 85
60. Perturbation of joint eigenbasis
Theorem (Cardoso 1994) Let A = ΦΛΦT
be a symmetric
matrix with simple δ-separated spectrum (|λi − λj| ≥ δ) and
B = ΦΛΦT
+ E. Then, the joint approximate eigenvectors of
A, B satisfy
ˆφi = φi +
j=i
αijφj + O( 2
)
where αij = φT
i Eφj/2(λj − λi) ≤ E 2/2δ
Consequently, span{ˆφ1, . . . , ˆφk} ≈ span{φ1, . . . , φk}
Cardoso 1994; Kovnatsky, Bronstein2
, Glashoff, Kimmel 2013
60 / 85
61. Perturbation of joint eigenbasis
Theorem (Cardoso 1994) Let A = ΦΛΦT
be a symmetric
matrix with simple δ-separated spectrum (|λi − λj| ≥ δ) and
B = ΦΛΦT
+ E. Then, the joint approximate eigenvectors of
A, B satisfy
ˆφi = φi +
j=i
αijφj + O( 2
)
where αij = φT
i Eφj/2(λj − λi) ≤ E 2/2δ
Consequently, span{ˆφ1, . . . , ˆφk} ≈ span{φ1, . . . , φk} i.e., k first
approximate joint eigenvectors can be expressed as linear
combinations of k ≥ k eigenvectors: ˆΦ ≈ ¯ΦS, ˆΨ ≈ ¯ΨR, where
¯Φ = (φ1, . . . , φk ), ¯ΛX = diag(λX
1 , . . . , λX
k )
¯Ψ = (ψ1, . . . , ψk ), ¯ΛY = diag(λY
1 , . . . , λY
k )
Cardoso 1994; Kovnatsky, Bronstein2
, Glashoff, Kimmel 2013
61 / 85
62. Subspace coupled diagonalization
Find two sets of coupled approximate eigenvectors ˆΦ, ˆΨ
min
ˆΦ, ˆΨ
off( ˆΦ
T
LX
ˆΦ) + off( ˆΨ
T
LY
ˆΨ) + µ FT ˆΦ − GT ˆΨ 2
F
s.t. ˆΦ
T
ˆΦ = I, ˆΨ
T
ˆΨ = I
Coupling: given a set of corresponding vectors F, G, make
their Fourier coefficients coincide ˆΦ
T
F ≈ ˆΨ
T
G
Kovnatsky, Bronstein2
, Glashoff, Kimmel 2013
62 / 85
63. Subspace coupled diagonalization
Find two sets of coupled approximate eigenvectors ˆΦ, ˆΨ
min
R,S
off(RT ¯ΦT
LX
¯ΦR) + off(ST ¯ΨT
LY
¯ΨS) + µ FT ¯ΦR − GT ¯ΨS 2
F
s.t. RT ¯ΦT ¯ΦR = I, ST ¯ΨT ¯ΨS = I
Coupling: given a set of corresponding vectors F, G, make
their Fourier coefficients coincide ˆΦ
T
F ≈ ˆΨ
T
G
Based on perturbation Theorem, express the joint approximate
eigenbases as linear combinations ˆΦ = ¯ΦS, ˆΨ = ¯ΨR
Kovnatsky, Bronstein2
, Glashoff, Kimmel 2013; Cardoso 1994
63 / 85
64. Subspace coupled diagonalization
Find two sets of coupled approximate eigenvectors ˆΦ, ˆΨ
min
R,S
off(RT ¯ΦT
LX
¯Φ
¯ΛX
R) + off(ST ¯ΨT
LY
¯Ψ
¯ΛY
S) + µ FT ¯ΦR − GT ¯ΨS 2
F
s.t. RT ¯ΦT ¯Φ
I
R = I, ST ¯ΨT ¯Ψ
I
S = I
Coupling: given a set of corresponding vectors F, G, make
their Fourier coefficients coincide ˆΦ
T
F ≈ ˆΨ
T
G
Based on perturbation Theorem, express the joint approximate
eigenbases as linear combinations ˆΦ = ¯ΦS, ˆΨ = ¯ΨR
Kovnatsky, Bronstein2
, Glashoff, Kimmel 2013; Cardoso 1994
64 / 85
65. Subspace coupled diagonalization
Find two sets of coupled approximate eigenvectors ˆΦ, ˆΨ
min
R,S
off(RT ¯ΛXR) + off(ST ¯ΛY S) + µ FT ¯ΦR − GT ¯ΨS 2
F
s.t. RT
R = I, ST
S = I
Coupling: given a set of corresponding vectors F, G, make
their Fourier coefficients coincide ˆΦ
T
F ≈ ˆΨ
T
G
Based on perturbation Theorem, express the joint approximate
eigenbases as linear combinations ˆΦ = ¯ΦS, ˆΨ = ¯ΨR
Kovnatsky, Bronstein2
, Glashoff, Kimmel 2013; Cardoso 1994
65 / 85
66. Subspace coupled diagonalization
Find two sets of coupled approximate eigenvectors ˆΦ, ˆΨ
min
R,S
off(RT ¯ΛXR) + off(ST ¯ΛY S) + µ1 FT
+
¯ΦR − GT
+
¯ΨS 2
F
+µ2 FT
−
¯ΦR − GT
−
¯ΨS 2
F s.t. RT
R = I, ST
S = I
Coupling: given a set of corresponding vectors F, G, make
their Fourier coefficients coincide ˆΦ
T
F ≈ ˆΨ
T
G
Based on perturbation Theorem, express the joint approximate
eigenbases as linear combinations ˆΦ = ¯ΦS, ˆΨ = ¯ΨR
Decoupling: given a set of corresponding vectors F−, G−,
make their Fourier coefficients as different as possible
Kovnatsky, Bronstein2
, Glashoff, Kimmel 2013; Cardoso 1994
66 / 85
67. Subspace coupled diagonalization
Find two sets of coupled approximate eigenvectors ˆΦ, ˆΨ
min
R,S
off(RT ¯ΛXR) + off(ST ¯ΛY S) + µ1 FT
+
¯ΦR − GT
+
¯ΨS 2
F
+µ2 FT
−
¯ΦR − GT
−
¯ΨS 2
F s.t. RT
R = I, ST
S = I
Laplacians are not used explicitly: their first k
eigenfunctions ¯Φ, ¯Ψ and eigenvalues ¯ΛX, ¯ΛY are
pre-computed
Kovnatsky, Bronstein2
, Glashoff, Kimmel 2013; Cardoso 1994
67 / 85
68. Subspace coupled diagonalization
Find two sets of coupled approximate eigenvectors ˆΦ, ˆΨ
min
R,S
off(RT ¯ΛXR) + off(ST ¯ΛY S) + µ1 FT
+
¯ΦR − GT
+
¯ΨS 2
F
+µ2 FT
−
¯ΦR − GT
−
¯ΨS 2
F s.t. RT
R = I, ST
S = I
Laplacians are not used explicitly: their first k
eigenfunctions ¯Φ, ¯Ψ and eigenvalues ¯ΛX, ¯ΛY are
pre-computed - any Laplacian can be used!
Kovnatsky, Bronstein2
, Glashoff, Kimmel 2013; Cardoso 1994
68 / 85
69. Subspace coupled diagonalization
Find two sets of coupled approximate eigenvectors ˆΦ, ˆΨ
min
R,S
off(RT ¯ΛXR) + off(ST ¯ΛY S) + µ1 FT
+
¯ΦR − GT
+
¯ΨS 2
F
+µ2 FT
−
¯ΦR − GT
−
¯ΨS 2
F s.t. RT
R = I, ST
S = I
Laplacians are not used explicitly: their first k
eigenfunctions ¯Φ, ¯Ψ and eigenvalues ¯ΛX, ¯ΛY are
pre-computed - any Laplacian can be used!
Problem size is 2k × k, independent of the number of
samples
Kovnatsky, Bronstein2
, Glashoff, Kimmel 2013; Cardoso 1994
69 / 85
70. Subspace coupled diagonalization
Find two sets of coupled approximate eigenvectors ˆΦ, ˆΨ
min
R,S
off(RT ¯ΛXR) + off(ST ¯ΛY S) + µ1 FT
+
¯ΦR − GT
+
¯ΨS 2
F
+µ2 FT
−
¯ΦR − GT
−
¯ΨS 2
F s.t. RT
R = I, ST
S = I
Laplacians are not used explicitly: their first k
eigenfunctions ¯Φ, ¯Ψ and eigenvalues ¯ΛX, ¯ΛY are
pre-computed - any Laplacian can be used!
Problem size is 2k × k, independent of the number of
samples
No bijective correspondence
Kovnatsky, Bronstein2
, Glashoff, Kimmel 2013; Cardoso 1994
70 / 85
77. Manifold Alignment
831 120×100 images of a human face
698 64×64 images of a statue
manually coupled datasets, using 25 points sampled with FPS
results compared to manifold alignment (MA)
Ham, Lee, Saul 2005
77 / 85
81. Summary
Framework for multimodal data analysis
working in the subspace of the eigenvectors of the Laplacians
... and with only partial correspondences
81 / 85
82. Summary
Framework for multimodal data analysis
working in the subspace of the eigenvectors of the Laplacians
... and with only partial correspondences
We have:
some papers (see our Web pages)
82 / 85
83. Summary
Framework for multimodal data analysis
working in the subspace of the eigenvectors of the Laplacians
... and with only partial correspondences
We have:
some papers (see our Web pages)
code and data
83 / 85
84. Summary
Framework for multimodal data analysis
working in the subspace of the eigenvectors of the Laplacians
... and with only partial correspondences
We have:
some papers (see our Web pages)
code and data
extensions to other applications / fields
84 / 85