Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Optimal Transportin Imaging Sciences      Gabriel Peyréwww.numerical-tours.com
Statistical Image ModelsColors distribution: ) each pixel         Source image (X                           point in R3   ...
Statistical Image ModelsColors distribution: ) each pixel         Source image (X                                         ...
Statistical Image ModelsColors distribution: ) each pixel         Source image (X                                         ...
Overview• Wasserstein Distance• Sliced Wasserstein Distance• Color Transfer• Wasserstein Barycenter• Gaussian Texture Models
Discrete Distributions                        N 1Discrete measure: µ =         pi   Xi   Xi   Rd       pi = 1             ...
Discrete Distributions                        N 1Discrete measure: µ =          pi   Xi   Xi   Rd       pi = 1            ...
Discrete Distributions                        N 1Discrete measure: µ =          pi   Xi    Xi    Rd         pi = 1        ...
From Images to Statistics                               Source image (X )Discretized image f   RN   d                     ...
From Images to Statistics         Source image (X )Discretized image f      RN   d                                        ...
From Images to Statistics         Source image (X )Discretized image f      RN   d                                        ...
From Images to Statistics            Source image (X )Discretized image f      RN   d                                     ...
From Images to Statistics            Source image (X )Discretized image f      RN   d                                     ...
Optimal Transport Distances                                              Grayscale: 1-DVector X RN d                 µX = ...
Optimal Transport Distances                                                       Grayscale: 1-DVector X RN d             ...
Optimal Transport Distances                                                            Grayscale: 1-DVector X RN d        ...
Optimal Transport Distances                                                             Grayscale: 1-DVector X RN d       ...
Computing Transport DistancesExplicit solution for 1D distribution (e.g. grayscale images):                               ...
Computing Transport DistancesExplicit solution for 1D distribution (e.g. grayscale images):                               ...
Computing Transport DistancesExplicit solution for 1D distribution (e.g. grayscale images):                               ...
Overview• Wasserstein Distance• Sliced Wasserstein Distance• Color Transfer• Wasserstein Barycenter• Gaussian Texture Models
Approximate Sliced DistanceKey idea: replace transport in Rd by series of 1D transport.                                   ...
Approximate Sliced DistanceKey idea: replace transport in Rd by series of 1D transport.                                   ...
Approximate Sliced DistanceKey idea: replace transport in Rd by series of 1D transport.                                   ...
Approximate Sliced DistanceKey idea: replace transport in Rd by series of 1D transport.                                   ...
Sliced AssignmentTheorem: X is a local minima of E(X) = SW (µX , µY )2                           N, X = Y
Sliced AssignmentTheorem: X is a local minima of E(X) = SW (µX , µY )2                           N, X = YStochastic gradie...
Sliced AssignmentTheorem: X is a local minima of E(X) = SW (µX , µY )2                           N, X = YStochastic gradie...
Sliced Assignment Theorem: X is a local minima of E(X) = SW (µX , µY )2                            N, X = Y Stochastic gra...
Overview• Wasserstein Distance• Sliced Wasserstein Distance• Color Transfer• Wasserstein Barycenter• Gaussian Texture Models
pplication to Color Transfer                Color Histogram Equalization                                                  ...
pplication to Color Transfer                Color Histogram Equalization                                                  ...
pplication to Color Transfer                Color Histogram Equalization                                                  ...
pplication to Color Transfer                  Color Histogram Equalization                                                ...
Sliced Wasserstein TransfertSolving min ||f0    f1 ⇥ || is computationally untractable.            NApproximate Wasserstei...
Sliced Wasserstein TransfertSolving min ||f0              f1 ⇥ || is computationally untractable.                  NApprox...
pplication to Color Transfer             Color Transfert                               Sliced Wasserstein projection of X ...
Color ExchangeSource image (X (X ) Source image )Source image (XfInput image ) Source image (X )               0       Tar...
Overview• Wasserstein Distance• Sliced Wasserstein Distance• Color Transfer• Wasserstein Barycenter• Gaussian Texture Models
Wasserstein Barycenter                                                                             µ2Barycenter of {(µi , ...
Wasserstein Barycenter                                                                               µ2Barycenter of {(µi ...
Wasserstein Barycenter                                                                               µ2Barycenter of {(µi ...
Special Case: 2 DistributionsCase L = 2:     µt argmin (1     t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2            µ                 ...
Special Case: 2 DistributionsCase L = 2:     µt argmin (1         t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2              µ           ...
Linear Programming ResolutionDiscrete setting:   i = 1, . . . , L,   µi =   k Xi (k)                                      ...
Linear Programming ResolutionDiscrete setting:          i = 1, . . . , L,     µi =     k Xi (k)L-way interaction cost:   C...
Linear Programming ResolutionDiscrete setting:          i = 1, . . . , L,     µi =     k Xi (k)L-way interaction cost:   C...
Linear Programming ResolutionDiscrete setting:            i = 1, . . . , L,      µi =     k Xi (k)L-way interaction cost: ...
Linear Programming ResolutionDiscrete setting:            i = 1, . . . , L,         µi =   k Xi (k)L-way interaction cost:...
Sliced Wasserstein BarycenterSliced-barycenter:       µX that solves           min           i SW (µX , µXi )2            ...
Sliced Wasserstein BarycenterSliced-barycenter:        µX that solves               min        i SW (µX , µXi )2          ...
Sliced Wasserstein BarycenterSliced-barycenter:        µX that solves               min        i SW (µX , µXi )2          ...
Sliced Wasserstein BarycenterSliced-barycenter:        µX that solves               min        i SW (µX , µXi )2          ...
. . . Step compute Sliced-Wasserstein Barycenter ofofof color statistics;    Step 1: compute Sliced-Wasserstein Barycenter...
. . . Step compute Sliced-Wasserstein Barycenter ofofof color statistics;    Step 1: compute Sliced-Wasserstein Barycenter...
. . . Step compute Sliced-Wasserstein Barycenter ofofof color statistics;    Step 1: compute Sliced-Wasserstein Barycenter...
Overview• Wasserstein Distance• Sliced Wasserstein Distance• Color Transfer• Wasserstein Barycenter• Gaussian Texture Models
Texture Synthesis                                  “random” Problem: given f0 , generate f                                ...
Texture Synthesis                                      “random” Problem: given f0 , generate f                            ...
Texture Synthesis                                       “random” Problem: given f0 , generate f                           ...
Texture Synthesis                                       “random” Problem: given f0 , generate f                           ...
Gaussian Texture ModelInput exemplar:   f0   RN   d                                    N2                                 ...
Gaussian Texture ModelInput exemplar:      f0     RN   d                                     N2                           ...
Gaussian Texture ModelInput exemplar:       f0     RN   d                                      N2                         ...
Gaussian Texture ModelInput exemplar:       f0     RN   d                                      N2                         ...
Spot Noise Model [Galerne et al.]Stationarity hypothesis:   (periodic BC)   X(· + )   X
Spot Noise Model [Galerne et al.]Stationarity hypothesis:    (periodic BC)              X(· + )   XBlock-diagonal Fourier ...
Spot Noise Model [Galerne et al.]Stationarity hypothesis:    (periodic BC)              X(· + )   XBlock-diagonal Fourier ...
Spot Noise Model [Galerne et al.]Stationarity hypothesis:     (periodic BC)              X(· + )    XBlock-diagonal Fourie...
Spot Noise Model [Galerne et al.]Stationarity hypothesis:    (periodic BC)              X(· + )    XBlock-diagonal Fourier...
Example of SynthesisSynthesizing f = X( ), X     N (m, ):      = 0,    ˆ       ˆ              f ( ) = f0 ( )w( )          ...
Gaussian Optimal TransportInput distributions (µ0 , µ1 ) with µi = N (mi ,     i ).Ellipses: Ei = x     Rd  (mi    x)   i ...
Gaussian Optimal TransportInput distributions (µ0 , µ1 ) with µi = N (mi ,                     i ).Ellipses: Ei = x       ...
Wasserstein GeodesicsGeodesics between (µ0 , µ1 ): t   [0, 1]   µtVariational caracterization:  (W2 is a geodesic distance...
Wasserstein GeodesicsGeodesics between (µ0 , µ1 ): t     [0, 1]   µtVariational caracterization: (W2 is a geodesic distanc...
Wasserstein GeodesicsGeodesics between (µ0 , µ1 ): t        [0, 1]    µtVariational caracterization: (W2 is a geodesic dis...
Geodesic of Spot NoisesTheorem: Let for i = 0, 1, µi = µ(f [i] ) be spot noises,               ˆ        ˆi.e. ˆ i ( ) = f ...
Geodesic of Spot NoisesTheorem: Let for i = 0, 1, µi = µ(f [i] ) be spot noises,               ˆ        ˆi.e. ˆ i ( ) = f ...
OT Barycenters                      µ2                 µ1                      µ3
OT Barycenters                      µ2                 µ1                      µ3
OT Barycenters                      µ2                 µ1                      µ3
2-D Gaussian BarycentersEuclidean     Optimal transport   Rao
Spot Noise BarycentersInput
Spot Noise BarycentersInput
Results following the geodesic path:                               3Abstract.- This paper tackles static and dynamic textu...
Conclusion                           Source image (X )Image modeling with statistical constraints      Colorization, synth...
Conclusion                           Source image (X )Image modeling with statistical constraints      Colorization, synth...
Conclusion                                          Source image (X )Image modeling with statistical constraints      Colo...
Nächste SlideShare
Wird geladen in …5
×

Optimal Transport in Imaging Sciences

Slides of the presentation at the conference "Optimal Transport to Orsay", Orsay, France, June 18-22, 2012.

Ähnliche Bücher

Kostenlos mit einer 30-tägigen Testversion von Scribd

Alle anzeigen

Ähnliche Hörbücher

Kostenlos mit einer 30-tägigen Testversion von Scribd

Alle anzeigen
  • Als Erste(r) kommentieren

Optimal Transport in Imaging Sciences

  1. 1. Optimal Transportin Imaging Sciences Gabriel Peyréwww.numerical-tours.com
  2. 2. Statistical Image ModelsColors distribution: ) each pixel Source image (X point in R3 Source image after color transfer Style image (Y ) J. Rabin Wasserstein Regularization
  3. 3. Statistical Image ModelsColors distribution: ) each pixel Source image (X point in R3 Optimal transport framework Sliced Wasserstein projection Applications Application to Color Transfer Optimal transport framework Sliced Wasserstein projection Applications Application to Color Transfer Sliced Wasserstein projection color style Source image after of X to transfer image color statistics Y Style image (Y ) J. Rabin Wasserstein Regularization Source image (X ) Sliced Wasserstein projection of X to style image color statistics Y Input image Source image (X ) Modified color transfer Source image after image Style image (Y ) J. Rabin Wasserstein Regularization
  4. 4. Statistical Image ModelsColors distribution: ) each pixel Source image (X point in R3 Optimal transport framework Sliced Wasserstein projection Applications Application to Color Transfer Optimal transport framework Sliced Wasserstein projection Applications Application to Color Transfer Sliced Wasserstein projection color style Source image after of X to transfer image color statistics Y Style image (Y ) J. Rabin Wasserstein Regularization Source image (X ) Sliced Wasserstein projection of X to style image color statistics Y Input image Source image (X ) Modified color transfer Source image after image Style image (Y ) Texture synthesisOther applications: J. Rabin Wasserstein Regularization Texture segmentation
  5. 5. Overview• Wasserstein Distance• Sliced Wasserstein Distance• Color Transfer• Wasserstein Barycenter• Gaussian Texture Models
  6. 6. Discrete Distributions N 1Discrete measure: µ = pi Xi Xi Rd pi = 1 i=0 i
  7. 7. Discrete Distributions N 1Discrete measure: µ = pi Xi Xi Rd pi = 1 i=0 i Point cloudConstant weights: pi = 1/N . Xi Quotient space: RN d / N
  8. 8. Discrete Distributions N 1Discrete measure: µ = pi Xi Xi Rd pi = 1 i=0 i Point cloud HistogramConstant weights: pi = 1/N . Fixed positions Xi (e.g. grid) Xi Quotient space: A ne space: RN d / N {(pi )i i pi = 1}
  9. 9. From Images to Statistics Source image (X )Discretized image f RN d fi Rd = R3 N = #pixels, d = #colors. Source image Style image (Y ) J. Rabin Wasserstein Regularizatio
  10. 10. From Images to Statistics Source image (X )Discretized image f RN d fi Rd = R3 N = #pixels, d = #colors. Source imageDisclamers: images are not distributions. image (Y ) Style J. Rabin Wasserstein Regularizatio Needs an estimator: f µf Modify f by controlling µf .
  11. 11. From Images to Statistics Source image (X )Discretized image f RN d fi Rd = R3 N = #pixels, d = #colors. Source imageDisclamers: images are not distributions. image (Y ) Style J. Rabin Wasserstein Regularizatio Needs an estimator: f µf Modify f by controlling µf . XiPoint cloud discretization: µf = fi i
  12. 12. From Images to Statistics Source image (X )Discretized image f RN d fi Rd = R3 N = #pixels, d = #colors. Source imageDisclamers: images are not distributions. image (Y ) Style J. Rabin Wasserstein Regularizatio Needs an estimator: f µf Modify f by controlling µf . XiPoint cloud discretization: µf = fi iHistogram discretization: µf = pi Xi i 1 Parzen windows: pi = (xi fj ) Zf j
  13. 13. From Images to Statistics Source image (X )Discretized image f RN d fi Rd = R3 N = #pixels, d = #colors. Source imageDisclamers: images are not distributions. image (Y ) Style J. Rabin Wasserstein Regularizatio Needs an estimator: f µf Modify f by controlling µf . XiPoint cloud discretization: µf = fi i Today’s focusHistogram discretization: µf = pi Xi i 1 Parzen windows: pi = (xi fj ) Zf j
  14. 14. Optimal Transport Distances Grayscale: 1-DVector X RN d µX = Xi(image, coe cients, . . . ) i Colors: 3-D B G R
  15. 15. Optimal Transport Distances Grayscale: 1-DVector X RN d µX = Xi(image, coe cients, . . . ) i Colors: 3-D B GOptimal assignment: ⇥ ⇥ argmin ||Xi Y (i) ||p R N i
  16. 16. Optimal Transport Distances Grayscale: 1-DVector X RN d µX = Xi(image, coe cients, . . . ) i Colors: 3-D B GOptimal assignment: ⇥ ⇥ argmin ||Xi Y (i) ||p R N iWasserstein distance: Wp (µX , µY )p = ||Xi Y (i) ||p i Metric on the space of distributions.
  17. 17. Optimal Transport Distances Grayscale: 1-DVector X RN d µX = Xi(image, coe cients, . . . ) i Colors: 3-D B GOptimal assignment: ⇥ ⇥ argmin ||Xi Y (i) ||p R N iWasserstein distance: Wp (µX , µY )p = ||Xi Y (i) ||p i Metric on the space of distributions. Projection on statistical constraints: C = {f µf = µY } ProjC (f ) = Y
  18. 18. Computing Transport DistancesExplicit solution for 1D distribution (e.g. grayscale images): Xi Yi sorting the values, O(N log(N )) operations.
  19. 19. Computing Transport DistancesExplicit solution for 1D distribution (e.g. grayscale images): Xi Yi sorting the values, O(N log(N )) operations.Higher dimensions: combinatorial optimization methods Hungarian algorithm, auctions algorithm, etc. O(N 5/2 log(N )) operations. intractable for imaging problems.
  20. 20. Computing Transport DistancesExplicit solution for 1D distribution (e.g. grayscale images): Xi Yi sorting the values, O(N log(N )) operations.Higher dimensions: combinatorial optimization methods Hungarian algorithm, auctions algorithm, etc. O(N 5/2 log(N )) operations. intractable for imaging problems.Arbitrary distributions: µ= pi Xi ⇥= qi Yi i i Wp (µ, ) solution of a linear program. p
  21. 21. Overview• Wasserstein Distance• Sliced Wasserstein Distance• Color Transfer• Wasserstein Barycenter• Gaussian Texture Models
  22. 22. Approximate Sliced DistanceKey idea: replace transport in Rd by series of 1D transport. Xi [Rabin, Peyr´, Delon & Bernot 2010] eProjected point cloud: X = { Xi , ⇥}i . Xi ,
  23. 23. Approximate Sliced DistanceKey idea: replace transport in Rd by series of 1D transport. Xi [Rabin, Peyr´, Delon & Bernot 2010] eProjected point cloud: X = { Xi , ⇥}i . Xi ,Sliced Wasserstein distance: (p = 2) SW (µX , µY )2 = W (µX , µY )2 d || ||=1
  24. 24. Approximate Sliced DistanceKey idea: replace transport in Rd by series of 1D transport. Xi [Rabin, Peyr´, Delon & Bernot 2010] eProjected point cloud: X = { Xi , ⇥}i . Xi ,Sliced Wasserstein distance: (p = 2) SW (µX , µY )2 = W (µX , µY )2 d || ||=1Theorem: E(X) = SW (µX , µY )2 is of class C 1 and E(X) = Xi Y⇥ (i) , d . where N are 1-D optimal assignents of X and Y .
  25. 25. Approximate Sliced DistanceKey idea: replace transport in Rd by series of 1D transport. Xi [Rabin, Peyr´, Delon & Bernot 2010] eProjected point cloud: X = { Xi , ⇥}i . Xi ,Sliced Wasserstein distance: (p = 2) SW (µX , µY )2 = W (µX , µY )2 d || ||=1Theorem: E(X) = SW (µX , µY )2 is of class C 1 and E(X) = Xi Y⇥ (i) , d . where N are 1-D optimal assignents of X and Y . Possible to use SW in variational imaging problems. Fast numerical scheme : use a few random .
  26. 26. Sliced AssignmentTheorem: X is a local minima of E(X) = SW (µX , µY )2 N, X = Y
  27. 27. Sliced AssignmentTheorem: X is a local minima of E(X) = SW (µX , µY )2 N, X = YStochastic gradient descent of E(X): Step 1: choose at random. E (X) = W (X , Y )2 Step 2: X( +1) = X( ) E (X ( ) )
  28. 28. Sliced AssignmentTheorem: X is a local minima of E(X) = SW (µX , µY )2 N, X = YStochastic gradient descent of E(X): Step 1: choose at random. E (X) = W (X , Y )2 Step 2: X( +1) = X( ) E (X ( ) )X( ) converges to C = {X µX = µY }.
  29. 29. Sliced Assignment Theorem: X is a local minima of E(X) = SW (µX , µY )2 N, X = Y Stochastic gradient descent of E(X): Step 1: choose at random. E (X) = W (X , Y )2 Step 2: X( +1) = X( ) E (X ( ) ) X( ) converges to C = {X µX = µY }. Final assignmentNumerical observation: X ( ) ProjC (X (0) )
  30. 30. Overview• Wasserstein Distance• Sliced Wasserstein Distance• Color Transfer• Wasserstein Barycenter• Gaussian Texture Models
  31. 31. pplication to Color Transfer Color Histogram Equalization 1 Input color images: fi RN 3 . projectioniof= to style Sliced Wasserstein ⇥ X N x fi (x) image color statistics Y Optimal transport framework Sliced Wasserstein projection Applications Application to Color Transfer Source image (X ) f1 f0 Sliced Wasserstein projec image color statistics Y f0 Source image after color transfer µ1 image (Y ) Style Source image (X ) µ0 J. Rabin Wasserstein Regularization
  32. 32. pplication to Color Transfer Color Histogram Equalization 1 Input color images: fi RN 3 . projectioniof= to style Sliced Wasserstein ⇥ X N x fi (x) image color statistics Y Optimal assignement: min ||f0 f1 ⇥ || N Optimal transport framework Sliced Wasserstein projection Applications Application to Color Transfer Source image (X ) f1 f0 Sliced Wasserstein projec image color statistics Y f0 Source image after color transfer µ1 image (Y ) Style Source image (X ) µ0 J. Rabin Wasserstein Regularization
  33. 33. pplication to Color Transfer Color Histogram Equalization 1 Input color images: fi RN 3 . projectioniof= to style Sliced Wasserstein ⇥ X N x fi (x) image color statistics Y Optimal assignement: min ||f0 f1 ⇥ || N Transport: T : f0 (x) R3 f1 ( (i)) R3 Optimal transport framework Sliced Wasserstein projection Applications Application to Color Transfer Source image (X ) f1 f0 Sliced Wasserstein projec image color statistics Y f0 Source image after color transfer µ1 image (Y ) Style Source image (X ) µ0 T J. Rabin Wasserstein Regularization
  34. 34. pplication to Color Transfer Color Histogram Equalization 1 Input color images: fi RN 3 . projectioniof= to style Sliced Wasserstein ⇥ X N x fi (x) image color statistics Y Optimal assignement: min ||f0 f1 ⇥ || N Optimal transport framework Sliced Wasserstein projection Applications Transport: T : f0 (x) R3Application to Color Transfer R3 f1 ( (i)) Optimal transport framework Sliced Wasserstein projection Applications Application to Color Transfer ˜ Equalization: ) f0 = T (f0 ) ˜ f0 = f1 Sliced Wasserstein projection of X to sty Source image (X image color statistics Y f1 f0 T (f ) 0 Sliced Wasserstein projec image color statistics Y Source image (X ) T f0 Source image after color transfer µ1 image (Y ) Style Source image (X ) µ0 Source image after color transfer µ1 Style image (Y ) T J. Rabin Wasserstein Regularization J. Rabin Wasserstein Regularization
  35. 35. Sliced Wasserstein TransfertSolving min ||f0 f1 ⇥ || is computationally untractable. NApproximate Wasserstein projection: ˜ f0 solves min E(f ) = SW (µf , µf1 ) f and is close to f0
  36. 36. Sliced Wasserstein TransfertSolving min ||f0 f1 ⇥ || is computationally untractable. NApproximate Wasserstein projection: ˜ f0 solves min E(f ) = SW (µf , µf1 ) f and is close to f0(Stochastic) gradient descent: f (0) = f0 f( +1) = f( ) E(f ( ) ) f( ) ˜ f0At convergence: µf0 = µf1 ˜
  37. 37. pplication to Color Transfer Color Transfert Sliced Wasserstein projection of X to style image color statistics Y Sliced Wasserstein projection of X to styl sty image color statistics Y Source image (X ) Input image f0 ) Source image (X ˜ Source image after image f0 Transfered color transfer Style image (Y ) Source image after color transfer Target image )f1 Style image (Y
  38. 38. Color ExchangeSource image (X (X ) Source image )Source image (XfInput image ) Source image (X ) 0 TargetXimage ⇥ f1 Y X ⇥⇥ Y X Y X ⇥ Y Style image (Y ) Style image (Y )Transfered image f0 Style image (Y ) ˜ Transfered X ˜ Y ⇥ image f1
  39. 39. Overview• Wasserstein Distance• Sliced Wasserstein Distance• Color Transfer• Wasserstein Barycenter• Gaussian Texture Models
  40. 40. Wasserstein Barycenter µ2Barycenter of {(µi , )}L : i i=1 i =1 ) L i 2,µ µ argmin W2 (µi , µ)2 2 (µ i µ W µ i=1 µ1 ) W2 W2 (µ 1, µ (µ 3 ,µ ) µ3
  41. 41. Wasserstein Barycenter µ2Barycenter of {(µi , )}L : i i=1 i =1 ) L i 2,µ µ argmin W2 (µi , µ)2 2 (µ i µ W µ i=1 µ1 ) W2 (µ 1, µIf µi = Xi , then µ = W2 (µ 3 X ,µ X = ) i Xi µ3 i Generalizes Euclidean barycenter.
  42. 42. Wasserstein Barycenter µ2Barycenter of {(µi , )}L : i i=1 i =1 ) L i 2,µ µ argmin W2 (µi , µ)2 2 (µ i µ W µ i=1 µ1 ) W2 (µ 1, µIf µi = Xi , then µ = W2 (µ 3 X ,µ X = ) i Xi µ3 i Generalizes Euclidean barycenter. Theorem: [Agueh, Carlier, 2010] if µ0 does not vanish on small sets, µ exists and is unique.
  43. 43. Special Case: 2 DistributionsCase L = 2: µt argmin (1 t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2 µ µ2 µ1 µt µt is the geodesic path.
  44. 44. Special Case: 2 DistributionsCase L = 2: µt argmin (1 t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2 µ µ2 µ1 µt µt is the geodesic path. NDiscrete point clouds: µ= k=1 Xi (k) , NAssignment: ⇥ ⇥ argmin ||X1 (k) X2 ( (k))||2 N k=1 N µ = k=1 X (k) , where X = (1 t)X1 (k) + tX2 ( (k))
  45. 45. Linear Programming ResolutionDiscrete setting: i = 1, . . . , L, µi = k Xi (k) µ2 µ1 µ µ3
  46. 46. Linear Programming ResolutionDiscrete setting: i = 1, . . . , L, µi = k Xi (k)L-way interaction cost: C(k1 , . . . , kL ) = i,j i j ||Xi (ki ) Xj (kj )||2 µ2 µ1 µ µ3
  47. 47. Linear Programming ResolutionDiscrete setting: i = 1, . . . , L, µi = k Xi (k)L-way interaction cost: C(k1 , . . . , kL ) = i j ||Xi (ki ) Xj (kj )||2 =1 i,jL-way coupling matrices: P PL =1 =1 µ2 µ1 µ µ3
  48. 48. Linear Programming ResolutionDiscrete setting: i = 1, . . . , L, µi = k Xi (k)L-way interaction cost: C(k1 , . . . , kL ) = i j ||Xi (ki ) Xj (kj )||2 =1 i,jL-way coupling matrices: P PLLinear program: =1 min Pk1 ,...,kL C(k1 , . . . , kL ) =1 P PL k1 ,...,kL µ2 µ1 µ µ3
  49. 49. Linear Programming ResolutionDiscrete setting: i = 1, . . . , L, µi = k Xi (k)L-way interaction cost: C(k1 , . . . , kL ) = i j ||Xi (ki ) Xj (kj )||2 =1 i,jL-way coupling matrices: P PLLinear program: =1 min Pk1 ,...,kL C(k1 , . . . , kL ) =1 P PL k1 ,...,kLBarycenter: µ2 µ = Pk1 ,...,kL µ1 X (k1 ,...,kL ) k1 ,...,kL L µ µ3 X (k1 , . . . , kL ) = i=1 i Xi (ki )
  50. 50. Sliced Wasserstein BarycenterSliced-barycenter: µX that solves min i SW (µX , µXi )2 X i
  51. 51. Sliced Wasserstein BarycenterSliced-barycenter: µX that solves min i SW (µX , µXi )2 X iGradient descent: Ei (X) = SW (µX , µXi )2 L X( +1) = X( ) ⇥ i Ei (X ( ) ) i=1
  52. 52. Sliced Wasserstein BarycenterSliced-barycenter: µX that solves min i SW (µX , µXi )2 X iGradient descent: Ei (X) = SW (µX , µXi )2 L X( +1) = X( ) ⇥ i Ei (X ( ) ) i=1Advantages: µX is a sum of N Diracs. Smooth optimization problem.Disadvantage: Non-convex problem local minima.
  53. 53. Sliced Wasserstein BarycenterSliced-barycenter: µX that solves min i SW (µX , µXi )2 X iGradient descent: Ei (X) = SW (µX , µXi )2 L X( +1) = X( ) ⇥ i Ei (X ( ) ) µ1 i=1Advantages: µX is a sum of N Diracs. Smooth optimization problem.Disadvantage: µ2 µ3 Non-convex problem local minima.
  54. 54. . . . Step compute Sliced-Wasserstein Barycenter ofofof color statistics; Step 1: compute Sliced-Wasserstein Barycenter color statistics; Step 1: 1: compute Sliced-Wasserstein Barycenter color statistics; Application: Color Harmonization. . . Step compute Sliced-Wasserstein projection ofofof each image onto the Step 2: compute Sliced-Wasserstein projection each image onto the Step 2: 2: compute Sliced-Wasserstein projection each image onto the Barycenter; Barycenter;Barycenter; Raw image sequence Raw image sequence Raw image sequence Raw image sequence J. RabinRabinGREYC, University Caen Caen Approximate Wasserstein Metric forfor for Texture Synthesis and Mixing J. RabinGREYC, University of of Caen J. – – – GREYC, University of Approximate Wasserstein Metric Texture Synthesis and Mixing Approximate Wasserstein Metric Texture Synthesis and Mixing
  55. 55. . . . Step compute Sliced-Wasserstein Barycenter ofofof color statistics; Step 1: compute Sliced-Wasserstein Barycenter color statistics; Step 1: 1: compute Sliced-Wasserstein Barycenter color statistics; Application: Color Harmonization. . . Step compute Sliced-Wasserstein projection ofofof each image onto the Step 2: compute Sliced-Wasserstein projection each image onto the Step 2: 2: compute Sliced-Wasserstein projection each image onto the Barycenter; Barycenter;Barycenter; Raw image sequence Compute Wasserstein Raw image sequence Raw image sequence Raw image sequence barycenter J. RabinRabinGREYC, University Caen Caen Approximate Wasserstein Metric forfor for Texture Synthesis and Mixing J. RabinGREYC, University of of Caen J. – – – GREYC, University of Approximate Wasserstein Metric Texture Synthesis and Mixing Approximate Wasserstein Metric Texture Synthesis and Mixing
  56. 56. . . . Step compute Sliced-Wasserstein Barycenter ofofof color statistics; Step 1: compute Sliced-Wasserstein Barycenter color statistics; Step 1: 1: compute Sliced-Wasserstein Barycenter color statistics; Application: Color Harmonization. . . Step compute Sliced-Wasserstein projection ofofof each image onto the Step 2: compute Sliced-Wasserstein projection each image onto the Step 2: 2: compute Sliced-Wasserstein projection each image onto the Barycenter; Barycenter;Barycenter; Raw imagestein BarycenterSliced Wasserstein Barycenter Experimental results Applications Conclusionerstein BarycenterSliced Wasserstein Barycenter Experimental results Applications Conclusionin Barycenter Sliced Wasserstein Barycenter Experimental results Applications Conclusion sequence lor transferor transferolor transfer Color harmonization of several imagesColor harmonization of of several images Color harmonization several images Step 1: compute Sliced-Wasserstein Barycenter of color statistics;. . . Step 1: compute Sliced-Wasserstein Barycenter color statistics; Step 1: compute Sliced-Wasserstein Barycenter of of color statistics; Step 2: compute Sliced-Wasserstein projection of each image onto the. . . Step 2: compute Sliced-Wasserstein projection each image onto thethe Step 2: compute Sliced-Wasserstein projection of of each image onto Compute Barycenter;Barycenter; Barycenter; Wasserstein Raw image sequence Raw image sequence Raw image sequence barycenter J. RabinRabinGREYC, University Caen Caen Approximate Wasserstein Metric forfor for Texture Synthesis and Mixing J. RabinGREYC, University of of Caen J. – – – GREYC, University of Approximate Wasserstein Metric Texture Synthesis and Mixing Approximate Wasserstein Metric Texture Synthesis and Mixing Project on the barycenter
  57. 57. Overview• Wasserstein Distance• Sliced Wasserstein Distance• Color Transfer• Wasserstein Barycenter• Gaussian Texture Models
  58. 58. Texture Synthesis “random” Problem: given f0 , generate f perceptually “similar”Exemplar f0
  59. 59. Texture Synthesis “random” Problem: given f0 , generate f perceptually “similar” Probability analysis distribution µ = µ(p)Exemplar f0
  60. 60. Texture Synthesis “random” Problem: given f0 , generate f perceptually “similar” Probability analysis synthesis distribution µ = µ(p)Exemplar f0 Outputs f µ(p)
  61. 61. Texture Synthesis “random” Problem: given f0 , generate f perceptually “similar” Probability analysis synthesis distribution µ = µ(p)Exemplar f0 Outputs f µ(p) Gaussian models: µ = N (m, ), parameters p = (m, ).
  62. 62. Gaussian Texture ModelInput exemplar: f0 RN d N2 N2 d = 1 (grayscale), d = 3 (color) N3 N1 Images N1 Videos
  63. 63. Gaussian Texture ModelInput exemplar: f0 RN d N2 N2 d = 1 (grayscale), d = 3 (color) N3Gaussian model: X µ = N (m, ) N1 Images N1 Videos m RN d , RN d Nd
  64. 64. Gaussian Texture ModelInput exemplar: f0 RN d N2 N2 d = 1 (grayscale), d = 3 (color) N3Gaussian model: X µ = N (m, ) N1 Images N1 Videos m RN d , RN d NdTexture analysis: from f0 RN d , learn (m, ). highly under-determined problem.
  65. 65. Gaussian Texture ModelInput exemplar: f0 RN d N2 N2 d = 1 (grayscale), d = 3 (color) N3Gaussian model: X µ = N (m, ) N1 Images N1 Videos m RN d , RN d NdTexture analysis: from f0 RN d , learn (m, ). highly under-determined problem.Texture synthesis: given (m, ), draw a realization f = X( ). Factorize = AA (e.g. Cholesky). Compute f = m + Aw where w drawn from N (0, Id).
  66. 66. Spot Noise Model [Galerne et al.]Stationarity hypothesis: (periodic BC) X(· + ) X
  67. 67. Spot Noise Model [Galerne et al.]Stationarity hypothesis: (periodic BC) X(· + ) XBlock-diagonal Fourier covariance: ˆ y = f computed as y ( ) = ˆ ( )f ( ) ˆ 2ix1 ⇥1 2ix2 ⇥2 ˆ where f ( ) = f (x)e N1 + N2 x
  68. 68. Spot Noise Model [Galerne et al.]Stationarity hypothesis: (periodic BC) X(· + ) XBlock-diagonal Fourier covariance: ˆ y = f computed as y ( ) = ˆ ( )f ( ) ˆ 2ix1 ⇥1 2ix2 ⇥2 ˆ where f ( ) = f (x)e N1 + N2 xMaximum likelihood estimate (MLE) of m from f0 : 1 i, mi = f0 (x) Rd N x
  69. 69. Spot Noise Model [Galerne et al.]Stationarity hypothesis: (periodic BC) X(· + ) XBlock-diagonal Fourier covariance: ˆ y = f computed as y ( ) = ˆ ( )f ( ) ˆ 2ix1 ⇥1 2ix2 ⇥2 ˆ where f ( ) = f (x)e N1 + N2 xMaximum likelihood estimate (MLE) of m from f0 : 1 i, mi = f0 (x) Rd N x 1MLE of : i,j = f0 (i + x) f0 (j + x) Rd d N x
  70. 70. Spot Noise Model [Galerne et al.]Stationarity hypothesis: (periodic BC) X(· + ) XBlock-diagonal Fourier covariance: ˆ y = f computed as y ( ) = ˆ ( )f ( ) ˆ 2ix1 ⇥1 2ix2 ⇥2 ˆ where f ( ) = f (x)e N1 + N2 xMaximum likelihood estimate (MLE) of m from f0 : 1 i, mi = f0 (x) Rd N x 1MLE of : i,j = f0 (i + x) f0 (j + x) Rd d N x ˆ ˆ = 0, ˆ ( ) = f0 ( )f0 ( ) Cd d is a spot noise = 0, ˆ ( ) is rank-1.
  71. 71. Example of SynthesisSynthesizing f = X( ), X N (m, ): = 0, ˆ ˆ f ( ) = f0 ( )w( ) ˆ w N (N 1 ,N 1/2 IdN ) Cd C Convolve each channel with the same white noise. Input f0 RN 3 Realizations f
  72. 72. Gaussian Optimal TransportInput distributions (µ0 , µ1 ) with µi = N (mi , i ).Ellipses: Ei = x Rd (mi x) i 1 (mi x) c E0 E1
  73. 73. Gaussian Optimal TransportInput distributions (µ0 , µ1 ) with µi = N (mi , i ).Ellipses: Ei = x Rd (mi x) i 1 (mi x) c E0 T E1 Theorem: If ker( 0) Im( 1) = {0}, T (x) = Sx + m1 m0 where 1/2 1/2 1/2 1/2 1/2 S= 1 + 0,1 1 0,1 =( 1 0 1 ) W2 (µ0 , µ1 )2 = tr ( 0+ 1 2 0,1 ) + ||m0 m1 ||2 ,
  74. 74. Wasserstein GeodesicsGeodesics between (µ0 , µ1 ): t [0, 1] µtVariational caracterization: (W2 is a geodesic distance) µt = argmin (1 t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2 µ
  75. 75. Wasserstein GeodesicsGeodesics between (µ0 , µ1 ): t [0, 1] µtVariational caracterization: (W2 is a geodesic distance) µt = argmin (1 t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2 µ µ1 µ0 µOptimal transport caracterization: µt = ((1 t)Id + tT ) µ0
  76. 76. Wasserstein GeodesicsGeodesics between (µ0 , µ1 ): t [0, 1] µtVariational caracterization: (W2 is a geodesic distance) µt = argmin (1 t)W2 (µ0 , µ)2 + tW2 (µ1 , µ)2 µ µ1 µ0 µOptimal transport caracterization: µt = ((1 t)Id + tT ) µ0 µ0Gaussian case: µt = Tt µ0 = N (mt , t) mt = (1 t)m0 + tm1 t = [(1 t)Id + tT ] 0 [(1 t)Id + tT ] the set of Gaussians is geodesically convex. µ1
  77. 77. Geodesic of Spot NoisesTheorem: Let for i = 0, 1, µi = µ(f [i] ) be spot noises, ˆ ˆi.e. ˆ i ( ) = f [i] ( )f [i] ( ) . Then t [0, 1], µt = µ(f [t] ) f [t] = (1 t)f [0] + tg [1] ˆ f [1] ( ) ˆ f [0] ( ) ˆ g [1] ( ) = f [1] ( ) ˆ ˆ |f [1] ( ) ˆ f [0] ( )|
  78. 78. Geodesic of Spot NoisesTheorem: Let for i = 0, 1, µi = µ(f [i] ) be spot noises, ˆ ˆi.e. ˆ i ( ) = f [i] ( )f [i] ( ) . Then t [0, 1], µt = µ(f [t] ) f [t] = (1 t)f [0] + tg [1] ˆ f [1] ( ) ˆ f [0] ( ) ˆ g [1] ( ) = f [1] ( ) ˆ ˆ |f [1] ( ) ˆ f [0] ( )| f [0] t f [1] 0 1
  79. 79. OT Barycenters µ2 µ1 µ3
  80. 80. OT Barycenters µ2 µ1 µ3
  81. 81. OT Barycenters µ2 µ1 µ3
  82. 82. 2-D Gaussian BarycentersEuclidean Optimal transport Rao
  83. 83. Spot Noise BarycentersInput
  84. 84. Spot Noise BarycentersInput
  85. 85. Results following the geodesic path: 3Abstract.- This paper tackles static and dynamic texture mixing by combining the statistical properties of an input set of images or videos. We focus on spot noiseimages%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Dynamic Textures Mixingtextures that follow a stationary and Gaussian model which can be learned from the given exemplars. From here, we define, using optimal transport, the distance f [0] f [1] f [2]between texture models, derive the geodesic path, and define the barycenter between several texture 2 models. These derivations are useful because they allow the 4user to navigate inside the set of texture A T L A interpolating a new texture model at each element 1 the set. From these new interpolated models, new textures M models, B C O D E of 5can be synthesized of arbitrary size in space and time. Numerical results obtained from a library of exemplars show the ability of our method to generate newcomplex realistic static and dynamic textures. 6 7 [See web page] 0Results following the geodesic path: <%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Start 3 images%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% f [0] f [1]2 f [2]MATLABCODE 4 1 0 1 5 2 3 4 5 6 6 Exemplars 0 7<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Start <%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%images%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% <%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% f [0] f [1] f [0] [2] f [1] [2] f f 0 1 2 3 4 5 6 7<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%> f [0] [1] [2] f 0 f 1 2 3 4 5 6 <%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% <%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% f [0] [1] f [2] f 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 <%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%> f [0] [1] [2] f f 0 1 2 3 4 5 6 7
  86. 86. Conclusion Source image (X )Image modeling with statistical constraints Colorization, synthesis, mixing, . . . Source ima Style image (Y ) J. Rabin Wasserstein Regula
  87. 87. Conclusion Source image (X )Image modeling with statistical constraints Colorization, synthesis, mixing, . . .Wasserstein distance approach Source ima Fast sliced approximation. Style image (Y ) J. Rabin Wasserstein Regula
  88. 88. Conclusion Source image (X )Image modeling with statistical constraints Colorization, synthesis, mixing, . . . 14 AnonymousWasserstein distance approach Source ima Fast sliced approximation. Style image (Y ) J. Rabin Wasserstein RegulaExtension to a wide range of imaging problems. Color transfert, segmentation, . . . P (⇥⇥ ) P (⇥ ) P (⇥ c ) P (⇥⇥ ) in P (⇥ ) out P (⇥ ) c

    Als Erste(r) kommentieren

    Loggen Sie sich ein, um Kommentare anzuzeigen.

  • SinaNakhostin

    Nov. 17, 2015
  • shr386

    May. 20, 2017
  • juneokumura

    Sep. 7, 2017
  • neo884

    Oct. 23, 2017
  • KotaroKikuchi1

    Mar. 22, 2018
  • mccha3

    Jun. 23, 2018
  • MinsooKim188

    Jul. 13, 2020
  • jaehyeonBahk

    Apr. 20, 2021
  • richardjhur

    Jul. 16, 2021

Slides of the presentation at the conference "Optimal Transport to Orsay", Orsay, France, June 18-22, 2012.

Aufrufe

Aufrufe insgesamt

2.607

Auf Slideshare

0

Aus Einbettungen

0

Anzahl der Einbettungen

121

Befehle

Downloads

96

Geteilt

0

Kommentare

0

Likes

9

×