Color transfer between high-dynamic-range images
•Download as PPTX, PDF•
0 likes•1,349 views
Color transfer methods alter the look of a source image with regards to a reference image. So far, the proposed color transfer methods have been limited to low-dynamic-range (LDR) images. Unlike LDR images, which are display-dependent, high-dynamic-range (HDR) images contain real physical values of the world luminance and are able to capture high luminance variations and finest details of real world scenes. Therefore, there exists a strong discrepancy between the two types of images. In this paper, we bridge the gap between the color transfer domain and the HDR imagery by introducing HDR extensions to LDR color transfer methods. We tackle the main issues of applying a color transfer between two HDR images. First, to address the nature of light and color distributions in the context of HDR imagery, we carry out modifications of traditional color spaces. Furthermore, we ensure high precision in the quantization of the dynamic range for histogram computations. As image clustering (based on light and colors) proved to be an important aspect of color transfer, we analyze it and adapt it to the HDR domain. Our framework has been applied to several state-of-the-art color transfer methods. Qualitative experiments have shown that results obtained with the proposed adaptation approach exhibit less artifacts and are visually more pleasing than results obtained when straightforwardly applying existing color transfer methods to HDR images.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Color transfer between high-dynamic-range images
Similar to Color transfer between high-dynamic-range images (12)
More from Olivier Le Meur
Recently uploaded
Recently uploaded (20)
Color transfer between high-dynamic-range images
- 1. Color transfer between high- dynamic-range images H. Hristova, R. Cozot, O. Le Meur, K. Bouatouch University of Rennes 1 Rennes, France
- 2. Outline ● Introduction - Main objective - Contributions ● Extension to the HDR domain of a color transfer method ● Results and evaluation ● Generalization for state-of-the-art color transfer methods ● Conclusion 2
- 3. Main goal ● Carrying out a color transfer between two HDR images directly in the HDR domain Input Reference 3 ● Solution: apply color transfer methods to stylize an HDR image with regards to a reference image
- 4. Why do LDR color transfer methods need to be extended to the HDR domain? ● LDR color spaces - well predict the color gamut for luminance levels between zero and the display white point - uncertain applicability to HDR images ● Color trend above the perfect diffuse white 4
- 5. Why do LDR color transfer methods need to be extended to the HDR domain? ● Assumption: a unique multivariate Gaussian distribution ● HDR domain: to fit the high range of lightness of HDR images we need to assume mixture of Gaussian distributions 5
- 6. Why do LDR color transfer methods need to be extended to the HDR domain? ● Lightness - approximated by luminance in the LDR domain ● HDR domain - distinguish between the absolute luminance and the lightness (the L channel of CIE Lab) 6
- 7. Contributions ● Adaptation of [Hristova et al., 2015] color transfer method to HDR images - HDR color spaces - Modifications of the clustering step and of the image classification ● Cluster-based local chromatic adaptation transform ● Generalization for state-of-the-art color transfer methods 7
- 8. 8 Extension to HDR images • Linear search for significant peaks in the image hue histogram - Colors-based style images: more than one significant color cluster - Light-based style images: one significant color cluster Input and reference images Color space conversion Image classification Clustering and mapping Color transfer Chromatic adaptation transform Final result [Hristovaetal.,2015] • The number of significant peaks determines the number of clusters - Colors-based style images: hue histogram - Light-based style images: luminance histogram
- 9. Extension to HDR images Input and reference images Color space conversion Image classification Clustering and mapping Color transfer Chromatic adaptation transform Final result 9 [Hristovaetal.,2015] LDR images CIE Lab L channel of CIE Lab HDR images hdr-CIELab Log- luminance ● Dashed line: cubic function of L channel (CIE Lab) ● Solid line: Michaelis-Menten function by which we replace the cubic function of L channel (CIE Lab) ● hdr-CIELab color space [Fairchild et al., 2004] [Fairchild et al., 2004] ModificationsModified
- 10. Extension to HDR images 10 LDR images CIE Lab L channel of CIE Lab L-based clustering HDR images hdr-CIELab Log- luminance Log- luminance clustering [Hristovaetal.,2015] Logarithmic transform Input and reference images Color space conversion Image classification Clustering and mapping Color transfer Chromatic adaptation transform Final result ModifiedModifications
- 11. Extension to HDR images 11 Local CAT Cluster- based local CAT [Hristovaetal.,2015] LDR images CIE Lab L channel of CIE Lab L-based clustering HDR images hdr-CIELab Log- luminance Log- luminance clustering Input and reference images Color space conversion Clustering and mapping Color transfer Chromatic adaptation transform Final result Image classification ModifiedModifications
- 12. Extension to HDR images 12 Gaussianlow-passfilter (h) (m) (sh) (sh) (m) (h) Local CAT Cluster- based local CAT [Hristovaetal.,2015] LDR images CIE Lab L channel of CIE Lab L-based clustering HDR images hdr-CIELab Log- luminance Log- luminance clustering Input and reference images Color space conversion Clustering and mapping Color transfer Chromatic adaptation transform Final result Image classification ModifiedModifications
- 13. Extension to HDR images 13 Input ReferenceCluster-based local CAT Local CAT Cluster- based local CAT [Hristovaetal.,2015] LDR images CIE Lab L channel of CIE Lab L-based clustering HDR images hdr-CIELab Log- luminance Log- luminance clustering Input and reference images Color space conversion Clustering and mapping Color transfer Chromatic adaptation transform Final result Image classification ModifiedModifications
- 14. Extension to HDR images 14 Input ReferenceCluster-based local CAT Local CAT Cluster- based local CAT [Hristovaetal.,2015] LDR images CIE Lab L channel of CIE Lab L-based clustering HDR images hdr-CIELab Log- luminance Log- luminance clustering Input and reference images Color space conversion Clustering and mapping Color transfer Chromatic adaptation transform Final result Image classification ModifiedModifications
- 15. Objective evaluation of the results 15 ● 10 image pairs ● Two tone-mapping operators: [Durand et al., 2002] and [Reinhard et al., 2002] ● SSIM and Bhattacharya coefficient
- 16. Results 16 Input Reference Color transfer with CAT Color transfer without CAT Color transfer with CAT Color transfer without CAT [Hristova et al., 2015] HDR extension
- 17. Generalization and results 17 [Reinhard et al., 2001] - global method [Tai et al., 2005] - clustering (local transformations) Input Reference
- 18. Generalization and results 18 [Pitié et al., 2007] - CIE Lab [Pitié et al., 2007] - hdr-CIELab Input Reference
- 19. Generalization and results 19 [Bonneel et al., 2013] - luminance clustering [Bonneel et al., 2013] - log-luminance clustering Input Reference
- 20. Conclusion ● Extension of a novel local color transfer method [Hristova et al., 2015] - Modifications to CIE Lab -> hdr-CIELab - Luminance/Lightness -> Log-luminance ● Generalization to state-of-the-art methods ● Future work - Need for a more precise color mapping/color transformation between two HDR images - Need for better HDR color spaces 20
- 21. Thank you for your attention! 21
Editor's Notes
- Outline of the presentation: Introduction to the main objective and the main contributions of the paper Presenting an extension to the HDR domain of a novel local color transfer method (Followed by) results and evaluation Generalizing the developed extension to other state-of-the-art color transfer methods + results
- Our main goal is to carry out a color transfer between two HDR images directly in the HDR domain. -Why? -Because tone mapping operators compress the information contained in HDR images and therefore the plausibility of the tone-mapped images cannot be guaranteed. Solution to the problem: to apply color transfer methods between two HDR images. Color transfer methods build a transformation mapping between two LDR images and aim at “borrowing” the look (mood) of a reference image. Can color transfer methods be directly applied to the HDR domain? -> next slide
- The color transfer methods need to be extended to HDR images. -Why? -First: Traditional LDR color spaces, such as lab and CIE Lab, which are widely used in color transfer algorithms, predict well the color gamut for luminance levels below the display white point. However, when it comes to luminance levels way beyond the display white point, like those of HDR images, the applicability of these color spaces becomes uncertain. For example, when the lightness reaches high levels (above 255 in the figure), the imagery primaries (x, y) ranging from 0 to 1 cannot depict and predict the color gamut for these high levels of luminance. To predict the color trend above the perfect diffuse white, we need to perform some modifications to traditional color spaces.
- Second: most color transfer methods build their transformations based on the assumption that a multivariate Gaussian distribution is sufficient to fit the luminance and chroma variations. However, this is hardly true in the LDR domain. The latter hold for the HDR domain. The high luminance range of an HDR image cannot be fitted only by one Gaussian distribution. Therefore, we need to assume a mixture of Gaussian distributions.
- Finally: Usually, the luminance is approximated by the lightness (the L channel from CIE Lab color space) in the LDR domain. Nevertheless, there is a distinguish between the luminance and the lightness in the HDR domain. On one hand, the luminance of HDR images is presented by the Y channel in XYZ color space and it is called absolute luminance. On the other hand, lightness is represented by the L channel of CIE Lab color space. Both of them are not equal in the context of HDR imagery. Therefore, color transfer methods which use these two notions in their algorithms need to be properly adapted to the HDR domain.
- Our main contributions are: the development of extension to HDR images of a new local color transfer method [Hristova et al., 2015], consisting of: - modifications of traditional color spaces (their replacement with HDR-based ones) - modifications of the clustering and classification algorithms of the local method: distinguish between lightness and luminance The development of novel cluster-based local chromatic adaptation transform for adapting the colors of the input HDr image to a reference white point.
- The local method by [Hristova et al., 2015] is a new method for color and style transfer consisting of several steps. These steps (either independently or as a combination) can be used in other color transfer methods. Therefore, we will first present the extension of [Hristova et al., 2015] and then, we will generalize that extension to other color transfer methods. [Hristova et al., 2015] starts with a conversion to CIE Lab color space. Once both images are converted to that color space, they are classified according to their main features (either colors or light) as shown on the figure. Two types of images are considered: colors-based and light-based ones. The classification boils down to finding significant peaks in the hue histogram of both images. During the classification step, we determine also the number of clusters which will be passed to the clustering process. On one hand, the number of color clusters is determined by the hue histogram. On the other hand, the number of luminance clusters is defined by the number of significant peaks in the luminance histogram of the image. After the classification, both images are clustered to Gaussian clusters and then, the input/reference clusters are mapped. Four policies are developed to map the clusters. They take into account both the luminance histogram and the luminance-hue distributions of images. A color transfer is carried out between each pair of corresponding clusters. An optimal transformation is used (Monge-Kantorovich optimization problem). Finally, a chromatic adaptation transform is applied to adapt the colors of the output image to the reference illuminant.
- The following extension of [Hristova et al., 2015] is proposed. The extension is applied to HDR images instead to LDR images. One of the issues of the direct applying of the method to HDR images has been already discussed: the uncertainty in predicting the color gamut for high luminance values. Therefore, instead of using the LDR CIE Lab color space, we perform modifications to its channels to derive hdr-CIELab color space [Fairchild et al., 2004]. The cubic function of the L channel (the dashed line) is replaced by Michaelis-Menten function (the solid line) for better approximation of the lightness in HDR images. Moreover, this modification influences the predictability of the color trend for HDR images. Furthermore, as the image classification depends strongly on the number of significant peaks in the image hue histogram, if we ensure a better predictability of the image hue, then we ensure the successful classification of images and the successful estimation of the number of color clusters (as the number of significant peaks corresponds to the number of color clusters). How about the number of light/luminance clusters? -> we adopt log-luminance histogram in the place of the L channel histogram -> explanation on the next slide.
- To find the optimal number of clusters for an image during the classification step, we need to find number of significant peaks in the luminance histogram of the image. Moreover, light-based images are clustered according to the luminance histogram. Finally, three of the mapping policies are based on luminance histogram information. In the LDR domain we can use the L channel of CIE Lab color space in the place of the luminance. However, in the HDR domain we use the absolute luminance and more precisely, the logarithmic transform of the absolute luminance (which we call log-luminance). The log-luminance is a good approximation of the brightness. Furthermore, the logarithmic transform preserves the regions of minima and maxima of a histogram and therefore we are able to recover the shadows, midtones and highlights from the log-luminance histogram. To this end, log-luminance histogram is used in the classification and clustering steps of the algorithm in the place of the lightness channel of hdr-CIELab color space. Once we carry out the image clustering and we map the reference to the input clusters, we apply an optimal transformation between each pair of corresponding clusters. We call the obtained image an output. The last step adapts the colors of the output image to the reference illuminant.
- In the LDR domain, CAT algorithm adapts the input image to a global reference point. The novel cluster-based local CAT algorithm, however, first clusters both images into regions (highlights, midtones, shadows). We search for significant peaks in the log-luminance histogram and then, we define the region limits as the minima between two significant peaks.
- Like in the LDR domain, we apply a low-pass Gaussian filter to the input image to compute locally (like a “white” image) the input illuminant. Then, we cluster the input image into regions. Each region corresponds to part of the “white” image which we take into account to carry out the transform region-wisely. As far as the reference image is concerned, we estimate a representative white point for each reference region. Then, this white point becomes a global white point for the region. The reference white points for each region are shown in blue dashed lines.
- Here is a video example of how the result changes over the iterations. Around the 10th iteration, the input image is overall adapted to the reference illuminant. However, there is a slight cast of the greenish input reference on the result. It is removed at the end of the iteration.
- Another example of illuminant adaptation (video).
- We used 10 image pairs to obtain 10 HDR image results for the original [Hristova et al., 2015] method and its extension. We evaluated the results in the LDR domain after tone-mapping them. SSIM and Bhattacharya coefficient are two complementary metrics (used also by [Hristova et al., 2015]) measuring respectively the degree of artifacts and how successful the color transfer is. The HDR extension obtains the highest scores for both metrics and for both tone-mapping operators.
- The results support the objective evaluation. The first column represents the input and reference images. The second column shows the results obtained with the method [Hristova et al., 2015] directly applied to HDR images (with and without local CAT). The third column shows the results obtained with the HDR extension of the method (with and without local CAT). If we apply [Hristova et al., 2015] directly to HDR images, then the method it opt for artifacts. Moreover, the details are lost. On the other hand, the HDR extension manages to preserve the details and properly transfers the reference colors to the output image.
- Now we show several results for state-of-the-art methods. The first row consists of the input and reference images. The first image on the second row is the result of the direct applying of the state-of-the-art method to HDR images and the second image on the second row presents the result of the same state-of-the-art method, modified in order to enhance the color transfer between two HDR images. [Reinhard et al., 2002] is a global method which assumes that the light and colors of the image can be fitted by a unique Gaussian distribution. As discussed, one Gaussian distribution is not sufficient to fit the high luminance range of HDR images. Therefore, to enhance the effect of the color transfer, we recommend to carry out Reinhard’s transformation locally on pairs of clusters. To this end, we recommend [Tai et al., 2005] to be used in the place of [Reinhard et al., 2005] in the HDR domain.
- [Pitié et al., 2007] is a global method which applies optimal mapping between the input and reference images. In the LDR domain, the methods adopts CIE Lab color space. As we can see from the first result on the figure, CIE Lab is less effective from this type of transformation between HDR images than hdr-CIELab. The latter color space predicts better the color palette of the reference image and manages to transfer it better to the final result.
- As a local method, [Bonneel et al., 2013] performs clustering to both input and reference images. The clustering is luminance-based. To extend the method to HDR images, we replace the luminance-based clustering with log-luminance-based clustering (as shown for [Hristova et al., 2015] earlier in the presentation). Moreover, instead of carrying out the transformation in CIE Lab color space, we adopt hdr-CIELab color space in the HDR domain. That way, the proposed extension lessens the visual artifacts in the final result (in comparison to the direct application of [Bonneel et al., 2013]).