Visual and thermal cameras have large modality gap while providing different information of the same scene. The spectral mapping of thermal imagery into color imagery is a challenging task due to inherit nonlinear relationship. This paper deals with the automatic colorization of thermal imagery into color images using deep encoder-decoder convolutional neural network architecture. The presented approach is trained and evaluated on an online thermal/color dataset, where the network is trained on the spectral mapping of thermal to color images
9. Usman Qayyum Thermal Colorization using DNN
Motivation/Contributions
9
• Our work is the first one of colorizing thermal into RGB
images using deep neural network.
• A novel encoder-decoder deep neural network for end-to-end
learning.
• Colorizing single channel thermal image into multichannel
image
12. Usman Qayyum Thermal Colorization using DNN
Proposed Architecture
12
• Encoder layer compresses or summarizes the information.
• resolution of the feature map decreases whereas the depth
increases.
• 3x3 2d convolution filters are being used
• Throughout in our network we have used ReLu activation function
• The decoder layer upsamples the feature map
• Increase resolution and reduce depth map in order to reconstruct
the encoder spectral mapping
13. Usman Qayyum Thermal Colorization using DNN
Pre-Processing
13
• HSV separates the luminance component
(V) from the chrominance component
• Thermal and color images are registered
using homography by manually selected
corresponding points.
14. Usman Qayyum Thermal Colorization using DNN
DNN Training
14
Where !(!!; !) ∈ !!"#"!
is the output of the predicted color image for j
th
training pairs. The ! ∈
!!"#"!
, is the ground truth color image in HSV color-space and! ∈ !!"#"!
is the input thermal
image.!represents the estimated parameters, i is the every image pixel and H, W are image dimensions
Loss
15. Usman Qayyum Thermal Colorization using DNN
Results: Evaluation
• Quantitative Evaluation
− RMSPROP, SGD, ADAM
Convergence
• Qualitative Evaluation
− Euclidean norm between
predicted & ground truth
Where Ediff is the Euclidean difference between the predicted
colorized image (IP) and ground truth image (IG) for every image
pixel i.
15
16. Usman Qayyum Thermal Colorization using DNN
Results: Qualitative Evaluation
16
• We have used the online dataset of OSU thermal color (>5k
Images, 70% used for training, 30% for testing)
• busy pathway intersection with occasional cloudy
scenes.
17. Usman Qayyum Thermal Colorization using DNN
Limitations
17
• Doesn’t recover true color
• Acquisition time and sun elevation angle
(not integrated)
* [1]
18. Usman Qayyum Thermal Colorization using DNN
Conclusion
18
• Thermal Colorization
• Encoder-Decoder Deep Neural Network
Architecture
• Results
[1] Gustav l., et al. ”Learning Representations for Automatic Colorization”, ECCV 2016
[2] Cheng, Z., Yang, Q., and Sheng, B. . Deep colorization. ICCV 2015.
[3] Deshpande, A., Rock, J., and Forsyth, D. . Learning large-scale automatic image colorization. ICCV 2015.
[4] Levin, A., Lischinski, D., and Weiss, Y. (2004). Colorization using optimization. ACM Transactions on Graphics (TOG), 23(3).
[5] Luan, Q., Wen, F., Cohen-Or, D., Liang, L., Xu, Y.-Q., and Shum, H.-Y. (2007). Natural image colorization. In Eurographics
conference on Rendering Techniques.
[6] Zhang, R., Isola, P., and Efros, A. A. (2016). Colorful image colorization. In ECCV.
[7] Vijay B, Alex K. and Roberto C., Segnet, A Deep convolutional encoder-decoder architecture for image segmentation, PAMI 2017
