ICME2019 Tutorial: Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 4: retinex model based low light enhancement

1. IEEE ICME-2019
8:30 – 12:00, July 08
Shanghai, China
Intelligent
Image/Video Editing
Tutorial

2. STRUCT Group
Jiaying Liu
Wenhan Yang
Retinex Model-Based
Low Light Enhancement
Part 4

3. Low-Light Image Enhancement
Outline
Background and Related Work / 066
Robust Retinex Model / 085
Sequential Decomposition / 115
Deep Retinex Network / 139

4. Low-Light Image Enhancement
Outline
Background and Related Work / 066
Robust Retinex Model / 085
Sequential Decomposition / 115
Deep Retinex Network / 139

5. STRUCT GroupBackground
Low-light condition
 Low visibility
 Low contrast
 Intensive noise
5

6. STRUCT GroupBackground
Simple operations
 e.g. Histogram Equalization
6

7. STRUCT GroupBackground
Simple operations
 e.g. Histogram Equalization
7

8. STRUCT Group08
Representative Work
Related Works
Histogram
Equalization
 Enhance the contrast
 Over-enhancement / under-enhancement
 Amplify the noise
Before HE After HE

9. STRUCT Group09
Representative Work
Related Works
Histogram
Equalization
Dehazing Method
 Inverted low-light images vs. hazy images
 Invert  dehaze  invert again
 Require an additional denoising process
Low-Light Inversion Dehazing Result

10. STRUCT Group010
Representative Work
Related Works
Histogram
Equalization
Dehazing Method Retinex Model
 Retinex-based methods
 Retinex decomposition
 Generate results
S R L 
1
enhanceS R L
 
Gamma
Correction
Low-Light Image
Enhanced Image
Illumination (L)
Reflectance (R)

11. STRUCT GroupRelated Works
Retinex model based methods
 Retinex Model
 Estimate L, R = I/L, refine R
11

12. STRUCT GroupRelated Works
Retinex model based methods
 Retinex Model
 Estimate L, R = I/L, refine R
 NPE[TIP13] Bright-pass filter  preserve naturalness
12

13. STRUCT GroupRelated Works
Retinex model based methods
 Retinex Model
 Estimate L, R = I/L, refine R
 NPE[TIP13]
 Fusion based[SP16]
Bright-pass filter  preserve naturalness
13

14. STRUCT GroupRelated Works
Retinex model based methods
 Retinex Model
 Estimate L, R = I/L, refine R
 NPE[TIP13]
 Fusion based[SP16]
 LIME[TIP17]
Bright-pass filter  preserve naturalness
14

15. STRUCT GroupRelated Works
Retinex model based methods
 Retinex Model
 Estimate L, R = I/L, refine R
 NPE[TIP13]
 Fusion based[SP16]
 LIME[TIP17]
 Estimate L and R simultaneously
Bright-pass filter  preserve naturalness
15

16. STRUCT GroupRelated Works
Retinex model based methods
 Retinex Model
 Estimate L, R = I/L, refine R
 NPE[TIP13]
 Fusion based[SP16]
 LIME[TIP17]
 Estimate L and R simultaneously
 PIE[TIP15]
Bright-pass filter  preserve naturalness
16

17. STRUCT GroupRelated Works
Retinex model based methods
 Retinex Model
 Estimate L, R = I/L, refine R
 NPE[TIP13]
 Fusion based[SP16]
 LIME[TIP17]
 Estimate L and R simultaneously
 PIE[TIP15]
 SRIE[CVPR16]
Bright-pass filter  preserve naturalness

18. STRUCT GroupRelated Works
Retinex model based methods
 Retinex Model
 Estimate L, R = I/L, refine R
 NPE[TIP13]
 Fusion based[SP16]
 LIME[TIP17]
 Estimate L and R simultaneously
 PIE[TIP15]
 SRIE[CVPR16]
 CEID[TIP17]
Bright-pass filter  preserve naturalness
18

19. STRUCT GroupRelated Works
 Cannot handle noise
Input SRIE[CVPR16]
Retinex model based methods
19

20. STRUCT Group020
Representative Work
Related Works
Histogram
Equalization
Dehazing Method
Retinex Model
Learning-Based
Method
Low-Light Image Dataset
Regression Model OutputInput
Low-Light Image Dataset
…

21. STRUCT GroupRelated Works
Learning based methods
 LLNet[PR17]
 Deep autoencoder
21

22. STRUCT GroupRelated Works
Learning based methods
 LLCNN[VCIP17]
 Inception module
 Residual learning
 SSIM loss
22

23. Low-Light Image Enhancement
Outline
Background and Related Work / 066
Robust Retinex Model / 085
Sequential Decomposition / 115
Deep Retinex Network / 139

24. STRUCT Group24 Robust Retinex Model for Low Light Enhancement
 Robust Retinex Model for Low Light Enhancement
Structure-Revealing Low-Light Image Enhancement Via Robust
Retinex Model
Mading Li, Jiaying Liu, Wenhan Yang, Xiaoyan Sun, and Zongming Guo
TIP 2018
I R L  I R L N
Input image Retinex Model Robust Retinex Model

25. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
 An additional noise term
 Drawbacks of conventional model
 Focus on the estimation of L
 Got noisy reflectance for
 Calculate both R and L iteratively
 Introduce noise to illumination by minimizing
25
' /R R N L 
2
|| ||FR L S

26. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
 An additional noise term
 Priors for low-light images
 Illumination map  piece-wise smoothed
 Reflectance map  low contrast
 Noise map  relatively low intensity
26

27. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
 The optimization function
27

28. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
 The optimization function
Data fidelity term
28

29. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
 The optimization function
Input image Illumination map
Illumination constraint
29

30. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
 The optimization function
Reflectance constraint
30

31. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
 The optimization function
Without constraint With constraint
Reflectance constraint
31

32. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
 The optimization function
Noise constraint
Input image w/o constraint w/ constraint Noise map
32

33. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
 The solution
 Importing an auxiliary variable T
 Augmented Lagrange equation
33

34. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
 The solution
 Sub-problem R
 Sub-problem L
 Sub-problem N
34

35. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
 The solution
 Sub-problem T
 Updating auxiliary variables
35

36. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
PIE[TIP15] SRIE[CVPR16] Proposed
Input image LIME[TIP17]
NPE[TIP13]
36

37. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
PIE[TIP15] SRIE[CVPR16] Proposed
Input image LIME[TIP17]
NPE[TIP13]
37

38. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
Input image LIME[TIP17]
NPE[TIP13] PIE[TIP15] SRIE[CVPR16] ProposedHE
38

39. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
PIE[TIP15]
SRIE[CVPR16] Proposed
Input image LIME[TIP17]
NPE[TIP13]
39

40. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
PIE[TIP15]
SRIE[CVPR16] Proposed
Input image LIME[TIP17]
NPE[TIP13]
40

41. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
Input image LIME[TIP17]
NPE[TIP13] PIE[TIP15] SRIE[CVPR16] ProposedHE
41

42. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
 Objective criteria
The lower, the better quality The higher, the better quality
42

43. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
Input image
43

44. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
HE
44

45. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
CLAHE
45

46. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
Gamma Correction
46

47. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
LIME
47

48. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
NPE
48

49. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
SRIE
49

50. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
Proposed method
50

51. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
ProposedInput image Fu[ICIP14]
 Underwater image enhancement
51

52. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
 Smoky/hazy image enhancement
ProposedInput image He[CVPR09]
52

53. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
 Enhancement of images taken under dusty weather
ProposedInput image Fu[MMSP14]
53

54. Low-Light Image Enhancement
Outline
Background and Related Work / 066
Robust Retinex Model / 085
Sequential Decomposition / 115
Deep Retinex Network / 139

55. STRUCT Group55 Sequential Decomposition for Low Light Enhancement
 Sequential Decomposition for Low Light Enhancement
Joint Enhancement and Denoising Method via Sequential
Decomposition
Xutong Ren, Mading Li, Wen-Huang Cheng, and Jiaying Liu
ISCAS 2018

56. STRUCT GroupSequential Decomposition for Low Light Enhancement
Motivation
 Motivation
 Existing methods seldom consider noise
 Enhancement procedure
 Amplifies existing noise
Low-Light Image NPEA
56

57. STRUCT GroupSequential Decomposition for Low Light Enhancement
Architecture
In RGB SpaceS
Illumination Estimation
Reflectance Estimation
L
R
S’
෠𝐿
W G
Restrict Matrices
L’
57

58. STRUCT GroupSequential Decomposition for Low Light Enhancement
Illumination Estimation
2
1
ˆarg min || || || ||F
L
L L L  
Low-Light Image Initial Illumination Estimated Illumination
 Estimate illumination independent from reflectance
58

59. STRUCT GroupSequential Decomposition for Low Light Enhancement
Reflectance Estimation
 Estimate reflectance based on refined illumination
and original image
2 2 2
arg min || / || || || || ||F F F
R
R S L W R R G      
Low-Light Image S / L Estimated Reflectance
59

60. STRUCT GroupSequential Decomposition for Low Light Enhancement
Reflectance Estimation
 Use weighted matrices to restrict noise
ˆ| |/ ˆ(1 )
0, if | |ˆ
, otherwise
S
G e S
S
S
S



 
  
  
   
 
1
| |
W
S eps

 
G W
60

61. STRUCT GroupSequential Decomposition for Low Light Enhancement
Solution
 Estimate the illumination map
 Approximate:
 Rewrite the original problem:
 Simplify:
2
1
x {h,v}
( ( ))
|| || .
ˆ| L( ) |
d
d d
L x
L
x eps

 
 
 
2
2
x {h,v}
( ( ))ˆarg min || || .
ˆ| L( ) |
d
F
L d d
L x
L L
x eps



 
 
 
2 2
x {h,v}
ˆarg min || || ( ) ( ( )) .F d d
L d
L L A x L x

    
61

62. STRUCT GroupSequential Decomposition for Low Light Enhancement
Solution
 Estimate the illumination map
 Estimate the reflectance map
{h,v}
ˆDiag( )T
d d d
d
I D a D l l

 
  
 

{h,v} {h,v}
{h,v}
Diag( )
/
T T
d d d d d
d d
T
d d
d
I D w D D D r
s l D g
 

 

 
  
 
 
 

62

63. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
 Experimental settings
 All experiments are performed in MATLAB R2017a with
4G RAM and Intel Core i5-4210H CPU @2.90GHz.
 In our experiment the parameters α, β and γ are
empirically set as 0.007, 0.001 and 0.016.
 In our experiment the parameters ε and σ are set to 10
and λ is set to 6.
63

64. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
 LIME (top panel) and ours (bottom panel)
Input images Illumination Reflectance Result images Details
64

65. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
 LIME (top panel) and ours (bottom panel)
Input images Illumination Reflectance Result images Details
65

66. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
 Low-light Image Enhancement Results
(a) Input (b) HE (c) SRIE
(d) NPEA (e) LIME (f ) Our Method
66

67. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
 Low-light Image Enhancement Results
(a) Input (b) HE (c) SRIE
(d) NPEA (e) LIME (f ) Our Method
67

68. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
 Noisy Low-Light Image Enhancement Results
Input
68

69. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
 Noisy Low-Light Image Enhancement Results
PIE
69

70. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
 Noisy Low-Light Image Enhancement Results
SRIE
70

71. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
 Noisy Low-Light Image Enhancement Results
LIME
71

72. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
 Noisy Low-Light Image Enhancement Results
NPEA
72

73. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
 Noisy Low-Light Image Enhancement Results
Our Method
73

74. STRUCT GroupSequential Decomposition for Low Light Enhancement
Summary
 Based on a refined Retinex model
 Noise-removal and Enhancing
 Sequential decomposition
74

75. Low-Light Image Enhancement
Outline
Background and Related Work / 066
Robust Retinex Model / 085
Sequential Decomposition / 115
Deep Retinex Network / 139

76. STRUCT Group76 Deep Retinex Decomposition
 Deep Retinex Decomposition
Deep Retinex Decomposition for Low-Light Enhancement
Chen Wei*, Wenjing Wang*, Wenhan Yang, Jiaying Liu
* indicates equal contributions
BMVC 2018

77. STRUCT GroupDeep Retinex Decomposition
Hand-Crafted Retinex
 Hand-crafted constraints and manipulation
 Limited model capacity
OutputInput
Retinex
Decomposition
Adjusted
Decomposition
77

78. STRUCT GroupDeep Retinex Decomposition
Hand-Crafted Retinex
 Not easy to be adaptive to complex and varying low-light
conditions
Under-enhancementOver-enhancement Boundary artifacts
78

79. STRUCT GroupDeep Retinex Decomposition
Direct End-to-End Learning
 Difficulties in directly recovering normal-light images
Inherent ambiguity
Low-Light Image Dataset
Regression Model OutputInput
Low-Light Image Dataset
…
79

80. STRUCT GroupDeep Retinex Decomposition
Direct End-to-End Learning
 Regression to mean
 Over-smoothed results with degraded contrast
Over-smoothness Degraded contrast
80

81. STRUCT GroupDeep Retinex Decomposition
Our Solution: Retinex-Net
 Retinex Theory + Deep Learning
Low-Light Image Dataset
OutputInput
Retinex
Decomposition
Adjusted
Decomposition
Regression
Model
…
81

82. STRUCT GroupDeep Retinex Decomposition
Architecture of Retinex-Net
82

83. STRUCT GroupDeep Retinex Decomposition
Architecture of Retinex-Net
, ,
recon ij i j j
i low normal j low normal
L R I S
 
  
ir low normalL R R 
, ,
exp( )is j i g j i
i low normal j h v
L I R
 
    
 Reconstruction Loss
 Constant Reflectance Loss
 Illumination Smoothness Loss
83

84. STRUCT GroupDeep Retinex Decomposition
Architecture of Retinex-Net
, ,
exp( )is j i g j i
i low normal j h v
L I R
 
     ,
is i
i low normal
L I

 
 Illumination Smoothness Loss
84

85. STRUCT GroupDeep Retinex Decomposition
Architecture of Retinex-Net
 Training Phase
85

86. STRUCT GroupDeep Retinex Decomposition
Architecture of Retinex-Net
 Testing Phase
86

87. STRUCT GroupDeep Retinex Decomposition
Architecture of Retinex-Net
 Training Phase
87

88. STRUCT GroupDeep Retinex Decomposition
Architecture of Retinex-Net
 Testing Phase
88

89. STRUCT GroupDeep Retinex Decomposition
Real Photography Pairs
 LOw Light paired dataset (LOL)
 1000 low/normal-light image pairs
 500 are collected by changing only exposure time and ISO
 Various scenes, e.g., houses, clubs, streets, etc.
89

90. STRUCT GroupDeep Retinex Decomposition
Dataset
 Synthetic Pairs from Raw Images
 1000 raw images from RAISE[Dang-Nguyen 2015]
 Fitting the histogram of Y channel in YCbCr to real low-light images
 Online available: https://daooshee.github.io/BMVC2018website/
90

91. STRUCT GroupDeep Retinex Decomposition
Experiments: Image Decomposition
 Compared Methods
 NPE[Wang2013], Naturalness preserved enhancement algorithm
 LIME[Guo2017], Illumination Estimation based method
 Evaluation Dataset
 LOL, Evaluation set of LOL dataset, containing 50 images
91

92. STRUCT GroupDeep Retinex Decomposition
Decomposition
Low-Light Image R by LIME I by LIME
Normal-Light Image R by LIME I by LIME
92

93. STRUCT GroupDeep Retinex Decomposition
Decomposition
Low-Light Image R by NPE I by NPE
Normal-Light Image R by NPE I by NPE
93

94. STRUCT GroupDeep Retinex Decomposition
Decomposition
Low-Light Image R by Retinex-Net I by Retinex-Net
Normal-Light Image R by Retinex-Net I by Retinex-Net
94

95. STRUCT GroupDeep Retinex Decomposition
Decomposition
Low-Light Image R by LIME I by LIME
Normal-Light Image R by LIME I by LIME
95

96. STRUCT GroupDeep Retinex Decomposition
Decomposition
Low-Light Image R by NPE I by NPE
Normal-Light Image R by NPE I by NPE
96

97. STRUCT GroupDeep Retinex Decomposition
Decomposition
Low-Light Image R by NPE I by NPE
Normal-Light Image R by NPE I by NPE
97

98. STRUCT GroupDeep Retinex Decomposition
Experiments: Low-Light Enhancement
 Compared Methods
 DeHz[Dong2011], De-hazing based method
 NPE[Dong2011], Naturalness preserved enhancement algorithm
 SRIE[Fu2016], Simultaneous Reflection and Illumination Estimation
 LIME[Guo2017], Illumination Estimation based method
 Evaluation Dataset
 LIME[Guo2017], 10 low-light images
 MEF[Guo2017], 17 images sequences with multiple exposure levels
 DICM[Lee2013], 69 captured images with commercial digital cameras
98

99. STRUCT GroupDeep Retinex Decomposition
Experiments: Low-Light Enhancement
 Visual Results
99

100. STRUCT GroupDeep Retinex Decomposition
Experiments: Low-Light Enhancement
 Visual Results
100

101. STRUCT GroupDeep Retinex Decomposition
Experiments: Low-Light Enhancement
 Visual Results
101

102. STRUCT GroupDeep Retinex Decomposition
Experiments: Low-Light Enhancement
 Visual Results
102

103. STRUCT GroupDeep Retinex Decomposition
Experiments: Low-Light Enhancement
 Visual Results
103

104. STRUCT GroupDeep Retinex Decomposition
Experiments: Low-Light Enhancement
 Visual Results
104

105. STRUCT GroupDeep Retinex Decomposition
Experiments: Low-Light Enhancement
 Visual Results
105

106. STRUCT GroupDeep Retinex Decomposition
Experiments: Low-Light Enhancement
 Visual Results
106

107. STRUCT GroupDeep Retinex Decomposition
Experiments: Low-Light Enhancement
 Visual Results
107

108. STRUCT GroupDeep Retinex Decomposition
Experiments: Low-Light Enhancement
 Visual Results
108

109. STRUCT GroupDeep Retinex Decomposition
Experiments: Low-Light Enhancement
 Visual Results
109

110. STRUCT GroupDeep Retinex Decomposition
Experiments: Low-Light Enhancement
 Visual Results
110

111. STRUCT Group
liujiaying@pku.edu.cn
yangwenhan@pku.edu.cn