Master Discussion "Gray Image Coloring Using Texture Similarity Measures" Faculty of Computers and Information - Menoufia University 2007

1. Gray Image Coloring Using Texture
Similarity Measures
by
E. Noura Abd El-Moez Semary
Thesis Submitted in accordance with the requirements of
The University of Monofiya for the degree of
Master of Computers and Information 1
( Information Technology )

2. Thesis summary on:
Gray Image Coloring Using Texture
Similarity Measures
Prof. Mohiy Prof. Nabil Dr.Waiel .S.
Supervised by:
.M.Hadhoud .A.Ismail Al-Kilani
Presented by
E. Noura Abd El-Moez Semary
For Master degree in Computers and Information
IT department, Faculty of Computers and information,
Menofia University

3. ‫:ملخص رسالة بعنوان‬
‫تلوين الصور الرمادية بإستخدام معايير تشابه‬
‫النسجة‬
‫د. وائل شوقي‬ ‫أ.د. نبيل عبد الواحد‬ ‫أ.د. محي محمد‬
‫:تحت إشراف‬
‫الكيلني‬ ‫إسماعيل‬ ‫هدهود‬
‫:مقدم من‬
‫م . نورا عبد المعز السباعي سمري‬
‫للحصول على درجة الماجستير في الحاسبات والمعلومات‬
‫قسم تكنولوجيا المعلومات - كلية الحاسبات و المعلومات - جامعة المنوفية‬

4. Outlines
Outlines
 Introduction Introduction

Automatic

 Automatic coloring in the literature coloring in the
literature
 TRICS ‘Texture Recognition based Image‘Texture
TRICS

Recognition
Coloring System’ based Image
Coloring
 Results System’
Results

 Conclusion Conclusion

Future work

 Future work
4

5. Introduction
Gray image principles
Outlines
 Introduction
 Automatic
coloring in the
literature
 TRICS ‘Texture
Recognition
based Image
Coloring
System’
Gray values  Results
 Conclusion
 Future work
.
.
.
.
.
.
.
.
.
.
.
0
50
100
150
200
250
255
5

6. Introduction
Gray image principles
Image Size Possible No. Actually No. Total size
Image
(pixel) colors used colors (byte)
px 120× 170 16,777,216 15,998 60.0 kb
170 ×120 px 256 246 21.2 kb
6

7. Introduction
Coloring Problem
There are two definitions to describe the gray value as an equation of
the three basic components of RGB color model (red, green, blue):
1: Intensity (most common used)
Gray = (Red + Green + Blue) /3
2: Luminance (NTSC standard for luminance)
Gray = (0.299 × Red) + (0.587 × Green) + (0.114 × Blue)
RGB Color R, G, B values Gray value Gray Color
87 ,150 ,100 128
147, 87, 149 128
149, 147, 87 128
THERE IS NO MATHEMATICAL
FORMULA
TO CONVERT FROM GRAY TO RGB 7

8. Introduction
Coloring Problem
HSL Color wheel Grayed Color Wheel
Similar Gray Values Similar Gray Values
8

9. Introduction
Coloring Types
1 . Hand coloring
 Adobe Photoshop
and Paintshop Pro
 Layers
 Changing Hue
 BlackMagic, photo
colorization software,
version 2.8, 2003
9

10. Introduction
Coloring Types
2 . Semi automatic
coloring
 Pseudocoloring is a
common example for
semi automatic
coloring technique
10

11. Introduction
Coloring Types
Outlines
3 . Automatic coloring  Introduction
 Automatic
i. Transformational coloring coloring in the
literature
ii. Matched image coloring  TRICS ‘Texture
Recognition
iii. User selected coloring based Image
Coloring
System’
 Results
 Conclusion
 Future work
11

12. Automatic coloring in the literature
1. Transformational Coloring
Outlines
 A transformation function Tk is applied on
Introduction 
Automatic 
the intensity value of each pixel coloring in the
Ig(i,j)
literature
resulting in the chromatic value Ick(i,j) for
TRICS ‘Texture 
Recognition
channel k based Image
Coloring
System’
 Results
Ic k (i, j ) = Tk [ Ig (i, j )]  Conclusion
 Future work
12

13. Automatic coloring in the literature
1. Transformational Coloring
 Al-Gindy et al * system.
× Results have unnatural look
* A. N. Al-Gindy, H. Al Ahmad, R. A. Abd Alhameed, M. S. Abou Naaj and P. S. Excell
’Frequency Domain Technique For Colouring Gray Level Images’ 2004 found in
www.abhath.org/html/modules/pnAbhath/download.php?fid=32 13

14. Automatic coloring in the literature
2. Matched image coloring
 The most similar pixel color is transferred to the corresponding gray
one by the color transfer technique proposed by E.Reinhard*;
l α β
* Reinhard, E. Ashikhmin, M., Gooch B. And Shirley, P., Color Transfer between Images, IEEE Computer
Graphics and Applications, September/October 2001, 34-40 14

15. Automatic coloring in the literature
2. Matched image coloring
 “Global matching procedure” of T. Welsh et al*
 “Local color transfer” of Y. Tai et al**.
× All these algorithms fail, when different colored
regions have similar intensities
* T. Welsh, M. Ashikhmin, K. Mueller. “Transferring color to greyscale images.” In Proceedings of the
29th Annual Conference on Computer Graphics and interactive Techniques, pp 277–280, 2002
** Y. Tai, J. Jia, C. Tang ‘Local Color Transfer via Probabilistic Segmentation by Expectation-
maximization‘, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(CVPR'05), Volume 1, pp. 747-754, 2005
15

16. Automatic coloring in the literature
2. Matched image coloring
 Welsh et al proposed also another technique to improve
the coloring results when the matching results are not
satisfying. It was achieved by asking users to identify
and associate small rectangles, called “swatches” in both
the source and destination images to indicate how
certain key colors should be transferred
16

17. Automatic coloring in the literature
3. User selection coloring
Outlines
Introduction
 Automatic
coloring in the
literature
 TRICS ‘Texture
Recognition
based Image
 User selection coloring gives high quality colors Coloring
× User dependent color quality System’
 Results
× Time-consuming  Conclusion
× Colorization must be fully recomputed for any slight change Future work
 in the
initial marked pixels
* A. Levin, D. Lischinski, Y. Weiss. “Colorization using optimization.” ACM Transactions on Graphics,
Volume 23, Issue 3, pp.689–694, 2004 17

18. TRICS System
Research Objectives
Outlines
 To simulate the human vision in coloring
Introduction

process Automatic

coloring in the
literature
 To be fully automatic coloring system TRICS ‘Texture

Recognition
 To spend so little execution time as possible
based Image
Coloring
as a basic requirement for video coloring.
System’
 Results
 Conclusion
 Future work
18

19. TRICS System
Structure
Gray image
1 2 A
Segmentation
Features extraction Segmentation
(Joint, wavelets, laws,…) (Mean Shift, K-Mean, FCM,..)
Segmented image,
Clusters
3 4
Classification B
Features extraction Classification
(Co-occurrence, Tamura, Wavelets energies) (K-NN classifier,..)
Samples features
Class labels
Classes Hues Database
5 6 7 Coloring
C
Convert image to Set Hue, Saturation, and Convert to
HSV channels Brightness RGB
19
Colored image

20. TRICS System
Structure 1. Segmentation Stage
 Feature extraction: (Pixel based )
1. pixel position
2. pixel intensity
3. texture features
 wavelets coefficients
 Laws kernels coefficients.
20

21. TRICS System
Structure 1. Segmentation Stage
1. Wavelets coefficients
× Quarter the image size.
 Up sampling
 Upper level construction
21

22. TRICS System
Structure 1. Segmentation Stage
Up sampling
Upper level construction
22

23. TRICS System
Structure 1. Segmentation Stage
2. Laws Kernels :
Level L5 = [ 1 4 6 4 1]
Edge E5 = [ -1 –2 0 2 1]
Spot S5 = [ -1 0 2 0 –1]
Wave W5= [ -1 2 0 –2 1]
Ripple R5 = [ 1 –4 6 –4 1]
-1 -4 -6 -4 -1
0 0 0 0 0
L5S5’ = 2 8 12 8 2
0 0 0 0 0
-1 -4 -6 -4 -1
23

24. TRICS System
Structure 1. Segmentation Stage
 Segmentation technique:
 Mean Shift *
 K-mean (Fast k-mean) **
 Adaptive Fast k-mean
* D. Comaniciu and P. Meer. ‘Mean shift: A robust approach toward feature space
analysis.’ PAMI, 24(5):603–619, May 2002
** C.Elkan, ‘Using the triangle inequality to accelerate k-Means.’ In Proc. of ICML
2003. pp 147--153 24

25. Mean Shift Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
25

26. Mean Shift Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
26

27. Mean Shift Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
27

28. Mean Shift Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
28

29. Mean Shift Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
29

30. Mean Shift Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
30

31. Mean Shift Region of
interest
Center of
mass
Objective : Find the densest region
31

32. TRICS System
Structure 1. Segmentation Stage
1. Mean Shift :
× So slow
× Many parameters
- hs=16,hr=16,m=500 - hs=8,hr=8,hw=4,m=500
(170×256) - Time: 0 34 15 - Time: 0 39 54
- classes : 9 - classes : 7
32

33. Fast (Accelerated) K-mean*
 Lemma 1: Let p be a point and
let c1 and c2 be centers.
c1
 If E(c1,c2) ≥ 2E(p,c1) then
E(p,c2) ≥ E(p,c1).
p
 Lemma 2: Let p be a point and
let c1 and c2 be centers. Then
 E(p,c2) ≥ max{0,E(p,c1) – E(c1,c2)} c2
* C.Elkan, ‘Using the triangle inequality to accelerate k-Means.’ In Proceedings of the 20th ICML,
Washington DC, 2003. pp 147--153 33

34. TRICS System
Structure 1. Segmentation Stage
2. Fast K-mean :
 with spatial features
× structured segmentation
 Increase no. clusters
k=3 k=9
34

35. TRICS System
Structure 1. Segmentation Stage
2. Fast K-mean :
 without spatial features
× scattered regions of same cluster
 disjoint region separation
1 1 1 2
2 3
1 4
3 5
before after
35

36. TRICS System
Structure 1. Segmentation Stage
2. Fast K-mean :
× Small regions (noise)
 Small regions elimination
Original Before After
36

37. TRICS System
Structure 1. Segmentation Stage
3. Adaptive Fast K-mean :
 Clusters number generation
 Minimum region size estimation
Fully automatic segmentation technique
37

38. TRICS System
Structure 1. Segmentation Stage
a. Clusters number generation :
• CEC “Combined Estimation Criterion”*:
•If the VRC index, for k clusters, is smaller f n
than 98 of the VRC index, for k-1 clusters, WCSS = ∑∑ ( xij − mij ) 2
the CEC is not satisfied.
i =1 j =1
f n
•If the VRC index, for k clusters, is larger
than 102 of the VRC index for k-1 clusters, or TSS = ∑∑ ( xij − M i ) 2
if k=1, the CEC is satisfied. i =1 j =1
BCSS = TSS − WCSS
•If the VRC index, for clusters, is smaller than
102 but larger than 98 of the VRC index for (n − k ) BCSS
clusters, the CEC is satisfied only if TSS for VRC =
k-1 clusters is smaller than 70 of TSS for k (k − 1)WCSS
clusters.
* D.Charalampidis, T.Kasparis, ‘Wavelet-Based Rotational Invariant Roughness Features for
Texture Classification and Segmentation’. IEEE Transactions on Image Processing.Vol.11.No.8
38
August 2002

39. TRICS System
Structure 1. Segmentation Stage
Start k=1
Gray Image
Fast k-mean
k=k+1 Calculate CEC Segmented Image
Yes
CEC Satisfied?
No
Stop…
Clusters number =k 39

40. TRICS System
Structure 1. Segmentation Stage
b. Minimum region size estimation :
• Split the disjoint regions.
• Count all regions size.
• Sort regions size and calculate the step between them.
• Select the regions size of step more than the largest
image dimension.
• Consider the minimum region size.
40

41. TRICS System
Structure 1. Segmentation Stage
Original gray wavelets Laws
A- F- T- E- EM-
Image Features Clusters Regions
time time time time time
19 12 42 31 59
Wavelets 4 8
sec sec min sec sec
size = 170×256
M=500, 10
EM = 324 11 3 min 31 59
Laws 4 7
sec sec 36 sec sec
sec
A-time : adaptive fast k-mean time, F-time: fixed k fast k-mean time, T-time: traditional k-
mean , E-time: elimination time , EM-time : Elimination time with estimating minimum size 41
region

42. TRICS System
Structure Database set1
 The training set consists of 32 classes of Brodatz texture
database
 Each image has a size of 256 × 256. Each image was
mirrored horizontally and vertically to produce a 512 ×512
image.
 The image is split into 16 images of size 128 ×128.
256 × 256 512 × 512 16 × 128 × 128
42

43. TRICS System
Structure Database set2
 The training set consists of 9 classes ‘cloud, sky, sea, sand,
tree, grass, stone, water, and wood’
 Each class has number of samples from 12 to 25 samples.
 These samples are taken from real natural images as random
64x64 rectangles.
43

44. TRICS System
Structure Database
 Database record:
 Sample
 Class (level1,level2)
 58 Features (6 Moment statistics, 4 Co-ocurance measures, 3
Tamura, and “ 15 wavelets mean, 15 wavelets variance , 15
wavelets energy” for five levels wavelets decomposition)
 Hue
44

45. TRICS System
Structure 2. Classification Stage
 Feature extraction (Region based)
 Rectangular region:
1. Maximum rectangle
2. 64 x 64 rectangle
× Arbitrary shape
 Padding rectangle*
* Ying Liu, Xiaofang Zhou, Wei-Ying Ma, ‘Extracting Texture Features
from Arbitrary-shaped Regions for Image Retrieval‘. 2004 IEEE
International Conference on Multimedia and Expo., Taipei, Jun. 2004
45

46. TRICS System
Structure 2. Classification Stage
 Feature extraction (Region based)
 Region based features:
 GLCM * measures (Energy, Entropy, Inertia, Homogeneity )
 Tamura * (Coarseness, Contrast , Directionality’)
 Wavelets coefficients for 5 levels
 Mean and variance **
 Energy ***
* P.Howarth, S.Ruger,: Evaluation of texture features for content-based image retrieval. In:
proceedings of the International Conference on Image and Video Retrieval, Springer-Verlag (2004)
326–324
** O. Commowick – C. Lenglet – C. Louchet, ‘Wavelet-Based Texture Classification and Retrieval’
2003 found in http://www.tsi.enst.fr/tsi/enseignement/ressources/mti/classif-textures/
*** Eka Aulia, ‘Hierarchical Indexing For Region Based Image Retrieval’, Master thesis of Science in 46
Industrial Engineering, Louisiana State University and Agricultural and Mechanical College, May 2005

47. TRICS System
Structure 2. Classification Stage
 GLCM and Tamura
× Scale variant features, not suitable for natural
textures
47

48. TRICS System
Structure 2. Classification Stage
 Wavelets mean and variance :
× Values were very scattered and the results
were not accurate for most cases.
 Wavelets energies :
Classification accuracy of 92% using “leave-
one-out” (each (sub) image is classified one
by one so that other (sub) images serve as
the training data) method
48

49. TRICS System
Structure 2. Classification Stage
a. Classification technique
 KNN classifier with (k=1,k=5,k=10,k=20)
 Distance Metric is L2 “Euclidean distance”
1
 2
2
E ( I , J ) =  ∑ I (i ) − J (i ) 
 i 
 k=5 gives accuracy up to 94% using “N-fold “ (the
collection of (sub) images is divided into N disjoint
sets, of which N-1 serve as training data in turn and
the Nth set is used for testing)
49

50. TRICS System
Structure 2. Classification Stage
50

51. TRICS System
Structure 3. Coloring Stage
 Color model conversion
 HSV/HSB color model
Change in Saturation Change in Brightness Change in Hue
Hue = 0, Luminance=0.5 Hue = 0, Saturation = 1 Sat=1, Luminance=1
 HSI/HLS color model
Change in Saturation Change in Luminance Change in Hue
Hue = 0, Luminance=0.5 Hue = 0, Saturation = 1 Sat=1, Luminance=1
51

52. TRICS System
Structure 3. Coloring Stage
 HSV & HSL Channels
52

53. TRICS System
Structure 3. Coloring Stage
 Setting Channels values
 Brightness
The gray image itself
 Hue
One hue value for each texture
 Saturation
HSV: 1- brightness HSI: 0.5(1-lightness)
53

54. TRICS System
Structure 3. Coloring Stage
54

55. TRICS System
Structure 3. Coloring Stage
Outlines
 Introduction
 Automatic
coloring in the
literature
 TRICS ‘Texture
Recognition
based Image
Coloring
System’
 Results
 Conclusion
 Future work
55

56. Results and Conclusion
Results
 Perfect Results Outlines
 Introduction
 Automatic
coloring in the
literature
 TRICS ‘Texture
Recognition
based Image
Coloring
System’
 Results
 Conclusion
 Future work
56

57. Results and Conclusion
Results
 Perfect Results
57

58. Results and Conclusion
Results
 PANN Database:
58

59. Results and Conclusion
Results
Misclassified results
2 classification levels:
•if the KNN results in 5 classes “grass, sea, water, grass, sea”
•The traditional solution is the class of the grass.
•The 2 levels classification solution is sea.
•(Sea and water), (trees and grass), (sky and clouds) and (wood
and stone) are considered as one class in level one.
59

60. Results and Conclusion
Comparisons
HSV/HSB
HIS/HLS
60

61. Results and Conclusion
Comparisons
 Local color transfer
Outlines
 Introduction
 Automatic
coloring in the
literature
 Global Image Matching  TRICS ‘Texture
Recognition
based Image
Coloring
System’
 Results
 Conclusion
 Future work
61

62. Results and Conclusion
Conclusion
 Outlines
We proposed a new computer coloring technique
that simulates the human vision in this area.
 Introduction
 Automatic
coloring in the
 The proposed coloring system is contributed for

literature
TRICS ‘Texture
coloring gray natural scenes. Recognition
based Image
Coloring
System’
 The execution time of TRICS is minimized using
 Results
Conclusion
Fast k-mean segmentation technique and the
 Future work
results are enhanced by splitting the disjoint regions
and by eliminating small regions.
62

63. Results and Conclusion
Conclusion
 Clusters number generation algorithm and the
minimum region size estimation algorithm
increase the professionalism of the system but
also increases the time of the execution. And by
using both of them TRICS becomes a fully
unsupervised intelligent recognition based
coloring system.
 HSV coloring model is very suitable for our
system and the coloring results have good
natural look.
63

64. Results and Conclusion
Conclusion
Outlines
 We consider our proposed system structure as
 Introduction
an abstract structure for building any more Automatic

coloring in the
intelligent coloring systems for any other types of
literature
images  TRICS ‘Texture
Recognition
based Image
Coloring
System’
 Our proposed system results perform the other
 Results
coloring systems.  Conclusion
 Future work
64

65. Future work
Gray image
Segmentation A Outlines
Features extraction Segmentation
(Joint, wavelets, laws,…) (Mean Shift, K-Mean,  Introduction
FCM,..)
 Automatic
Segmented image,
Clusters coloring in the
literature
Classification B  TRICS ‘Texture
Features extraction Classification
(Co-occurrence, Tamura, Wavelets energies) (K-NN classifier,..) Recognition
based Image
Samples features Coloring
Class labels System’
Classes Hues Database
 Results
 Conclusion
Coloring
 Future work
C
Convert image to Set Hue, Saturation, and Convert to
HSV channels Brightness RGB
65
Colored image

66. Future work
Intelligent System for Classifying the image
Gray image
Segmentation A
Features extraction Segmentation
(Joint, wavelets, laws,…) (Mean Shift, K-Mean,
FCM,..)
Segmented image,
Clusters
Classification B
Features extraction Classification
(Co-occurrence, Tamura, Wavelets energies) (K-NN classifier,..)
Samples features
Adaptive Learning Class labels SOFM
Classes Hues Database
Coloring
C
Convert image to Set Hue, Saturation, and Convert to
HSV channels Brightness RGB
66
Colored image

67. Future work
 Segmentation and classification stages
are research areas and any improvement
will increase the accuracy of the system.
 Using different types of features and
training set enables the system for
coloring images like manmade images,
indoors, and people photos .
67

68. List Of Publications
 Noura A.Semary, Mohiy M. Hadhoud, W. S. El-Kilani, and Nabil A.
Ismail, “Texture Recognition Based Gray Image Coloring”, The
24th National Radio Science Conference (NRSC2007), pp. C22,
March 13-15, 2007, Faculty of Engineering, Ain-Shams Univ.,
Egypt.
68

69. ‫…‪Thanks‬‬
‫و الحمد لله الذي بفضله‬
‫تتم الصالحات‬
‫96‬