This paper proposed a facial expression recognition approach based on Gabor wavelet transform. Gabor wavelet filter is first used as pre-processing stage for extraction of the feature vector representation. Dimensionality of the feature vector is reduced using Principal Component Analysis and Local binary pattern (LBP) Algorithms. Experiments were carried out of The Japanese female facial expression (JAFFE) database. In all experiments conducted on JAFFE database, results obtained reveal that GW+LBP has outperformed other approaches in this paper with Average recognition rate of 90% under the same experimental setting.

1. 1

2. M. Abdulrahman, T. R. Gwadabe, F. J. Abdu & A. Eleyan
Department of Electrical & Electronics Engineering
Mevlana University Konya, Turkey
Presented by
MUZAMMIL ABDULRAHMAN
2013

3.  Introduction
 Application
of FER
 Basic Steps of FER
 Principal Component Analysis
 Local Binary Pattern
 Gabor Wavelet Transform
 Classification
 Simulation Results
 Conclusion
 References
3

4. DEFINITION

Face Recognition (FR) can be described as classifying a
face either known or unknown, after comparing it with
known individuals stored in a database

Facial Expression Recognition (FER) system is a
computer application for automatically identifying or
verifying people’s emotions reflected on their faces from
a digital image or a video frame from a video source by
comparing it with database.
4

5. Facial recognition utilizes distinctive features of the
face such as distinct micro elements like: Mouth,
Nose , Eye, Cheekbones, Chin, Lips, Forehead, Ears
The distance between the eyes, the length of the nose,
and the angle of the jaw give rise to the type of
expression. Below are the 7 Facial Expressions types
Angry
Disgust
Fear
Happy
Neutral
Sad
Surprise
5

6. 
Human computer interaction
 Automated
 Video
access control
surveillance
6

7.  Games
 Security
 Patient
condition monitoring
7

8. FER involve the following steps:
 Face detection
 Facial expression data extraction
 Facial expression classification
The following algorithms can be used in a Holistic-based
approach to extract the facial expression features:
 Principal Component Analysis PCA
 Linear Discriminant Analysis LDA
 Local Binary Patterns LBP
 Discrete Wavelet Transform DWT
 Gabor Wavelet Transform GWT
 Discrete Cosine Transform DCT

9. 

The aim of the PCA is to reduce the
dimensionality of the raw data (features) while
retaining as much as possible of the variation
present in the dataset.
Speeds up the computational time.
9

10. PCA
10

11. 
The database
 a1 

÷
a2 ÷
=
M ÷

a 2 ÷
÷
 N 
 b1 

÷
b2 ÷
=
M ÷

b 2 ÷
÷
 N 
 a1 + b1 + L + h1 

÷
r 1  a2 + b2 + L + h2 ÷
m=
,
M
M ÷
MM

a 2 +b 2 +L+ h 2 ÷
÷
N
N 
 N
 c1 

÷
c2 ÷
=
M ÷

c 2 ÷
÷
 N 
 d1 

÷
d2 ÷
=
M ÷

d 2 ÷
÷
 N 
where M = 213
11

12. 
Then subtract it from the training faces
 a1 − m1 
 b1 − m1 
 c1 − m1 
 d1 − m1 

÷

÷

÷

÷
r  a2 − m2 ÷ r  b2 − m2 ÷ r  c2 − m2 ÷ r  d 2 − m2 ÷
am =
, bm =
, cm =
, dm =
,
M
÷
M
÷
M
÷
M
÷
M
M
M
M




a 2 − m 2 ÷
÷
b 2 − m 2 ÷
÷
c 2 − m 2 ÷
÷
d 2 − m 2 ÷
÷
N 
N 
N 
N 
 N
 N
 N
 N
 e1 − m1 

÷
r  e2 − m2 ÷
em =
,
M
M ÷

e 2 − m 2 ÷
÷
N 
 N

r 
fm = 




f1 − m1 
 g1 − m1 
 h1 − m1 
÷

÷

÷
f 2 − m2 ÷ r  g 2 − m2 ÷ r  h2 − m2 ÷
, gm =
, hm =
M
M
M
M ÷
M ÷
M ÷
÷


÷
g 2 − m 2 ÷
÷
h 2 − m 2 ÷
÷
f N 2 − mN 2 
N 
N 
 N
 N
12

13. 
Now we build the matrix which is N2 by M

The covariance matrix which is N2 by N2
r r r r r r r r
A =  am bm cm d m em f m g m hm 


Cov = AA
Τ
Find eigenvalues of the covariance matrix
The matrix is very large
The computational effort is very big
We are interested in at most M eigenvalues
We can reduce the dimension of the matrix
13

14. 

Compute another matrix which is M by M
Τ
L=A A
Find the M eigenvalues and eigenvectors
• Eigenvectors of Cov and L are equivalent
Build matrix V from the eigenvectors of L
Eigenvectors of Cov are linear combination of image space
with the eigenvectors of L

U = AV r
V is Matrix of
Eigenvectors
r
r r r r r r
A =  am bm cm d m em f m g m hm 


Eigenvectors represent the variation in the faces
14

15. A: collection of the
training faces
U: Face Space /
Eigen Space
Compute for each face its projection onto the face space
r
Ω1 = U ( am ) , Ω 2 = U Τ
r
Ω5 = U Τ ( em ) , Ω 6 = U Τ
Τ
r
r
bm , Ω3 = U Τ ( cm ) , Ω 4 = U Τ
r
r
f m , Ω 7 = U Τ ( g m ) , Ω8 = U Τ
( )
( )
r
dm ,
r
hm
( )
( )
15

16. 
To recognize a Facial Expression
 r1 
 ÷
r2
= ÷
M ÷
 ÷
r 2 ÷
 N 

Subtract the average face from it
 r1 − m1 

÷
r2 − m2 ÷
r 
rm =
M
M ÷

r 2 − m 2 ÷
÷
N 
 N
16

17. 
Compute its projection onto the face space U
r
Ω = U ( rm )
Τ
17

18. 
Different illumination
18

19. 


Different head pose
Different alignment
Different facial expression
19

20. The LBP operator was originally designed for texture
description. The operator assigns a label to every pixel of an
image by thresholding the 3x3-neighborhood of each pixel
with the center pixel value and considering the result as a
binary number.
233
=
224
150
200
173
185
120
128
20
1
T h re s h o ld
1
1
0
0
1
0
B in a ry : 1 0 0 0 1 0 1 1
D e c im a l: 1 3 9
0
20

21. 21

22. Uniform Pattern: An LBP is called uniform if the binary pattern contains at most
two bitwise transitions from 0 to 1 or vice versa when the bit pattern is
considered circular
Example
 The patterns 00000000 (0 transitions)
 01110000 (2 transitions)
are uniform
 11001111 (2 transitions)

The patterns 11001001 (4 transitions) and 01010011 (6 transitions) are not uniform.
Advantages of Uniform LBP
P
 Save memory: With a non uniform pattern there is
Possible combinations while for uniform LBP there are patterns of


2
P( P − 1) + 2
Uniform LBP detects only the important local textures like spots, edges
and corners
22

23. 





Divide the examined face image to cells
For each pixel in a cell, compare the pixel to each of
its neighbors. Follow the pixels along a circle, i.e.
clockwise or counter-clockwise.
Where the center pixel's value is greater than the
neighbor, write "1". Otherwise, write "0". This gives
an 8-digit binary number (which is converted to
decimal).
Compute the histogram, over the cell, of the
frequency of each "number“ occurring.
Optionally normalize the histogram.
Concatenate normalized histograms of all cells. This
gives the feature vector for the face image.
23

24. 


A GW filter is an essential tool used to extract local
features which can be applied on images to extract
features aligned at particular angles (orientations).
The GWs filter captures significant visual features such
as spatial localization, orientation selectivity, frequency
selectivity, and phase relationship
The GWs kernel can be defined by the following
equation:
1
ψ ( x, y,ϖ ,θ ) =
e
2
2πσ
X ' +Y ' 2
−(
)
2
iϖ X '
2σ
e
24

25. 
where (x,y) denote the pixel position in
the spatial domain , ϖ is the central
frequency of a sinusoidal plane wave, θ is
the orientation of the Gabor filter and σ is
the standard deviation along x and y
directions. The parameters and can be
defined by the following equations:
X ' = X cos θ + γ sin θ , γ ' = − X sin θ + γ cos θ
25

26. 26

27. 




Having an input image I(x,y) of size MxN and a Gabor
wavelets kernel of Ψ u ,v ( x, y,ϖ ,θ )
The Gabor feature representation is obtained by
convolving the input image with 40 Gabor wavelet
kernels given by
Ψ ,v ( x, y ) = I ( X , Y ) ∗Ψ ,v ( x, y ,ϖ, θ)
u
u
Concatenate the magnitude of the convolved output
images of all the 40 feature vectors for each input face
image
Optionally before concatenation each image output is
down-sample by a factor of 16 or 32 and normalized to
zero mean and unit variance.
Apply Any dimensionality Reduction Algorithm to
reduce the size of the feature vector.
27

28. 
Gabor Wavelet Transform posses many properties
which make them attractive for many applications.
 Directional selectivity
 Invariance to shifts and rotations
 Insensitive and robust to facial expression changes
 Insensitive to illumination variations

Despite many advantages of Gabor wavelet based
algorithms in face recognition, it has major
disadvantages.
 High computational complexity
 High memory capacity requirement
 Feature vectors dimensions are extremely large
28

29. 
Compute the Euclidian distance in the face
space between the test face and all faces in
the Training data
ε = Ω − Ωi
2
i

2
for i = 1.. M
The expression with the minimum distance
from Test face to the Training will be
matched as the best expression of the Test
face.
29

30. JAFFE facial expression database was used to conduct our
experiments.
It contains 213 images of 10 different females each with 7
expressions posed by 3 or 4 examples of each of the seven facial
expressions under different illumination and head position.
The images are of the size 256x256
Each original image has been aligned by normalizing it.
A total of 137 images (64%) were used as training data, while
the remaining 76 images(36%) as testing data
The K-nearest neighbour, Euclidean distance (L2) was used as a
similarities measure to classify the facial expressions images.

31. 31

32. FRR(%) Comparisons For Different FER
Technique Using JAFFE
Experiment
PCA [1]
PCA+LDA [1] ASM & HMM [2] LBP [3]
SVM [4]
PCA NMF LNMF[5]
Recognition
80.00
95.11
88.79
85.57
94.5
63.25 65.50 64.50
128x96
230X250
64X64
44X32
Rate (%)
Face Dimension 128x96
40X30
32

33. 




Gabor wavelets were used as a pre-processing stage followed
by dimensionality reducing using PCA/LBP for facial
expression recognition in this paper.
Experimental evaluations the proposed approach were
conducted on JAFFE database.
The results obtained showed that pre-processing with Gabor
wavelets improves the performance of directly applying both
PCA and LBP.
Also the variation in illumination, hair and head position affect
the facial recognition rate.
Facial expression recognition proposed in this paper has an
improved performance when compared with the previous
works using different algorithms using the same JAFFE
database as seen in tables.
33

34. 




[1] H. Deng, L. Jin And L. Zhen, “A New Facial Expression
Recognition Method Based On Local Gabor Filter Bank And PCA
Plus LDA”, International Journal Of Information Technology Vol. 11
No. 11 2005, pp. 93
[2] W. Zhao And J. Zhang, “Using ASM-Optical Flow Method And
Hmm In Facial Expression Recognition”, IERI International
Conference On Affective Computing And Intelligent Interaction,
Lecture Notes In Information Technology, Vol.10, 2012 Pp. 268.
[3] S. Liao, W. Fan And D. Yeung, “Facial Expression Recognition
Using Advanced Local Binary Patterns, Tsallis Entropies And Global
Appearance Features”, IEEE, 2006 pp. 668.
[4] A. Bouzerdoum, S.L. Phung And P. Li, “Feature Selection For
Facial Expression Recognition”, IEEE, 2nd European Workshop On
Visual Information Processing USA, 2010 pp. 39
[5] I. Buciu And I. Pitas, “Application Of Non-Negative And Local Non
Negative Matrix Factorization To Facial Expression Recognition”,
IEEE Proceedings Of The 17th International Conference On Pattern
Recognition , 2004 1051-4651.
34

35. 35