Identify Defects in Gears Using Digital Image Processing
Final Thesis Presentation Licenseplaterecognitionincomplexscenes
1. Department of Electrical and Computer Engineering
Illinois Institute of Technology, Chicago
Masters’s Thesis Defense Presentation
2011 November 23
Dhawal S Wazalwar
Examination Committee
Dr Erdal Oruklu (advisor)
Dr Jafar Saniie
Dr Tricha Anjali
2. Introduction to license plate recognition system
Background of previously implemented systems
• A robust flow for license plate recognition
• License plate detection
• Character segmentation
• Feature extraction and character recognition
Experimental results
Conclusion
Further improvements
3. A rapidly evolving technology used in surveillance
systems like management of road traffic, speed checks,
vehicle parking etc.
Types
• Fixed camera systems
• Example:- Vehicle tracking at toll centers
• Mobile camera systems
• Example: - Camera mounted on police vehicles, which
constantly scan the license plates in all directions, during patrol
operation.
4. Image Frame Optical Character License Plate
Acquisition Processing Recognition Image Database
Focus is on improving license plate detection and
recognition performance for complex scenes.
5. Binary level processing
• Edge detection and mathematical morphological operations are
used to detect license plate information.
• Faster execution speed but sensitivity to unwanted edges is an
issue.
Gray level processing
• It uses local contrast information and thresholding techniques to
differentiate between all possible object region.
Color level processing
• Unique color information of the license plate characters and its
background details is used. These methods have more accuracy
but relatively slow in exceution.
• Greater accuracy but the huge amount of color data needs to be
processed, so slow in comparison with previous two methods.
6. Horizontal and vertical projection data based techniques
• Image column and row data is added to obtain a vector.
Binary algorithms and morphology based techniques
• Connected component analysis along with object measurements
like height, width, area is used.
Contour based techniques
• These techniques are based on shape descriptors like curvature,
gradient information.
Gray level histogram techniques
• Histogram valley is used for thresholding purposes, which is a key
aspect of character segmentation
7. Feature vector
• Uniqueness of feature decides the character recognition
performance.
• Horizontal and vertical projection data, region properties like area,
perimeter, orientation are commonly used features.
• Concept of stroke direction and elastic mesh showed an accuracy
of about 97.8-99.5 %
• Selection of Classifiers
• Statistical classifiers like support vector machine, Markov models
have been widely with success rate of about 95-99 %.
• Various architectures of artificial neural networks, mostly working
on binary data have been proposed.
8.
9. The efficiency of any license plate recognition
system depends on how accurately we can detect
the license plate location.
Key steps in the process
• Edge detection: The main objective is to obtain clear and
distinct edges of license plate with minimal noise content.
• Morphological dilation operation: This step helps to thicken
the edges obtained after the edge detection step.
• Region growing segmentation: It basically divides the entire
region into small segments based on pixel connectivity
information.
• Criterion for license plate selection: The main aim here is to
determine the most optimal criteria for identifying license
plate region after region growing segmentation.
10. Simple edge detection techniques using Prewitt, Sobel
operator work well for cases where there exists a clear
and distinct contrast between the license plate and
region surrounding it.
However, for cases where there exists a partial contrast
between license plate region and surrounding part, more
complex edge detection using Mexican hat operator is
required.
Mexican hat operator first performs smoothing operation
and then extracts edges. Mathematically represented as
r2
r 2 2 2 2
h r
2
4
e
11. Case 1: Clear and distinct contrast
Edge detection using
Example Image 1 Prewitt operator
Case 2: Partial Contrast
Edge detection using Edge detection using
Example Image 2 Prewitt operator Mexican Hat operator
12. Dilation is considered as local maximum operation used
to add pixel information to boundaries in an image.
It is mathematically represented as
A⊕B = { z | B ∩ A ≠ Φ}
where A is a binary image and B is the structuring
element used.
The size and nature of structuring element is important
since smaller size can negate the desired effect and
larger size can cause the license plate regions to be
connected to the surrounding area.
For our algorithm, best results were achieved with 3x3
ones matrix as a structuring element.
13. Blur license plate image Edge detection output Detection result without
dilation
Dilation operation output Detection result with
dilation
14. For detecting a license plate, we first need to come up
with all possible plate region. Region growing with pixel
connectivity information is used to divide entire image
into several sub-regions.
• The performance of region growing depends on
selection of
• Type of pixel connectivity information used
• Starting point for algorithm, seed point selection
• Stopping rule selected for the implementation
15. 4-pixel connectivity 8-pixel connectivity
8-pixel connectivity will have better performance, however,
computation time required will be more.
18. For every bright pixel in the binary output after dilation,
its neighboring four pixel connectivity information was
checked and region was grown based on this
information.
In order to speed up the process, sub-sampling was
used, where algorithm operation was performed for
every two pixels, instead of performing for each and
every pixel in the image.
A lower and upper bound was defined to remove the
unwanted regions (non license plate regions).
19. Before region Output after region
growing step growing
20. The main aim here is to determine the most optimal
criteria for identifying license plate region after region
growing segmentation.
Criterion 1: License plate dimensions
• Easy to implement
• Disadvantage is that the algorithm becomes extremely sensitive
to license plate dimensions.
Criterion 2: Euler number of a binary image
• Euler number of a binary image gives the number of objects
minus the closed curves (holes).
• Experimentally, it was found that in alphanumeric regions, value
of Euler number is zero or negative, while for other regions it has
a positive value.
• Algorithm may fail if vehicle image has alphanumeric characters
in more than one region.
21. Criterion 3 : Combination of Euler number of a binary
image and license plate dimensions
• A combined approach of using Euler number criterion as well as
license plate dimensions helps to overcome problem raised in
case of Criterion 2
Output using Criterion 2 Output using Criterion 3
22. After locating the license plate, the next important
step is to segment each character individually,
avoiding any possibility of joint characters.
Key steps in the process
• Image pre-processing: In this step, we apply median filter
followed by contrast enhancement operation.
• Thresholding: In this we apply local thresholding algorithm
which converts grayscale license plate into binary image for
further processing.
• Erosion operation: This operation basically helps to remove
dirt/shadow present in the license plate image.
• Region growing segmentation
23. Original Image Gaussian noise
Salt and pepper noise Speckle noise
24. Median filter is a non-linear spatial filter and is used
widely for image denoising.
In median filtering, we replace the original pixel data by
median of pixels contained in a prescribed window.
Median extremely useful because it important
characteristics of the image like boundaries, edges.
25. If the image acquired is blurred, it will be difficult to
process it further, since most of its character information
is not properly visible.
It was found that the character information for most of
the images in database is usually in the intensity range
of 0-0.4 on a scale of 1.
In order to obtain further contrast enhancement, we
increased the dynamic range for pixels in this (0-0.4)
intensity range.
27. Its basically a segmentation technique used to separate
object and background information.
Global thresholding
• A single threshold value is applied for entire image.
•
g ( x, y) 1, otherwiseT
0,
f ( x, y )
• It is easy to implement, however single dark or bright region can
corrupt the threshold value.
Local thresholding
• The entire image is divided into several parts by using windowing
operation and then perform the thresholding for each case
separately.
For our algorithm, we have used Otsu’ threshold method. In this
method, gray level histogram is used to determine the optimal
threshold value for which the intra class variance between the two
classes is minimal.
29. In cases where there exist dirt/shadow on license plate,
local thresholding may not be sufficient all the time.
Erosion operation is required to remove the dirt/shadow
part.
Erosion operation mathematically can be represented
as
X ΘB
x | B x X
The structuring element is selected such that character
information remains intact and only the noise part is
removed.
30. Original Image Local threshold operation
Output after erosion operation Segmented characters after
region growing
31. Two stage process
• Firstly we have to extract key features associated with the
characters.
• Secondly, a mechanism has to be defined to recognize these
characters based on the extracted features.
Feature extraction
• The robustness of character recognition algorithm depends on
how well it can recognize different versions of a single character.
• Algorithm should be able to distinguish between characters like D
and 0 or 5 and S because of the similarity in their appearance as
well as spatial information.
33. Feature 1: Region properties like area, perimeter,
orientation, minor-major axis length.
Feature 2: Projection information about X and Y axes
Feature 3: Some of the characters like 0, M, N are
symmetric about X-Y axes. This information can be used
to distinguish different characters especially 0 and D.
This symmetry information can be quantified by a
coefficient called as reflection symmetry coefficient.
34. Calculation
• Consider Cin as the original character matrix where information is
stored in binary format. The center of matrix is considered as the
origin. Cx and Cy are the transformed matrix obtained after
flipping about the X and Y axes respectively.
• Calculate the area of original matrix Cin
• Determine the union of transformed matrixes Cx and Cy with
original matrix Cin
• Calculate the areas of regions (Cin ∪ Cx) and (Cin ∪ Cy)
• The reflection coefficient can be determined as
Area Cin
Xcoefficient
Area Cin C x
Area Cin
Ycoefficient
Area Cin C y
35. A mathematical model called artificial neural network
based on interconnection of artificial neurons, which can
be trained by flow of external and internal information
was developed.
Extensively used for solving pattern classification,
clustering, function approximation.
Classification
• Feed forward neural networks
• Recurrent neural networks
36.
37.
38. Input layers to the neural network are the feature vectors
extracted for the all the characters. The overall size is
decided by the character set and the feature vector size.
Output nodes are the set of all possible characters that
are present in license plate information. The number of
output layers for our implementation is 33.
There is no generic rule as such for deciding the number
of hidden layers. In our design, the number of hidden
layers was experimentally found to be 25.
We obtained a set of 50 images for each character from
the database images and trained the neural network
using them.
39. Samples of character ‘5’
Samples of character ‘S’
If the characters are possibly between 5, 0, S and D,
then region properties like orientation, reflection
coefficient were again used in the 2nd stage
identification process and the final possible character
was identified
40. The proposed algorithm was tested on images with
varying illumination conditions taken during day as well
as night times, at different angles and difficult shadows.
Entire database was divided in following types:-
• Type 1: In these samples, license plate is fully visible and all its
characters are clear and distinct.
• Type 2: Images in these samples are little blurred due to
variations in illumination conditions.
• Type 3: These samples have little dirt or shadows on or near
license plate region.
• Type 4: In these samples, another text block apart from license
plate is present in the frame.
• Type 5: Plate details are present in two rows
41. License plate localization video demo
Source: CitySync single lane PAL demo video,
http://www.citysync.co.uk/
43. Plate Character
Image Total images
detection recognition
type tested
accuracy accuracy
Type 1 879 99.4% 97.81%
Type 2 442 96.3% 96.63%
Type 3 153 94.77% 95.69%
Type 4 44 100% 99.25%
Type 5 11 100% 97%
Overall Results 1529 98.1% 97.05%
44. OpenCV execution
MATLAB execution
Task time
time
Edge detection 0.69s 0.4s
Dilation 0.05s 0.03s
Segmentation 0.384s 0.3s
Total 1.13s 0.731s
45. Algorithm fails when license plate is black in color
Image with license plate Region growing output fail
black in color to detect license plate
46. The main objective is to segment the individual
characters and extract appropriate features from it to
recognize them.
A Neural network based approach is then used to train a
network using these feature vectors.
Our algorithm is currently able to recognize license plate
characters with an accuracy of 90-93%.
47. The algorithm presented in this paper can be extremely
useful in dealing with complex cases, while building a
real time license plate detection system.
The use of Mexican hat operator helps to improve the
performance of edge detection step, when there is only
partial contrast between license plate region and region
surrounding it.
Euler number criterion for a binary image helps to
reduce the sensitivity of algorithm to license plate
dimensions.
48. Preprocessing step using median filter and contrast
enhancement ensures performance in case of low
resolution and blurred images.
Reflection symmetry coefficient along with two stage
identification process improves the character recognition
performance significantly especially in dealing with
complex cases like recognizing 0 and D.
Local threshold operation prevents a single brighter or
darker region from corrupting the threshold value and
thereby improves binarization process.
49. Algorithm can be extended to mobile license plate
recognition systems.
Hardware implementation in FPGA
