1. Eye Tracking Techniques and
applications
eie426-EyeTracking-200903.ppt
03/06/13 EIE426-AICV 1

2. Outline
 Part I :Physiology of the EYE
 Part II : Mechanism of Eye Movements
 Part III :Eye Tracking Technology
 Part IV :Eye Tracking Applications
03/06/13 EIE426-AICV 2

3. How does our eye work
03/06/13 EIE426-AICV 3

4. Part I :Physiology of the EYE
HOW DOES OUR EYE WORK
03/06/13 EIE426-AICV 4

5. 03/06/13 EIE426-AICV 5

6. Part II : Mechanism of Eye Movements
CAN YOU BELIEVE YOUR EYE
03/06/13 EIE426-AICV 6

7. Part III :Eye Tracking Technology
HOW EYE TRACKER WORKS
03/06/13 EIE426-AICV 7

8. Outline
1. Current State and Application of Eye Gaze Tracking
(EGT) Technology
2. Theory and the Classification of EGT Technology
3. The Framework of EGT system

9. Ⅰ. Current State and Application of Eye
Gaze Tracking (EGT) Technology

10. Ⅰ. Current State and Application of Eye Gaze Tracking (EGT) Technology
 What is EGT
eye gaze——The line from the fovea through the center of
the pupil is the line of sight (LoS). Usually, we
take the optical axis as line of gaze (LoG). LoG
can be approximate to LoS. In fact, LoS
determines a person’s visual attention.
eye gaze Tracking——By image Processing,if the LoG or LoS
can be estimated, the point of regard (POR) is
computed as the intersection of the LoG (or LoS)
with the object of the scene or space.
 History and Development of EGT
Interests of Visual attention can be traced back to 1897. At that time, it was a kind of
diagnostic research, i.e. the recording of eye movement. Those technologies include
ophthalmometer, Mechanical Method, Electro-Oculography (EOG), Optical-based
Method , electromagnetic Oculography
eye movement initially applied in medical research, such as brain and physiology
analysis. with the development of Electronics, Computer and Image processing
technology, further research focuses on Eye Gaze Tracking Technology .

11. Search Coil

12. A picture wearing the EOG apparatus

13. Ⅰ. Current State and Application of Eye Gaze Tracking (EGT)
Technology
 eye tracking techniques ：
1. Direct Observation ： ophthalmometer, peep-hole method
2. Mechanic Method ： use level to record eye movement
3. Optical-based Method ： Reflection Method （ mirror or prism ）、 Pupil-
Corneal reflection 、 Purkinje Image
4. Electro-Oculography (EOG): recorded the difference in the skin potential
5. Electromagnetic Oculography ： The users gaze is estimated from
measuring the voltage induced in the search coil by an external electro-magnetic
field
Except Optical-based Method, the above methods are more or less low accuracy or
high intrusiveness . As a result, Modern Eye Gaze Tracking Techniques are mostly
based on Optical-based Method.

14. Ⅱ. Theory and the Classification of
EGT Technology

15. Ⅱ. Theory and the Classification of EGT Technology
 Image-Based EGT Technology
1. Infrared Light : out of visible light; Paired with filter to eliminate the
light of other wavelength ；
2. Eye camera :tracking eye movement and recording eye image
sequence
3. Image Processing: detecting the visual elements
4. Further Estimation: 。 Having pre-processed, the data of eye gaze
can be get from eye-movement model

16. Ⅱ. Theory and the Classification of EGT Technology
System Framework
Input
Image Image Feature
acquisition Estimation
Processing
Output
Data Eye
Estimation Tracking Calibration

17. Ⅱ. Theory and the Classification of EGT Technology
Function of Each Module
Image acquisition:
1 get bright pupil or dark pupil
Image Processing :
2 Filtering ,Noise reducing ,difference ,thresholding
Feature Estimation:
3 Estimating the center of Pupil and corneal reflection
Eye Tracking:
4 Estimation of eye gaze
Data Estimation:
5 Algorithm validation ,eye movement data analysis

18. Theory of EGT Technology
 2D to detect the Optical Axis (basic)
 2D to detect the Line of Sight (Advanced)
¹â Öá O A 2 D ·½ ·¨
ÊÓ Ïß Öá L O S 3 D ·½ ·¨

19. Classification of EGT Technology
view of i n t r u s i v e £º C o n t a c t l e n s ¡¢ P h o t o - t r a n s m i t e r ¡¢ I R l i g h t e m i t t i n g d i o d e s
humanity n o n - i n t r u s i v e £º u s e i m a g e - b a s e d M e t h o d
D i a g n o s t i c £º U s e t h e E y e m o v e m e n t c h a r a c t e r s t o P s y c h o l o g y a n d N e u r o s c i e n c e
view of ( n e e d to d is t in g u is h d if f e r e n t t y p e o f e y e m o v m e n t )
application I n t e r a c t i v e £º U s e E y e M o v m e n t t o d e c i d e P o i n t o f R e g a r d , u s e f o r i n t e r a c t i o n
£¨ n e e d t o k n o w t h e v i s i u a l a t t e n t i o n £©
B o d y - M o u n t e d £º g o g g l e ¡¢ h e l m e t ¡¢ b a c k p a c k , e t c
n o t e : a l t h o u g h l a b e l e d w i t h i n r u s i v e ,b u t m o r e a n d m o r e b o d y - M o u n t e d s y s t e m s
are b e c o m in g n o n -in tru s ive
View of
appearance
R e m o t e s y s t e m £º R e m o t e E Y E G a z e T r a c k e r ( R E G T )

20. Ⅱ. Theory and the Classification of EGT Technology
From View of Humanity
Usually By goggle , helmet …
Head
features ： high accuracy
Mounted
Eye Tracker defects ： intrusiveness
Usually on table
Remote features ： non-intrusive
Eye Tracker defects ： eye and head
relative motion

21. Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
The Purkinje images are reflections created at different
Purkinje Image layers of the eye structure. Eye gaze can be calculated
from these relative positions of these reflections
By placing electrodes around the eye, it is possible
EOG to measure small differences in the skin potential
corresponding to eye movement.
The Limbus is the boundary between the white sclera and the dark
Limbus-Scalar iris of the eye. By placing IR light emitting diodes and photo-
transistors, respectively, above and below the eye. the resulting
IR Tracking voltage difference is proportional to the angular deviation of the eye.
Training images are taken when the user is looking
ANN at a specific Calibration markers. Use ANN to decide
the eye gaze
Contact Lens Use a small coil embedded into a contact lens that is tightly fit
over the sclera . The user’s gaze is estimated from measuring the
voltage induced in the search coil by an external electro-magnetic
field.
Pupil-Corneal The IR source can generate a glint on corneal and
divide pupil from iris, the difference between can
reflection represent the eye gaze movement

22. Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
ThePurkinje images are reflectionsreflections
The Purkinje images are created at different
Purkinje Image layers of the eye structure. Eye gaze can be calculated
created at different layers of the eye
from these relative positions of these reflections
structure. Eye gaze can be
将两对氯化银皮肤表面电极分别置于眼睛左右、上下
EOG calculated from thesein the skin is possible
By placing electrodes around the eye, it
两侧 , 就能引起眼球变化方向上的微弱电信号 , 经放
to measure small differences
relativepotential
positions ofto eye movement.
corresponding these reflections
大后得到眼球运动的位置信息
The Limbus is the boundary between the white sclera and the dark
Limbus-Scalar The Limbus is the boundary between the white
iris of the eye. By placing IR light emitting diodes and photo-
transistors, the dark iris of the eye. This eye. the resulting
sclera andrespectively, above and below theboundary can
IR Tracking voltage be opticallyproportional to the angular deviation of the eye.
easily difference is detected and tracked
Training images are taken when the user is looking
ANN at a specific Calibration markers. Use ANN to decide
the eye gaze
Contact Lens Use a small coil embedded into a contact lens that is tightly fit
over the sclera . The user’s gaze is estimated from measuring the
voltage induced in the search coil by an external electro-magnetic
field.
Pupil-Corneal The IR source can generate a glint on corneal and
divide pupil from iris, the difference between can
reflection represent the eye gaze movement

23. Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
The Purkinje images are reflections created at different
Purkinje Image layers of the eye structure. Eye gaze can be calculated
from these relative positions of these reflections
By placing electrodes around the eye, it is possible
EOG to measure small differences in the skin potential
corresponding to eye movement.
Limbus-Scalar The Limbus is the boundary between the white
sclera and the dark iris of the eye. This boundary can
IR Tracking easily be optically detected and tracked
Training images are taken when the user is looking
ANN at a specific Calibration markers. Use ANN to decide
the eye gaze
Contact Lens Use a small coil embedded into a contact lens that is tightly fit
over the sclera . The user’s gaze is estimated from measuring the
voltage induced in the search coil by an external electro-magnetic
field.
Pupil-Corneal The IR source can generate a glint on corneal and
divide pupil from iris, the difference between can
reflection represent the eye gaze movement

24. Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
The Purkinje images are reflections created at different
Purkinje Image layers of the eye structure. Eye gaze can be calculated
from these relative positions of these reflections
By placing electrodes around the eye, it is possible
EOG to measure small differences in the skin potential
corresponding to eye movement.
Limbus-Scalar The Limbus isis the boundary betweenwhitewhite
The Limbus the boundary between the the
sclera and the dark iris of thethe eye. By placing IR
sclera and the dark iris of eye. This boundary can
IR Tracking light emitting diodes andand tracked
easily be optically detected photo-transistors,
respectively, above and below the eye. the
将两对氯化银皮肤表面电极分别置于眼睛左右、上下两
Training images are taken when the user is looking
ANN resulting voltage difference is proportional to
at a, specific
侧 就能引起眼球变化方向上的微弱电信号 , 经放大后
the angular Calibration of the eye. ANN to decide
得到眼球运动的位置信息 deviation markers. Use
the eye gaze
Contact Lens Use a small coil embedded into a contact lens that is tightly fit
over the sclera . The user’s gaze is estimated from measuring the
voltage induced in the search coil by an external electro-magnetic
field.
Pupil-Corneal The IR source can generate a glint on corneal and
divide pupil from iris, the difference between can
reflection represent the eye gaze movement

25. Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
The Purkinje images are reflections created at different
Purkinje Image layers of the eye structure. Eye gaze can be calculated
from these relative positions of these reflections
By placing electrodes around the eye, it is possible
EOG to measure small differences in the skin potential
corresponding to eye movement.
Limbus-Scalar The Limbus is the boundary between the white
sclera and the dark iris of the eye. This boundary can
IR Tracking easily be optically detected and tracked
Training images are taken when the user is looking
ANN at a specific Calibration markers. Use ANN to decide
the eye gaze
Contact Lens Use a small coil embedded into a contact lens that is tightly fit
over the sclera . The user’s gaze is estimated from measuring the
voltage induced in the search coil by an external electro-magnetic
field.
Pupil-Corneal The IR source can generate a glint on corneal and
divide pupil from iris, the difference between can
reflection represent the eye gaze movement

26. Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
The Purkinje images are reflections created at different
Purkinje Image layers of the eye structure. Eye gaze can be calculated
from these relative positions of these reflections
By placing electrodes around the eye, it is possible
EOG to measure small differences in the skin potential
corresponding to eye movement.
Limbus-Scalar The Limbus is the boundary between the white
sclera and the dark iris of the eye. This boundary can
IR Tracking easily be optically detected and tracked
Use a small coil taken when the usera contact
Training images are embedded into is looking
ANN lens that is tightly fit over the sclera decide
at a specific Calibration markers. Use ANN to . The
user’s gaze is estimated from measuring
the eye gaze
theavoltageembedded into a contact lens that coil byfit
induced in the search is tightly
将两对氯化银皮肤表面电极分别置于眼睛左右、上下
Contact Lens Use small coil
an the sclera . electro-magnetic field. , 经放
两侧 , 就能引起眼球变化方向上的微弱电信号
over externalThe user’s gaze is estimated from measuring the
大后得到眼球运动的位置信息
voltage induced in the search coil by an external electro-magnetic
field.
Pupil-Corneal The IR source can generate a glint on corneal and
divide pupil from iris, the difference between can
reflection represent the eye gaze movement

27. Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
The Purkinje images are reflections created at different
Purkinje Image layers of the eye structure. Eye gaze can be calculated
from these relative positions of these reflections
By placing electrodes around the eye, it is possible
EOG to measure small differences in the skin potential
corresponding to eye movement.
Limbus-Scalar The Limbus is the boundary between the white
sclera and the dark iris of the eye. This boundary can
IR Tracking easily be optically detected and tracked
Training images are taken when the user is looking
ANN at a specific Calibration markers. Use ANN to decide
the eye gaze
Contact Lens Use a small coil embedded into a contact lens that is tightly fit
over the sclera . The user’s gaze is estimated from measuring the
voltage induced in the search coil by an external electro-magnetic
field.
Pupil-Corneal The IR source can generate a glint on corneal and
divide pupil from iris, the difference between can
reflection represent the eye gaze movement

28. Ⅱ. Theory and the Classification of EGT Technology
EGC Technology Accuracy Features
High accuracy, but the light is
Purkinje Image 0.017 0 hard to control and can only be
used in Lab
low robustness, low accuracy,
EOG 1.50-2° high intrusiveness
V 1 0 -7 0 Horizontal accuracy is better
Limbus Tracking than Vertical, but both are low
H 0.5 0 -7 0
No need of calibration,low
ANN 1.3-1.8° accuracy,
high accuracy, high
Contact Lens 0.08 0 intrusiveness
Pupil Corneal Reflection 1° The best one till now

29. Ⅱ. Theory and the Classification of EGT Technology
Feature of Eye Movement
D ia g n o s t ic ( Õï ÁÆ Ê½ ) £º ÊÓ Ïß ×· ×Ù µÄ ½á ¹û ×÷ Ϊ Ò» ¸ö ¶¨ Á¿ ÌØ ÐÔ £¬ ¿¼ ²ì ÈË µÄ ÊÓ ¾õ ºÍ ÐÄ Àí ѧ ÑÐ ¾¿
 view of
 从系统
se C
£¨ ¾ø ¶Ô ×¼ È· ÐÔ £¬ ¶Ô ÑÛ Çò ¶¯ ×÷ µÄ Çø ·Ö £©
ine rt ar cu t iv iv (e½» £º»¥ ʽ o) £ºn tÖ»a cÒªt ×·le ×Ù s ³ö¡¢ ÊÓ P Ïßh £¬ t ×÷ - Ϊt r ºóa n ¶Ës m½» it»¥ e µÄ ¡¢Êä IÈëR lig h t e m it t in g d io d e s
n o o r
In t
humanity
应用角度 n o n - in t r u s iv e £º£¨ Ïàu s¶Ôe ×¼imÈ· a £¬g eÒ»- °ãb a²» s Çø d ·Ö MÑÛ eÇòt hµÄo ÔË ¶¯ ÌØ ÐÔ £¬ Ö» Òª Öª µÀ ÊÓ Ïß ·½ Ïò £©
e d
T a x o n o m y a n d M o d e ls o f E y e M o v e m e n ts
Saccades
F ix a tio n a n g le : 1 o ~ 4 o S m o o t h P u r s u it
d u r a t io n :1 0 0 ~ 2 0 0 m s d u r a t io n £º 3 0 ~ 1 2 0 m s v o le c it y £º 1 o / s ~ 3 0 o / s
v o le c it y :4 0 0 o / s ~ 6 0 0 o / s

30. Ⅱ. Theory and the Classification of EGT Technology
Development
 Early stage : Direct Observation ,Mechanical Method
 Initial Methods, not used any more for high intrusiveness and poor accuracy
 Development ： EOG 、 Electromagnetic Oculography
Although have improved a lot and widely used , these method are disappearing because of the
intrusiveness
 Advanced ： Optical Method ：
Because of its’ high accuracy and low intrusiveness, Optical Method have made rapid progress
in recent years.
1. The Purkinje images are reflections created at different layers of the eye structure. and the
eye gaze can be calculated from these relative positions of these reflections
2. Photo-Oculography : it measures the eye movement during it's translation/rotation. with the
IR light source, shape of the pupil ,Limbus or corneal reflection is detected
3. Limbus-Scalar IR Method: They place IR light emitting diodes and IR light sensitive photo-
transistors, respectively, above and below the eye. Several such IR pairs can be mounted
on goggles or helmets, a photo-transistor transforms the reflected IR light into a voltage.
the resulting voltage difference is proportional to the angular deviation of the eye.
4. Pupil-Corneal Reflection: The IR source can generate a glint on corneal and divide pupil
from iris, the camera can extract the pupil which represent the eye gaze, on the other hand
the corneal reflection represent the head motion, as a result, the difference between them
represents the real eye gaze movement. 。

31. Ⅱ. Theory and the Classification of EGT Technology
Feature of Remote Eye Gaze Tracker
B o d y - M o u n t e d £º g o g g l e ¡¢ h e l m e t ¡¢ b a c k p a c k , e t c
n o t e : a l t h o u g h l a b e l e d w i t h i n r u s i v e ,b u t m o r e a n d m o r e b o d y - M o u n t e d s y s t e m s
are b e c o m in g n o n -in tru s ive
View of
appearance
 Robustness R e m o t e s y s t e m £º R e m o t e E Y E G a ze T r a c k e r (R E G T )
 Light Source
 Eyelash, Eyelids and Camera Position
 System Error
 Eye station (dry or wet)
 head motion
 Lower accuracy than
Head Mounted Eye Tracker

32. Ⅲ A Practical Eye Gaze
Tracking System

33. Ⅲ A Practical Eye Gaze Tracking System
Framework of EGT System
 Hardware ： a Pair of Cameras
 eye camera
 scene camera
 eye camera is for acquisition of pupil corneal reflection image,
scene camera is for: 1 mapping from the eye image coordinate to scene image coordinate ;2
Showing the combining result of POR and scene in same image ：
M o r n ite r O u tp u t o f S c e n e
V id e o S tr e a m
V is u a l S t im u lu s
E ye C a m era
IR S o u rc e
S cene C a m era
T r a c k in g o f
P u p il A u to
-C o rn e a l M a p p in g
R e fle c t io n
O u tp u t o f E y e IR S o u rc e
O p tic a l S y s te m
V id e o S tr e a m
P C

34. Ⅲ A Practical Eye Gaze Tracking System
appearance of EGT System

35. Ⅲ Practical Eye Gaze Tracking System
Theory of a Head Mounted Eye Tracker
Capture Estimate Coordinate Eye
Eye Camera Pupil-Corne Pupil-Corneal Mapping Tracking
al image reflection Result
22222222222
Calibration
222222222D
Mapping between Eye
xianglian
Coordinate to Scene Tracking
Coordinate
Scene Camera

36. Ⅲ Practical Eye Gaze Tracking System
Bright Pupil and Dark Pupil
Bright Pupil
Dark Pupil
Difference
Image
 When IR source is placed near the optical axis of the camera
bright pupil as can be seen ;while IR is placed off the optical
axis, a dark pupil can be seen. By Thresholding, a robust
pupil contour can be extracted .

37. Ⅲ Practical Eye Gaze Tracking System
Bright Pupil and Dark Pupil
Bright Dark Pupil
Pupil
Corneal Reflection
The IR source can generate a glint on corneal and divide
pupil from iris, the camera can extract the pupil which
represent the eye gaze, on the other hand the corneal
reflection represent the head motion, as a result, the
difference between them represents the real eye gaze
movement.

38. Ⅲ Practical Eye Gaze Tracking System

39. Ⅲ Practical Eye Gaze Tracking System

40. Ⅲ Practical Eye Gaze Tracking System

41. Part IV :Applications
HOW FAR YOU CAN FLY
03/06/13 EIE426-AICV 41

42. Applications
 Human Computer Interaction (HCI)—— efficiency, humanity
 Intelligent Control —— EGT and Weapon Control
 Human Movement Study ——Typing, physical training
 Psychology —— Antinational Neuroscience,
 Visual attention & Driving ——Aviation, navigation, driving,
traffic accidents inspection
 Scene and Image Perception—— Web, AD, Designing, Scene
03/06/13 EIE426-AICV 42

52. Most Recent Application
03/06/13 EIE426-AICV 52