Gaze-Net - 11th Augmented Human International Conference. Manitoba, Canada. [Presented Online]

1. Gaze-Net: Appearance-Based Gaze
Estimation using Capsule Networks
Bhanuka Mahanama(@mahanama94)
Yasith Jayawardana (@yasithmilinda)
Sampath Jayarathna (@openmaze)
Department of Computer Science
Old Dominion University

2. Gaze-Net: Appearance Based Gaze Estimation@NirdsLab
Outline
● Introduction
● Related work
● Approach
● Proposed Architecture
● Experiments and Results
● Conclusion
2/11

3. Gaze-Net: Appearance Based Gaze Estimation@NirdsLab
Introduction
● Gaze Estimation Applications
○ Physiological studies
○ Human-computer interaction
● Modern methods
○ Convolution Neural Networks
○ Facial Region
○ Ocular Region
3/11
Appearance based-multi user eye tracking
(https://mgaze.nirds.cs.odu.edu/)

4. Gaze-Net: Appearance Based Gaze Estimation@NirdsLab
Related Work
● Estimation methods
○ Fixed head-pose - early methods (Sewell et al.[2010])
○ Variable head pose
■ Explicit pose data (Zhang et al.[2015])
■ Implicit pose (Zhang et al.[2016], Krafka et al.[2017])
● Training methods
○ Data driven (Zhang et al.[2015, 2016])
○ User specific (Kassner et al. [2014], Huang et al.[2014], Papoutsaki et al.[2016])
4/11

5. Gaze-Net: Appearance Based Gaze Estimation@NirdsLab
Approach
● Two-step approach
○ Classify
○ Estimate
● Classification
○ Convolution NN
○ Capsule Network
● Estimation
○ Fully connected
● Regularization
○ Reconstruction
○ Estimation error
5/11
Left Top Middle Top Right Top
Left Bottom Middle Bottom Right Bottom

6. Gaze-Net: Appearance Based Gaze Estimation@NirdsLab
Training and Testing
● Training
○ MPIIGaze dataset (200,000+ images)
■ https://arxiv.org/abs/1711.09017
● Testing
○ MPIIGaze dataset
○ Columbia Gaze dataset (~5000 images)
6/11
MPIIGaze Dataset: Raw images
(https://www.mpi-inf.mpg.de/)
MPIIGaze Dataset: Processed
images
(https://www.mpi-inf.mpg.de/)

7. Gaze-Net: Appearance Based Gaze Estimation@NirdsLab
Experiments
● Metrics
○ Accuracy - Gaze categorization
○ Mean Absolute Error - Gaze estimation
● Experiment conditions
○ No regularization
○ Gaze estimation regularization
○ Image Reconstruction
○ Estimation + Reconstruction
7/11
Accuracy MAE (Estimation)
No Regularization 67.15 -
Image
Reconstruction
65.97 -
Gaze Error 63.98 2.88
Gaze Error +
Reconstruction
62.67 2.84
Figure 2: Comparison of MPIIGaze image
reconstructionwith the original images.Œ
top row shows the reconstructed images,
and the bottŠom row shows the original
images.
Table 1: Classi€cation Accuracy (ACC) and Mean
AbsoluteError (MAE) of Gaze Estimation for each
Regularizationmethod.

8. Gaze-Net: Appearance Based Gaze Estimation@NirdsLab
Transfer Learning
● Transfer Learning
○ Knowledge from one problem on another
● Dataset
○ Columbia Gaze Dataset
○ Ocular region extracted using PoseNet
■ PoseNet: Real-time pose estimation model
■ https://github.com/tensorflow/tfjs-
models/tree/master/posenet
○ Per participant experiments
8/11
Processed images from Columbia Gaze
Dataset

9. Gaze-Net: Appearance Based Gaze Estimation@NirdsLab
Transfer Learning - Experiments
● Conditions
○ No retraining
○ Retraining estimation
network
9/11
MAE
(Estimation)
No Retraining 10.04
Retraining Estimation
Network
5.92
Table 2: Mean Absolute Error (MAE) of
gaze estimation be-fore and a…er training
on Columbia Gaze Dataset.

10. Gaze-Net: Appearance Based Gaze Estimation@NirdsLab
Discussion
● Gaze estimation with ocular
images
○ Decoding head pose
○ Decoding eye rotation
● Transfer learning for
personalizing
○ Generalized model from
larger dataset
○ Personalized from a smaller
dataset
10/11
Figure 3: Dimension perturbations.Each row shows the
reconstruction when one of the 16 dimensions in the
GazeCaps output is tweaked by intervals of 0.125 in the
range[−0.25,0.25]

11. Gaze-Net: Appearance Based Gaze Estimation@NirdsLab
Questions?
● Ocular images are sufficient for
○ Decoding facial orientation
○ Eye rotation
○ Estimating gaze
● Transfer learning
○ Better performing personalized models
● More info
○ MGaze: https://mgaze.nirds.cs.odu.edu/
○ Research Group: @NirdsLab
○ Homepage: https://www.cs.odu.edu/~bhanuka/
○ Twitter: @mahanama94, @yasithmilinda, @openmaze
11/11