2022.07.15
So Predictable! Continuous 3D Hand Trajectory Prediction in Virtual
Reality
Nisal Menuka Gamage, Deepana Ishtaweera, Martin Weigel, and Anusha Withana
UIST 2021
SeongOuk Kim

Contents
• Introduction
• Related Works
• Design Goals
• User Study
• Hybrid Kinematic Regressive
Model
• Verification of the Model
• Discussion
• Limitations and Future Work
• Conclusion

3
Introduction
Motivations
• Detecting & Tracking user interaction is important in VR
• Prediction can help it
• Current prediction is usually focused on particular event
Contributions
• A kinematics-based prediction approach for structured and
unstructured ballistic 3D hand movements in VR activities
• A user- and activity-independent model with similar performance to
dependent models
• Evaluation of the model through cross-validation and a secondary
study with new participants and new activities

4
Related Works
Human Motion Prediction Techniques
• Template matching, HMM – hard for arbitrary movements
• Regression(end-point, EMG)
• Deep learning(RNN, CNN, GAN) – too heavy
Motion Prediction for VR
• Reducing lag, foveated rendering, or haptic retargeting
• Head motion, Hand motion
Kinematics of Hand Movements
• Dynamic end-effector models – Minimum jerk model [Hogan, et al]
• Dynamic models for specific task

5
Design Goals
Continuous Prediction
• Can predict at arbitrary time points in the future
Structured & Unstructured Motion
• Not limited to a specific task
User & Activity-Independent
• No re-training for new users & activities
Explainable Prediction
• No black-box
• Can explain

6
User Study
Participants
• 7 female, 13 male
• Mean age = 22.4y(SD= 5.1y)
• 1 left-handed
Apparatus
• Oculus Quest
• OptiTrack with 8 cameras
to record hand movement
• Sensor data + VR screen

7
User Study
T1: Structured Movement via 3D Pointing
• Move hands towards virtual points
• 2 distances(20cm, 40cm), and 2 angular
deviations(30˚, 45 ˚)

8
User Study
T2: Unstructured Movement via VR Gameplay
• 3 minutes for each game
• Beatsaber - slashing, highest average speed (0.72m/s), highest
horizontal & vertical span (0.85m, 0.95m)
• FitXR – Boxing, lower horizontal span (0.57m) & highest frontal
span (0.75m)
• Eleven – varied among users

9
User Study
Data Preparation and Presentation
• Data from Optitrack
• Gaussian filter to reduce noise
• 30s (<10% of data from T1 and 15% of data of T2) as the training
set

10
Hybrid Kinematic Regressive Model
𝑡0: initial time
𝑡: prediction time interval
k: identified value for kinematics
Classical Kinematics of Motion

11
Hybrid Kinematic Regressive Model
Prediction-Time Dependent Kinematics Regression

12
Hybrid Kinematic Regressive Model
Inferred Prediction-Time Independent Kinematic Modeling
• Direct classical + regressed piecewise models(Φ′)

13
Hybrid Kinematic Regressive Model
Inferred Prediction-Time Independent Kinematic Modeling
• Interpolated classical + regressed piecewise models(Φ′′)

14
Hybrid Kinematic Regressive Model
Results
• RMSE
 100ms – 0.8cm(SD=0.12cm)
 200ms – 0.85cm(SD=0.14cm)
 300ms – 3.15cm(SD=0.38cm)

15
Hybrid Kinematic Regressive Model
Results

16
Hybrid Kinematic Regressive Model
Results

17
Verification of the Model
Cross-Validation
• 4-folds
New Users & Activities
• 3 more participants & 2 new Activities(Sweeping,
dancing)

18
Discussion
• Better performance than non-naïve
baseline with low overhead
• Can be used for countering delay, error
correction, etc
• Well performed on new users
• Also worked when using Oculus data

19
Limitations and Future Work
Only voluntary ballistic movements
• Need to work on other types of tasks(e.g. steering)
Only participants in 18~39 and not disabled
• Need to test with other age groups
Only wrist data for prediction
• Can use other body data
• Hand ≠ wrist
3D motion is not only for VR
• Can be applied to other fields.

20
Conclusion
• Contributed to novel hybrid classical-regressive kinematic model for
continuous 3D hand trajectory
• Can be used for many areas, even not in VR
My thought
• Impressive for not using deep learning
• Can it be applied for 2-handed cases?

Thank you