Presentation by Romuald Deshayes at VL/HCC 2013 (San Jose, California) of joint paper with Tom Mens and Philippe Palanque about executable Petri net modeling of HMI applications

1. Behaviour and interaction modeling of 3D virtual objects
A Generic Framework for Executable
Gestural Interaction Models
Romuald Deshayes, Tom Mens, Philippe Palanque
Software Engineering Lab IRIT
2012-2013
Deshayes Romuald – UMONS 1 / 15

2. Behaviour and interaction modeling of 3D virtual objects
Table of Contents
1 Introduction
Context
Paper Goal
2 The Framework
Modular and Generic Framework
Layers and Gestures
3 Conclusion
4 Future Work
Deshayes Romuald – UMONS 2 / 15

3. Behaviour and interaction modeling of 3D virtual objects
Introduction
Context
Context
Different objects with different ways of interacting with them
Enhance computer with better insight in user interaction with
real and virtual objects
[Target Domain] Virtual and Augmented Reality applications such
as games, home automation and simulation
Deshayes Romuald – UMONS 3 / 15

4. Behaviour and interaction modeling of 3D virtual objects
Introduction
Context
Context
Context
Development of Human-Computer Interaction (HCI) applications
involving gestural interaction
High technical complexity (low level aspects)
Low learning curve
Deshayes Romuald – UMONS 4 / 15

5. Behaviour and interaction modeling of 3D virtual objects
Introduction
Paper Goal
Paper Goal
Goal
Provide a generic and efficient way of describing and executing software
involving gestural interaction with real or virtual objects
Based on executable models
Domain Specific Modeling Language (DSML)
A DSML simplifies the creation of gesture based application
Generic framework for gestural interaction
Deshayes Romuald – UMONS 5 / 15

6. Behaviour and interaction modeling of 3D virtual objects
Introduction
Paper Goal
Why Modeling
Why is executable modeling appropriate for developing HCI
applications ?
Visual aspects allow to hide low level details by raising the
level of abstraction (simplicity)
Formal analysis can help reduce the amount of bugs in
complex systems (security)
Behavior can be changed at runtime by editing the running
models (productivity)
Deshayes Romuald – UMONS 6 / 15

7. Behaviour and interaction modeling of 3D virtual objects
Introduction
Paper Goal
Why Modeling
Why is executable modeling appropriate for developing HCI
applications ?
Visual aspects allow to hide low level details by raising the
level of abstraction (simplicity)
Formal analysis can help reduce the amount of bugs in
complex systems (security)
Behavior can be changed at runtime by editing the running
models (productivity)
Challenge
counter critique from [SKF+94] that model-based approaches
"model the previous generation of UI"
Deshayes Romuald – UMONS 6 / 15

8. Behaviour and interaction modeling of 3D virtual objects
The Framework
Modular and Generic Framework
The framework
Goal
Provide a generic and efficient way of describing and executing
software involving gestural interaction with real or virtual objects
How ?
By creating a modular and generic framework that allows the
specification of executable models for manipulating objects with
gestures
I/O independence
Client/server approach for modularity
Layered architecture
Deshayes Romuald – UMONS 7 / 15

9. Behaviour and interaction modeling of 3D virtual objects
The Framework
Modular and Generic Framework
A Layered Architecture
A layered architecture
Each layer is modeled as an ICO (formalism based on
High-Level Petri nets)
Layers communicate through events
Deshayes Romuald – UMONS 8 / 15

10. Behaviour and interaction modeling of 3D virtual objects
The Framework
Layers and Gestures
The Layers and the Gestures
The layers
L1 detects new users, receives raw data and converts them to
positions w.r.t head position
L2 create low-level gestures and detect hand state changes
Move
Open
Close
L3 combine low-level gestures with users’ state to create
high-level gestures
SimpleDrag
ColinearDrag
NonColinearDrag
Deshayes Romuald – UMONS 9 / 15

11. Behaviour and interaction modeling of 3D virtual objects
The Framework
Layers and Gestures
Modeling the state of the users
Layer 3 : modeling the state of the users
Deshayes Romuald – UMONS 10 / 15

12. Behaviour and interaction modeling of 3D virtual objects
The Framework
Layers and Gestures
Modeling the interaction with virtual objects
Layer 4 contains a different
model for each object
Used to combine all existing
gestures into a model
describing how to interact
with an object
Example : a book
Deshayes Romuald – UMONS 11 / 15

13. Behaviour and interaction modeling of 3D virtual objects
Conclusion
Conclusion
Advantages of resorting to executable modeling for HCI application
development
Reduced technical complexity thanks to higher level of
abstraction
Dynamic execution particularly suitable for rapid prototyping
of interactive applications
Communicating models allow to concurrently interact with
multiple objects and enable separation of concerns
Our framework allows to easily specify gestural interaction
with real or virtual objects
Deshayes Romuald – UMONS 12 / 15

14. Behaviour and interaction modeling of 3D virtual objects
Future Work
Future Work
Current research and future work
Explore heterogeneity in the layers to use the most appropriate
formalism (MultiPM vs MonoPM)
Develop a DSML for Layer 4
Perform a controlled user study with CS students to evaluate
the productivity gain of using the framework and its usability.
Code generation
Deshayes Romuald – UMONS 13 / 15

15. Behaviour and interaction modeling of 3D virtual objects
Future Work
Thank you
Questions ?
Deshayes Romuald – UMONS 14 / 15

16. Behaviour and interaction modeling of 3D virtual objects
Future Work
Bibliography I
P. N. Sukaviriya, S. Kovacevic, J. D. Foley, B. A. Myers, D. R.
Olsen Jr., and M. Schneider-Hufschmidt.
Model- based user interfaces: What are they and why should
we care?
In Proc. UIST94, pages 133–135. ACM, November 1994.
Deshayes Romuald – UMONS 15 / 15