Developing intuitive interactive applications that are easy to maintain by developers is quite challenging, due to the complexity and the many technical aspects involved in such applications. In this article, we tackle the problem in two complementary ways. First, we propose a gestural interface to improve the user experience when interacting with applications that require the manipulation of 3D graphical scenes. Second, we reduce the complexity of developing such applications by modeling their executable behaviour using statecharts. We validate our approach by creating a modular and extensible Java framework for the development of in- teractive gesture-based applications. We developed a proof- of-concept application using this framework, that allows the user to construct and manipulate 3D scenes in OpenGL by using hand gestures only. These hand gestures are captured by the Kinect sensor, and translated into events and actions that are interpreted and executed by communicating statecharts that model the main behaviour of the interactive application.

1. Statechart modeling of interactive gesture-based applications
Statechart modeling of interactive
gesture-based applications
Romuald Deshayes Tom Mens
Software Engineering Lab
2010-2011
Deshayes Romuald – UMONS 1 / 20

2. Statechart modeling of interactive gesture-based applications
Table des matières
1 Problem Statement
2 Proof-of-concept application
3 Modeling interactive behaviour
4 Statechart models
5 The application framework
6 Conclusion and Future Work
Deshayes Romuald – UMONS 2 / 20

3. Statechart modeling of interactive gesture-based applications
Problem Statement
Outline : Problem Statement
Problem Statement
Deshayes Romuald – UMONS 3 / 20

4. Statechart modeling of interactive gesture-based applications
Problem Statement
Problem Statement
Challenge
Develop intuitive interactive applications that are easy to maintain
Why is it a challenge ?
Complex behaviour
Nondeterministic user actions
Same events lead to different actions
Contributions
Gestural user interface for 3D objects manipulation
Modeling executable behaviour using a visual formalism
Validation with an application framework
Deshayes Romuald – UMONS 4 / 20

5. Statechart modeling of interactive gesture-based applications
Proof-of-concept application
Outline : Proof-of-concept application
Proof-of-concept application
Deshayes Romuald – UMONS 5 / 20

6. Statechart modeling of interactive gesture-based applications
Proof-of-concept application
Proof-of-concept application
3D visual drawing tool
Uses gestures to create and manipulate 3D objects
OpenGL graphical library
Microsoft Kinect + NITE
Deshayes Romuald – UMONS 6 / 20

7. Statechart modeling of interactive gesture-based applications
Proof-of-concept application
Small video : www.youtube.com/watch?v=lVcqzWTnpMY
Deshayes Romuald – UMONS 7 / 20

8. Statechart modeling of interactive gesture-based applications
Modeling interactive behaviour
Outline : Modeling interactive behaviour
Modeling interactive behaviour
Deshayes Romuald – UMONS 8 / 20

9. Statechart modeling of interactive gesture-based applications
Modeling interactive behaviour
Modeling interactive behaviour
Context
Highly reactive event-driven systems
Gesture-based interaction
Proposed solutions
Visual modeling language
Statecharts
Petri nets
Amenable to formal analysis
Easier to evolve
Reduced complexity
Deshayes Romuald – UMONS 9 / 20

10. Statechart modeling of interactive gesture-based applications
Statechart models
Outline : Statechart models
Statechart models
Deshayes Romuald – UMONS 10 / 20

11. Statechart modeling of interactive gesture-based applications
Statechart models
Statechart models - Hand
Deshayes Romuald – UMONS 11 / 20

12. Statechart modeling of interactive gesture-based applications
Statechart models
Statechart models - Component
Deshayes Romuald – UMONS 12 / 20

13. Statechart modeling of interactive gesture-based applications
Statechart models
Statechart models - Scene
Deshayes Romuald – UMONS 13 / 20

14. Statechart modeling of interactive gesture-based applications
The application framework
Outline : The application framework
The application framework
Deshayes Romuald – UMONS 14 / 20

15. Statechart modeling of interactive gesture-based applications
The application framework
The application framework
Framework for developing interactive applications
Executable behaviour specified using statecharts
Client-server architecture
Deshayes Romuald – UMONS 15 / 20

16. Statechart modeling of interactive gesture-based applications
The application framework
The application framework II
Features
Generic
Abstract classes
Observer design pattern
Communication between statecharts
Uses Java SwingStates library
Deshayes Romuald – UMONS 16 / 20

17. Statechart modeling of interactive gesture-based applications
The application framework
The application framework III
<<library>>
Framework SwingStates
Server extends Thread
-serverSocket : DatagramSocket StateMachine
Controller left 0..1
1..* -machine : StateMachine Pointer
<<Interface>> right -open : boolean
+addComponent(c : Component) 0..1
BodyListener extends -machine : StateMachine
+addPointer(p : Pointer)
EventListener +addScene(s : Scene)
Scene
Main Application
+handMovedX(e : BodyEvent) 1..*
+handMovedY(e : BodyEvent) -machine : StateMachine
MainScene
+handMovedZ(e : BodyEvent) 1
+handClosed(e : BodyEvent)
+handOpened(e : BodyEvent)
selectedModel
1..*
* 0..1
Component Model
BodyAdapter 3DModel
-boundingBox : Rectangle -selected : boolean
+addAsListenerOf(s : Server) -machine : StateMachine
1..*
models
BodyAdapters OpenGLWidgets
Hand GLButton ExtendableMenu
-machine : StateMachine +changeActiveTexture() : void +extend() : void
+drawButton(gl : GL) : void +retract() : void
+startCounter() : void +drawMenu(gl : GL) : void
SpawningMenu
+addButton(button : GLButton) : void ImageMenu
+addText(text : String) : void ButtonMenu
+drawMenu(gl : GL) : void
+drawMenu(gl : GL) : void +addButton(button : GLBUtton) : void +onModel() : void
+spawn() : void +drawMenu(gl : GL) : void
+reduce() : void
Deshayes Romuald – UMONS 17 / 20

18. Statechart modeling of interactive gesture-based applications
Conclusion and Future Work
Outline : Conclusion and Future Work
Conclusion and Future Work
Deshayes Romuald – UMONS 18 / 20

19. Statechart modeling of interactive gesture-based applications
Conclusion and Future Work
Conclusion and Future Work
Statecharts
Appropriate and scalable formalism for modeling the
interactive gesture-based behaviour ?
Improvements
Externalise/decouple statechart representation from framework
Compare statecharts with Petri nets
Deshayes Romuald – UMONS 19 / 20

20. Statechart modeling of interactive gesture-based applications
Conclusion and Future Work
Future Work II
Apply our approach to other HCI applications
Other types of natural interfaces
Compare/combine with other UIML
Controlled experiments, carry out user studies, empirical
studies
Deshayes Romuald – UMONS 20 / 20