Presentation given at TEI 2011, 5th International Conference on Tangible, Embedded and Embodied Interaction, Madeira, Portugal, January 2011 ABSTRACT: Over the past few years, multi-touch user interfaces emerged from research prototypes into mass market products. This evolution has been mainly driven by innovative devices such as Apple’s iPhone or Microsoft’s Surface tabletop computer. Unfortunately, there seems to be a lack of software engineering abstractions in existing multi-touch development frameworks. Many multi-touch applications are based on hardcoded procedural low level event processing. This leads to proprietary solutions with a lack of gesture extensibility and cross-application reusability. We present Midas, a declarative model for the definition and detection of multi-touch gestures where gestures are expressed via logical rules over a set of input facts. We highlight how our rule-based language approach leads to improvements in gesture extensibility and reusability. Last but not least, we introduce JMidas, an instantiation of Midas for the Java programming language and describe how JMidas has been applied to implement a number of innovative multi-touch gestures.

1. A
Declarative
Multi-Touch
MIDAS Interaction
Framework
CHRISTOPHE SCHOLLIERS
LODE HOSTE
BEAT SIGNER
WOLFGANG DE MEUTER
VRIJE UNIVERSITEIT BRUSSEL
#MidasFW

2. MULTI-TOUCH IS EVERYWHERE!

3. HARDWARE CAPABILITIES VS
GESTURE RECOGNITION SOFTWARE
Pinch St ret c h D ra g
Rot a te Fl i c k

4. WHAT’S THE FUNDAMENTAL PROBLEM?
add 0 0,633750 0,293750 update 0 0,565000 0,335417 update 1 0,251250 0,620833 update 1 0,166250 0,343750
update 0 0,635000 0,293750 update 1 0,387500 0,450000 update 0 0,656250 0,666667 update 0 0,728750 0,306250
update 0 0,633750 0,293750 update 0 0,560000 0,345833 update 1 0,242500 0,604167 update 1 0,172500 0,331250
update 0 0,633750 0,295833 update 1 0,390000 0,462500 update 0 0,663750 0,662500 update 0 0,725000 0,300000
update 0 0,633750 0,297917 update 0 0,558750 0,352083 update 1 0,237500 0,591667 update 1 0,177500 0,318750
update 0 0,632500 0,300000 update 1 0,391250 0,479167 update 0 0,667500 0,658333 update 0 0,723750 0,295833
update 0 0,631250 0,300000 update 0 0,556250 0,358333 update 1 0,232500 0,583333 update 1 0,181250 0,314583
update 0 0,630000 0,300000 update 1 0,391250 0,489583 update 0 0,672500 0,656250 update 0 0,720000 0,291667
update 0 0,628750 0,300000 update 0 0,553750 0,366667 update 1 0,215000 0,558333 update 1 0,185000 0,310417
update 0 0,627500 0,300000 update 1 0,392500 0,502083 update 0 0,691250 0,637500 update 0 0,718750 0,289583
add 1 0,246250 0,289583 update 0 0,550000 0,377083 update 1 0,206250 0,543750 update 1 0,186250 0,304167
update 1 0,247500 0,289583 update 1 0,392500 0,516667 update 0 0,702500 0,612500 update 0 0,716250 0,287500
update 1 0,248750 0,291667 update 0 0,548750 0,385417 update 1 0,187500 0,510417 update 1 0,190000 0,302083
update 1 0,250000 0,291667 update 1 0,392500 0,531250 update 0 0,711250 0,589583 update 0 0,713750 0,281250
update 1 0,252500 0,291667 update 0 0,546250 0,395833 update 1 0,167500 0,485417 update 1 0,192500 0,300000
update 1 0,256250 0,291667 update 1 0,390000 0,543750 update 0 0,716250 0,568750 update 0 0,708750 0,277083
update 1 0,258750 0,291667 update 0 0,543750 0,408333 update 1 0,161250 0,477083 update 1 0,198750 0,293750
update 1 0,263750 0,291667 update 1 0,388750 0,556250 update 0 0,720000 0,556250 update 0 0,702500 0,270833
update 0 0,626250 0,300000 update 0 0,542500 0,422917 update 1 0,152500 0,466667 update 1 0,205000 0,289583
update 1 0,276250 0,291667 update 1 0,386250 0,566667 update 0 0,725000 0,539583 update 0 0,692500 0,266667
update 0 0,625000 0,300000 update 0 0,543750 0,429167 update 1 0,146250 0,460417 update 1 0,208750 0,289583
update 1 0,285000 0,295833 update 1 0,385000 0,577083 update 0 0,727500 0,527083 update 0 0,688750 0,264583
update 0 0,622500 0,300000 update 0 0,543750 0,441667 update 1 0,143750 0,456250 update 1 0,212500 0,287500
update 1 0,293750 0,300000 update 1 0,378750 0,595833 update 0 0,728750 0,510417 update 0 0,685000 0,264583
update 0 0,621250 0,302083 update 0 0,543750 0,450000 update 1 0,138750 0,447917 update 1 0,216250 0,285417
update 1 0,300000 0,304167 update 1 0,367500 0,633333 update 0 0,731250 0,502083 update 0 0,678750 0,256250
update 1 0,310000 0,310417 update 0 0,547500 0,475000 update 1 0,136250 0,445833 update 1 0,221250 0,283333
update 0 0,618750 0,302083 update 1 0,348750 0,660417 update 0 0,733750 0,485417 update 0 0,675000 0,252083
update 1 0,318750 0,316667 update 0 0,551250 0,504167 update 1 0,133750 0,441667 update 1 0,227500 0,283333
update 0 0,616250 0,302083 update 1 0,341250 0,666667 update 0 0,736250 0,472917 update 0 0,671250 0,250000
update 1 0,325000 0,320833 update 0 0,553750 0,514583 update 1 0,132500 0,439583 update 1 0,231250 0,281250
update 0 0,613750 0,302083 update 1 0,336250 0,675000 update 0 0,740000 0,458333 update 0 0,667500 0,250000
update 1 0,330000 0,322917 update 0 0,557500 0,527083 update 1 0,131250 0,435417 update 1 0,233750 0,281250
update 0 0,611250 0,302083 update 1 0,331250 0,679167 update 0 0,741250 0,447917 update 0 0,662500 0,247917
update 1 0,335000 0,325000 update 0 0,561250 0,541667 update 0 0,741250 0,439583 update 1 0,238750 0,279167
update 0 0,610000 0,302083 update 1 0,326250 0,683333 update 1 0,130000 0,431250 update 0 0,658750 0,247917
update 1 0,338750 0,329167 update 0 0,565000 0,552083 update 0 0,742500 0,422917 update 1 0,245000 0,279167
update 0 0,607500 0,304167 update 1 0,321250 0,685417 update 1 0,130000 0,429167 update 0 0,655000 0,245833
update 1 0,341250 0,333333 update 0 0,570000 0,572917 update 0 0,742500 0,416667 update 1 0,248750 0,277083
update 0 0,605000 0,304167 update 1 0,316250 0,685417 update 1 0,130000 0,427083 update 0 0,651250 0,245833
update 1 0,345000 0,335417 update 0 0,575000 0,583333 update 0 0,743750 0,402083 update 1 0,253750 0,277083
update 0 0,602500 0,304167 update 1 0,311250 0,685417 update 1 0,130000 0,425000 update 0 0,646250 0,250000
update 1 0,347500 0,337500 update 0 0,581250 0,595833 update 0 0,745000 0,395833 update 1 0,258750 0,275000
update 0 0,600000 0,304167 update 1 0,307500 0,683333 update 1 0,131250 0,422917 update 0 0,642500 0,250000
update 1 0,351250 0,341667 update 0 0,590000 0,612500 update 0 0,745000 0,387500 update 1 0,263750 0,272917
update 0 0,596250 0,304167 update 1 0,302500 0,681250 update 1 0,132500 0,418750 update 0 0,638750 0,254167
update 1 0,355000 0,347917 update 0 0,595000 0,618750 update 0 0,745000 0,383333 update 1 0,267500 0,270833
update 0 0,593750 0,306250 update 1 0,296250 0,679167 update 1 0,133750 0,416667 update 0 0,635000 0,256250
update 1 0,358750 0,352083 update 0 0,606250 0,633333 update 0 0,743750 0,377083 update 1 0,271250 0,270833
update 0 0,590000 0,308333 update 1 0,290000 0,675000 update 1 0,135000 0,412500 update 0 0,631250 0,258333
update 1 0,361250 0,358333 update 0 0,612500 0,639583 update 0 0,743750 0,360417 update 1 0,278750 0,270833
update 0 0,586250 0,312500 update 1 0,281250 0,664583 update 1 0,136250 0,408333 update 0 0,625000 0,262500
update 1 0,365000 0,368750 update 0 0,622500 0,654167 update 0 0,741250 0,354167 update 1 0,282500 0,270833
update 0 0,583750 0,314583 update 1 0,272500 0,656250 update 1 0,141250 0,400000 update 0 0,622500 0,262500
update 1 0,368750 0,379167 update 0 0,631250 0,660417 update 0 0,738750 0,337500 update 1 0,286250 0,270833
update 0 0,580000 0,316667 update 1 0,268750 0,647917 update 1 0,145000 0,389583 update 0 0,621250 0,264583
update 1 0,372500 0,389583 update 0 0,636250 0,662500 update 0 0,738750 0,333333 update 1 0,288750 0,268750
update 0 0,577500 0,318750 update 1 0,262500 0,639583 update 1 0,147500 0,383333 update 0 0,620000 0,266667
update 1 0,378750 0,410417 update 0 0,643750 0,664583 update 0 0,735000 0,325000 update 1 0,291250 0,268750
update 0 0,571250 0,329167 update 1 0,257500 0,629167 update 1 0,151250 0,372917 update 0 0,618750 0,268750
update 1 0,382500 0,420833 update 0 0,647500 0,664583 update 0 0,731250 0,316667 update 1 0,292500 0,268750
update 0 0,568750 0,333333 update 1 0,255000 0,627083 update 1 0,157500 0,360417 update 0 0,617500 0,268750
update 1 0,383750 0,435417 update 0 0,653750 0,666667 update 0 0,728750 0,308333 update 1 0,293750 0,268750

5. WHAT’S THE FUNDAMENTAL PROBLEM?
 Sequential code for the
processing of concurrent events
 Find spatial and temporal
relations in an immense pool
of x/y coordinates
 ~25 events / second / finger
 Approximative detection
 GUI-Event correlation
 Modularisation and composition
 implementing gestures is hard
 combining gestures is even harder

6. THE MIDAS APPROACH
SELECT * FROM Audience
WHERE interest > 0
Facts
Fact Base Rules

7. MIDAS ARCHITECTURE
 Infrastructure layer Midas
 Transform low level Application Layer
data into facts
GUI Shadows Model
 Core layer
 temporal and spatial Core Layer
operators Fact Inference Rule
 continuous queries via Base Engine Base
Rete algorithm
Infrastructure Layer
 Application layer Hardware *
Bridge Translator
 regular program
augmented with rules

8. MIDAS USES
(defrule PrintCursor
?c <- (Cursor)
=>
(printout t ‘‘A cursor is moving at: ’’ ?c.x ‘‘,’’ ?c.y))

9. MIDAS FILTERS
+
(defrule Tap
?appear <- (Cursor (finger ?f) (state ?*APPEAR*))
?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*))
(test (tShortlyAfter ?disappear ?appear))
=>
(printout t “A single tap detected at: “ ?appear.x “,” ?appear.y))
only a few lines of declarative code

10. MIDAS FINDS
+
(defrule Tap
?appear <- (Cursor (finger ?f) (state ?*APPEAR*))
?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*))
(test (tShortlyAfter ?disappear ?appear))
(test (sVeryNear ?disappear ?appear))
=>
(printout t “A single tap detected at: “ ?appear.x “,” ?appear.y)
(assert (Tap (x ?appear.x) (y ?appear.y) (on ?appear.on))
(retract ?appear ?disappear))

11. MIDAS ENCOURAGES
(defrule DoubleTap
?tap1 <- (Tap)
?tap2 <- (Tap)
(test (tShortlyAfter ?tap2 ?tap1))
(test (sVeryNear ?tap2 ?tap1))
=>
(printout t “A double tap detected at: “ ?tap1.x“,” ?tap2.y)
(retract ?tap1 ?tap2))
small and reusable building blocks

12. MIDAS ENCOURAGES
(defrule FixedUpUp
?fixed <- (Fixed)
?up1 <- (Up)
?up2 <- (Up)
(test (tEqual3 ?fixed ?up1 ?up2))
(test (sLeftOf ?fixed ?up1))
(test (sLeftOf ?up1 ?up2))
=>
(printout t “Complex scroll detected!”)
(retract ?fixed ?up1 ?up2))

13. MIDAS CORRELATES
add 0 0,633750 0,293750 update 0 0,565000 0,335417
update 0 0,635000 0,293750 update 1 0,387500 0,450000
(defrule Gui
update 0 0,633750 0,293750 update 0 0,560000 0,345833
update 0 0,633750 0,295833 update 1 0,390000 0,462500
update 0 0,633750 0,297917 update 0 0,558750 0,352083
update 0 0,632500 0,300000 update 1 0,391250 0,479167
?tap<- (Tap) update 0 0,631250 0,300000
update 0 0,630000 0,300000
update 0 0,628750 0,300000
update 0 0,627500 0,300000
update 0 0,556250 0,358333
update 1 0,391250 0,489583
update 0 0,553750 0,366667
update 1 0,392500 0,502083
?planet <- (Planet) add 1 0,246250 0,289583
update 1 0,247500 0,289583
update 1 0,248750 0,291667
update 0 0,550000 0,377083
update 1 0,392500 0,516667
update 0 0,548750 0,385417
(test (sInside ?tap ?planet))
update 1 0,250000 0,291667 update 1 0,392500 0,531250
update 1 0,252500 0,291667 update 0 0,546250 0,395833
update 1 0,256250 0,291667 update 1 0,390000 0,543750
=>
(printout t “Tap inside planet!”))
Fact Base
Shadows
(GUI elements)

14. MIDAS SUPPORTS
Templates
(defrule nextSlide
?goRight1 <- (FlickRight (tuioId ?*RH*))
?goLeft <- (FlickLeft (tuioId ?*LH*))
?goRight2 <- (FlickRight (tuioId ?*RH*))
(test (tMeets ?goRight1 ?goLeft ?goRight2))
=>
(call PowerPoint showNextSlide)
(retract ?goRight1 ?goLeft ?goRight2))
Implemented online gestures Kinect example

15. CONCLUSION
 Midas multi-touch interaction framework
 Declarative description of gestures (fact base and rule engine)
 extensibility and reusability of gestures
 correlation of GUI elements and events via shadow facts
 JMidas Java integration
 Ongoing work
 integration of new devices: Kinect, Digital Pen & Paper, ...
 extend Midas to support multimodal gestures
 Rapid prototyping of multi-touch gestures
Midas Demo, Tuesday @ 15.00-18.30 -- ID: 251
Research: http://goo.gl/OFpej - Youtube: http://goo.gl/MNDkT - Twitter: #MidasFW