Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Touchless Circular Menus
1. Touchless Circular Menus:
Toward an Intuitive UI for Touchless
Interactions with Large Displays
Debaleena Chattopadhyay & Davide Bolchini
AVI 2014 International Working Conference on Advanced Visual Interfaces
Como (Italy) May 27-30, 2014
9. Current Debate on UI “naturalness”
Norman, D. A. (2010). Natural user interfaces are not natural. interactions, 17(3), 6-10.
Wigdor, D., & Wixon, D. (2011). Brave NUI world: designing natural user interfaces for touch and gesture. Elsevier.
O'hara, K., Harper, R., Mentis, H., Sellen, A., & Taylor, A. (2013). On the naturalness of touchless: putting the “interaction” back
into NUI. ACM Transactions on Computer-Human Interaction (TOCHI), 20(1), 5.
“You are the controller”
–Microsoft®
“Most gestures are neither natural
nor easy to learn or remember.”
–Norman
Natural is ‘a design philosophy and a
source of metrics enabling an
iterative process to create a product.’
–Wigdor & Wixon
“Naturalness is not something
to be represented but is rather
an ‘occasioned property of
action’ [..].”
–O’Hara et al.
10. Intuitiveness of
an interface
Prior
knowledge
Expertise
Culture
Sensorimotor
Innate
Specialized knowledge
acquired with practice.
Knowledge of people
specific to culture.
Knowledge acquired since
childhood through interaction
with the world.
Reflexes and intrinsic
behavior.
unconscious
application of prior
knowledge
∝
Hurtienne, J., & Israel, J. H., 2007. Image schemas and their metaphorical extensions: intuitive patterns for tangible interaction.
Proc. TEI,127-134, ACM.
Intuitive Interaction Framework
11. How can we use sensorimotor
knowledge to inform the design
of intuitive touchless selection
mechanisms?
12. Device – less
interaction
Device – based
interaction
Menu Invocation
Mouse click, pen down, or touching
surface
?
Menu-Selection Delimiter
Breaking contact with the interface
Menu Shape
?
Linear, radial
?
Hand posture
Hand posture
Linear, radial
14. Prior
sensorimotor
knowledge
Some successful
features of device-
based menus
Touchless Circular Menus (TCM)
+
Directional strokes
in mid-air
Menu-invocation:
Reaching a region-of-
interest.
Menu-selection: Crossing
Shape: Radial
15. Touchless circular
menus relieve users
from both recalling a
precise vocabulary of
hand-postures and
strictly complying with
them.
17. To select a menu option, the user makes a directional stroke.
To give feedback, when successfully selected, the menu-
option changes color.
18. To mitigate accidental invocation, TCM appear opposite to the
user’s direction of approach.
To cancel the invoked menu, the user may continue in her
direction of movement, or move away from the menu.
19. To invoke the menu, the user must reach the region-of-
interest (ROI) of the target.
20. Hierarchical TCM: To select a command, the user first selects
a menu option, and then accesses the submenu by continuing
her trajectory.
22. To trigger the contextual menu, a user must cross the region-of-interest (ROI) of a
display object.
The ROI can be of any symmetrical shape around the center of the target, with its size
directly proportional to the technique’s sensitivity.
Menu Invocation
23. To select a command after triggering the menu, users cross it using a stroke in the
command’s direction.
Until the crossing happens, users can cancel TCM by moving in any direction away from
the triggered menu.
To allow easy escape routes, we designed the structure of TCM as a semicircular array
of options appearing at the top-left or the bottom-right corner of the target.
Menu Selection
24. As users approach the menu, to give them orientation, a trace is drawn connecting the
target and the users’ hand position.
To improve users’ pointing performance, the menu options would increase in amplitude
as users approached them.
To provide further feedback, menu options changed color when selected.
Menu Selection
25. Currently, the menu design scales up to two levels (5 x 5), with users performing
continuous strokes.
To operate the submenu, users change their track and cross another command.
Due to the lack of precision and control of freehand movements, TCM require users to
make inflections in their continuing trajectories, and thereby avoid accidental command-
selections.
Hierarchical TCM
27. Experiments
How effectiveness and
efficiency of TCM is
affected by their triggering
locations on the visual
interface?
1
Vs.
Touchless circular menus
Linear menus using a grab
gesture
2
Participants (N = 15)
were sitting away from a
large display.
28. Experimental Design
Sample Demographics
N = 15
All right-handed
4/15 females
8/15 had prior familiarity with touchless gestures
11/15 were below 30 years of age
Time/participant: 1 hour
1Experiment
9 trials
X 7 blocks
X 15 participants
= 945 trials
Experiment 2
6 trials
X 7 blocks
X 15 participants
= 630 trials
29. Apparatus
Display—
Built by Fakespace, eight 50”
projection cubes, each with a
resolution of 1600 X 1200
pixels.
160” wide and 60” high display
Total resolution: 15.3 M pixels.
Motion Tracking
Sensor—
Kinect for Windows™
UITS Research Technologies
Advanced Visualization Lab
32. Triggering location of TCM significantly
affected selection time and successful trigger rate.
“I felt I had to rush to
select the menu
option”
“I was surprised that I
could do so well.”
1Experiment
Results
33. Arm posture affects users’ control
on their hand movements and the
required effort.
1Experiment
2D plane locations relative to the user’s
body and arm configuration affects the
consumed endurance in touchless
interactions
Hincapié-Ramos, J. D., Guo, X., Moghadasian, P., & Irani, P. (2014). Consumed Endurance: A metric to quantify arm fatigue of
mid-air interactions. Proc. CHI, 1063-1072, ACM.
34. Menu triggering locations will
significantly affect the user
experience of contextual touchless
menus for large displays.
Arm posture affects users’
control on their hand
movements and the
required effort.
Contextual touchless
menus for large displays
will have different menu-
triggering locations.
1Experiment
35. Hypotheses
H3: We predicted TCM would be more efficient
than linear menus.
H4: We hypothesized that TCM would be
easier to use than linear menus.
Experiment 2
36. Experiment 2
Compared with linear menus, users were more
efficient with TCM, and perceived lower overall
workload.
But TCM were less effective than linear menus.
“This is how I envision
using touchless
gestures.”
Results
“It was a lot of effort.”
37. Experiment 2
Compared with linear menus
using grab gestures, participants
using TCM were twice faster in
selecting commands and perceived
lower workload.
However, TCM caused 3% more
errors than linear menus.
38. Experiment 2Why TCM were less effective?
1st gesture registration
to invoke the menu
2nd gesture
registration to
select the menu
Gesture relaxation
39. Experiment 2Why TCM were less effective?
1st gesture registration
to invoke the menu
2nd gesture
registration to
select the menu
No gesture relaxation
41. Minimize errors by constraining users’
freehand movements after triggering
the TCM.
Investigate bimanual gestures to operate
menu levels in hierarchical TCM.
42. Takeaways
Menu triggering locations significantly affect the user
experience of contextual touchless menus for large displays.
Simple directional movements for selecting commands is
more efficient and causes lower workload than (hand)
posture-based techniques.
TCM relieve users from learning posture-based commands,
and shift the interaction complexity from users’ input to the
visual interface.
Thank you.
We thank the study participants and UITS AVL. This research is partially supported by an IUPUI Research Support Funds Grant.
1
2
3
Efficient gesture relaxation techniques may have the
potential to increase effectiveness of touchless interactions.
4
Hinweis der Redaktion
While interacting with large, distant displays…Touchless interaction frees users from employing any intermediate technology and almost has a universal appeal. Large-display touchless interaction is being increasingly used in a number of usage contexts.
Public displays, where users interact for a brief amount of time and may not spend the time and effort to connect a hand-held device with the display.
Sterile operating rooms, where surgeons may need to browse medical images and cannot touch devices.
Interactive TVs, where users can use touchless interactions to browse multimedia sporadically or access their favorite commands.
Given this range of different scenarios, different stakeholders, and varied user expertise, it is a challenge to design an optimal touchless user experience for interacting with large displays.
However, touchless interaction is still in its infancy.
Specifically touchless interactions with large displays lacks a standardized user interface language for frequent user-operations, such as..
Command selection.
Whereas an extensive body of works investigated optimal menu designs for mouse, pen-input, or multitouch surfaces, very few have looked into touchless command-selection techniques …..especially for large displays.
Recent solutions that appeared in research venues require users to comply strictly with system-defined poses, such as pinching with fingers, or making different finger combinations. These require users to recall a vocabulary of gestures, and in-lab user-studies have reported their high mental and physical demand.
On product platform, recent solutions include touchless menus using grab gestures ..closing and opening the hand…. Despite the potential of such ‘smart interactions’, experts have commented on such products’ low user satisfaction.
The purpose of our work
Was to propose and validate a novel form of command selection language appropriate for touchless interaction with large displays.
Given the variety of contexts, stakeholders, and user expertise, while designing the touchless menu, we considered several aspects..
For example, the claimed naturalness of this interaction modality,
As well as its differences form other input modalities, such as pen or touch.
Ironically, when you are designing touchless interactions, the word “natural” comes both as your ally and your adversary.
As markerless motion tracking became mainstream, users became the controller.
Yet soon, gestures, lacking feedback and learnability, were critiqued to be neither natural, nor easy to learn or remember.
Natural, was then described as a design philosophy to enable the iterative creation of a product.
And naturalness was explained as not something to be represented by the mechanics of a touchless gesture, but rather experienced as a property of the action taking place.
To operationalize the ‘naturalness’ construct, we used the intuitive interaction framework.
This framework suggests that intuitiveness………..often the twin term of naturalness….. is directly proportional to the unconscious application of prior knowledge. Now prior knowledge can be classified as expertise, culture, sensorimotor or innate.
Expertise level of knowledge is specialized knowledge acquired with practice.
In the Culture level, we’d find knowledge of people specific to culture.
Sensorimotor level of knowledge is acquired since childhood through interaction with the world.
Finally innate knowledge is developed in a prenatal state, such as reflexes, or intrinsic behavior.
One important thing to note here… is that …..The higher the degree of specialization of knowledge, the smaller would be the potential number of users applying that knowledge unconsciously.
Touchless interaction with large displays appeals to a variety of domains where users would have different expertise and cultural backgrounds. Moreover, given the lack of users’ control in engaging or disengaging innate knowledge, our approach uses sensorimotor knowledge to inform the design of intuitive touchless command selection techniques.
But then the question stands as….
how can we use sensorimotor knowledge to inform the design of intuitive touchless selection mechanisms?
Specifically, how sensorimotor knowledge can inform the design of different features of a touchless menu?
For example, how would we invoke a touchless menu.
How would we end a touchless menu selection?
What should be an optimal shape of the touchless menu?
Until now, touchless command-selection techniques have been an extension of what has proven efficient for mouse-based, pen-based, or multitouch interfaces.
However, touchless interaction has a very distinctive feature. It is device less.
In existing touchless menus, user’s hand postures have strictly replaced the mechanics of other input devices. This trend may lead us to a local maxima where we’d fail to explore the potential of touchless interactions beyond the device-based paradigm.
In designing our touchless menu, we wanted to port some successful features of ‘device-based’ menus, but not strictly emulate them.
The outcome of our design exercise was touchless circular menus…
Building upon prior sensorimotor knowledge of making directional strokes in mid-air…
And some successful features of device-based menus, such as reaching a region-of-interest for menu-invocation….crossing using directional strokes for menu selection….and a radial shape
A touchless circular menu relieves users from both recalling a precise vocabulary of hand-postures and strictly complying with them.
Now let us see how the menu works..
To select a menu option, the user makes a directional stroke.
To give feedback, when successfully selected, the menu option changes color.
To mitigate accidental invocation, TCM appear opposite to the user’s direction of approach.
To cancel the invoked menu, the user may continue in her direction of movement, or move away from the menu.
To invoke the menu, the user must reach the region-of-interest of the target.
In hierarchical TCM, to select a command, the user first selects a menu option, and then she accesses the submenu by continuing her trajectory.
Let’s now dig a bit deeper into the design rationale for the touchless circular menus.
As we saw, to trigger the contextual menu, a user must cross the region-of-interest (ROI) of a display object.
The ROI can be of any symmetrical shape around the center of the target, with its size directly proportional to the technique’s sensitivity.
To select a command after triggering the menu, users cross it using a stroke in the command’s direction.
Until the crossing happens, users can cancel TCM by moving in any direction away from the triggered menu.
To allow easy escape routes, we designed the structure of TCM as a semicircular array of options appearing at the top-left or the bottom-right corner of the target.
As users approach the menu, to give them orientation, a trace is drawn connecting the target and the users’ hand position.
To improve users’ pointing performance, the menu options increase
in amplitude as users approached them.
To provide further feedback, menu options changed color when selected by crossing.
Currently, our menu design scales up to two levels …5 x 5……with users performing continuous strokes.
Users first select a root menu. Then to operate the submenu, users change their track and cross another command.
In device-based hierarchical menus, submenus appear in the same direction of the root menu.
Due to the lack of precision and control of freehand movements, TCM require users to make inflections in their continuing trajectories, and thereby avoid accidental command-selections.
We evaluated single-level TCM in two experiments.
TCM is a contextual menu for large displays.
So in our first experiment, we investigated how effectiveness and efficiency of TCM is affected by their triggering locations?
In the second experiment, we compared TCM with linear menus using grab gestures. For the linear menus, users invoked the menu with a grab gesture, and selected a menu –option using another grab gesture.
A grab gesture was defined as an open hand…closing and opening again.
For both these experiments, we had 15 participants, who were sitting away from a large display.
In experiment 1, we had 9 trials and 7 blocks of repetition.
In experiment 2, we had 6 trials and 7 blocks of repetition.
For both the experiments, we used the same sample of participants. They were all right-handed. 4 participants were females. 8 out of 15 participants had prior familiarity with touchless gestures.
And 11 out of 15 participants were below 30 years of age.
As apparatus for our experiment we used a large display and a motion tracking sensor. The large display was built by Fakespace labs…It is 160 inch wide and 60 inch in height..with a total resolution of 15.3 M pixels.
We would like to thank the IU Advanced visualization lab, a UITS Research Technologies division for the use of their facilities.
As the tracking sensor, we used a Kinect for windows.
For both of our experiments, we measured efficiency as the time on task and successful trigger rate.
Successful triggers were When users triggered a menu, and continued to select an option from the triggered menu.
If the menus was dismissed before any option was selected, it was considered an unsuccessful trigger.
We measured effectiveness as error rate….. To measure user satisfaction, users were asked to complete the system usability scale and the NASA task load index.
For experiment 1, we had two hypotheses: we predicted that triggering locations of the TCM will affect the performance time and the error rate.
In our first experiment, we found that Triggering location of TCM significantly affected selection time and successful trigger rate.
However, we did not find a significant effect of the triggering locations on the effectiveness of TCM.
Users had mixed reaction about the menu. While one said “I felt I had to rush to select the menu option.”, another was surprised that he could do so well.
Our findings suggest that arm posture affects users’ control on their hand movements and the required effort.
This is completely in line with another study that was published last month in CHI.
In this study, the authors found that “2D plane locations relative to the user’s body and arm configuration significantly affects the consumed endurance in touchless interactions.”
Now Contextual touchless menus for large displays will always have different menu-triggering locations.
Hence, Menu triggering locations will significantly affect the user experience of contextual touchless menus for large displays.
For experiment 2, we predicted that TCM would be more efficient, and easier to use than linear menus.
In our second experiment, we found that TCM were more efficient and cause lower workload than linear menus with grab gestures. However, TCM were less effective than linear menus.
Again, users had mixed reaction about the linear menu. While a 20-something participant said “This is how I envision using touchless gestures.”…..for a 50-something participant …“It was a lot of effort.”
Our findings suggest that Compared with linear menus using grab gestures, participants using TCM were twice faster in selecting commands and perceived lower workload.
However, TCM caused 3% more errors than linear menus.
So then the question was: “Why TCM were less effective?”
To figure that out we can take a closer look at the mechanics of both linear menus using grab gestures, and TCM.
For linear menus, one grab gesture was registered to invoke the menu, a second grab gesture was registered to select a menu option. The grab gesture was defined as an open hand..closing and opening again.
In between the two gesture registrations, users could move their hand freely without affecting the on-going interaction. That is, user’s cursor would move, but not inadvertently select a menu option.
Hence the gesture relaxation phase made the linear menus more effective.
However, for
TCM, after the menu was triggered, users could inadvertently
move their hand and select a wrong command or..dismiss the menu. Unlike linear
menus, TCM required users to strictly constrain their freehand movements after triggering the menu.
As our future work, we plan to…
To Minimize errors by constraining users’ freehand movements after triggering the TCM.
And to Investigate bimanual gestures to operate menu levels in hierarchical TCM…for which we also plan to conduct empirical evaluations.
So we proposed and evaluated TCM.
Now The final takeaways.
TCM relieve users from learning posture-based commands, and shift the interaction complexity from users’ input to the visual interface.
We found that Menu triggering locations significantly affect the user experience of contextual touchless menus for large displays.
Our results further suggest that Simple directional movements for selecting commands is more efficient and causes lower workload than (hand) posture-based techniques.
Finally, our findings indicate that efficient gesture relaxation techniques may have the potential to increase effectiveness of touchless interactions.
As a delimiter, dynamic gestures would be more
efficient than static poses, as users would not have to halt-and execute
a pose, but fluidly end the selection.