The document summarizes a study investigating how different sensory modalities (visual, auditory) impact immersion in digital games. Participants played a game where they dodged virtual cars using motion controls. Sessions included visual-only, audio-only, and audio-visual feedback conditions. Questionnaires found audio-visual provided the highest immersion. Performance was best with audio-visual and worse with audio-only. Most participants preferred audio-visual and found visual-only muted and audio-only intense. The study suggests achieving immersion requires relatable scenarios and natural inputs, while removing primary senses leads to more intense experiences.
Exploring How Visual and Auditory Feedback Impact Immersion
1. Designing Games for All: Exploring
Output and Immersion.
Investigating the impact of modality perception
on immersion in digital games
2. The Senses
Five Primary Sensory Modalities
• Vision
• Auditory
• Somatosensory (Touch and Proprioception)
• Olfactory (Sense of Smell)
• Gustatory (Sense of Taste)
3. Multimodal Interaction
•
•
Human Interaction and Communication is inherently
multimodal
– An everyday conversation includes more than just
auditory cues…
– A touch can convey a whole idea…
Human Computer Interaction remains limited
– Multimodal Input is common.
• Voice, Touch, Gaze, Motion, Gesture, BMI, etc.
– Output tends to be limited
• Visual, Auditory and occasionally haptic (vibration)
4. Immersion
•
•
•
Subjective by nature
Difficult to achieve and easy to destroy
Engagement is a vital ingredient
– Mental engagement is intensely subjective but promotes
immersion very effectively.
– Physical engagement is easier to achieve and directly
promotes immersion.
5. The importance of immersion
•
•
•
Immersion is an acknowledged ingredient of human crafted
experiences
– Books, Films, Theatre, Games…
The majority of these constructs are limited by their medium
with regards to their form and function
Even when technology comes into play, the extents are
often limited
– 3D Films, Force Feedback
6. The Project
Investigates the role of two sensory modalities, specifically
visual and auditory perception, in achieving effective
immersion and presence.
• Establishing an experimental design to provide balanced and
comparable sets of modal cues.
Do so with natural interfaces for low physical engagement
barriers.
• Performing a user study that would evaluate the modalities
both in isolation and in combination in order to study the
impact of each modality.
7. The Experiment
What did we do?
• We made participants dodge cars.
• Sometimes while blindfolded.
8.
9.
10. The Setup
•
•
•
Simulated a collision avoidance scenario
– Virtual Cars on a highway
– Mental Engagement
Multimodal Input
– Motion sensing technology (Kinect)
– High Physical Engagement
Multimodal Output
– Visual and Auditory Feedback
11.
12.
13.
14.
15. The Specifics
•
•
The Participants where asked to dodge the incoming cars for
as long as possible.
Nine measured sessions in total
– 3 With both visual and auditory feedback
– 3 With only visual feedback
– 3 With only auditory feedback
16. Data Collection and Evaluation
Primary desired data element:
• Qualitative Immersion Measures
– Method: IMEX questionnaire
Secondary data element
• Quantitative performance data
– Participant survival time per session
17. Quantitative Results - Immersion
Mean IMX score on 0-100 scale
90
The analysis showed that the modal output had a
significant impact on immersion.
80
70
60
50
Both Modalities
40
Audio Only
30
Visual Only
20
10
0
General
Immersion
Action Visceral Mental Visceral
18. Quantitative Results Performance and Preference
•
•
Performance was measured by the survival time of each
participant in each play through
– Insignificant difference between Audio-Visual and Visualonly sessions
– Significant difference between Audio-Visual and Audioonly sessions – as was expected
With regards to preference
– 75% preferred the Audio-Visual version
– 25% preferred the Audio-only version
– Noce preferred the visual only version
19. Qualitative Results & User Feedback
Overall, feedback from the users shows promising results:
• Positive reactions to the experience
• High levels of Immersion
• Interesting responses regarding value of audio
– A/V is optimal
– Visual only is “muted” and “disconnected”
– Audio only is “intense” and “thrilling”
20. Qualitative Results & User Feedback
Three themes surfaced:
• Rapid Immersion
– Physical and Mental engagement equals rapid immersion
– “Damn this is intense!”
• The importance of sound
– Removing a “secondary” modality has an impact
– “Something was missing, I was waiting for the crunch.”
• Engaging audio experience
– Removing a “primary” modality has a severe impact
– “That was absolutely terrifying! I was all there!”
21. Conclusions
•
The Starting point
– The objective: Design and implement a highly immersive
digital experience
– The hypothesis: Unimodal and multimodal experiences
create different levels of immersive response.
•
The findings
– Achieving Immersion: Relatable scenarios and natural
input greatly promote immersion
– Unimodal Output: Removing primary senses leads to
intense experiences
22. Potential Application
•
•
•
Enhanced Entertainment Experiences
Games Designed for All
– Vision Impairment
• AudioGames.net
– Dexterity/Mobility Impairment
• OneSwitch.org.uk
Games Designed for All vs. Accessible Games
http://www.gameaccessibilityguidelines.com
23. The next step
•
•
•
•
Contextual modality suppression
– Scenario based sensory deprivation
Further immersion enablers
– Oculus rift
Introduce additional output modalities
– Haptic feedback
Investigate immersive scenarios