Richard Skarbez presented a seminar titled "Cognitive Illusions in Virtual Reality: What do I mean? And why should you care?" as part of the SMART Seminar Series on the 4th March 2019. More information: https://news.eis.uow.edu.au/event/cognitive-illusions-in-virtual-reality-what-do-i-mean-and-why-should-you-care/ Keep updated with future events: http://www.uoweis.co/events/category/smart-infrastructure-facility

1. latrobe.edu.au
La Trobe University CRICOS Provider Code Number 00115M
Cognitive Illusions in Virtual Reality:
What do I mean?
And why should you care?
Rick Skarbez
4 March 2019

2. latrobe.edu.au
Slide 2
 VR systems try, to varying extents and
with varying levels of success, to replace
the sensory input we normally receive
from the real world with computer-
generated sensory input
– I argue that if a system doesn’t try to do
this, it shouldn’t be called VR
 VR aspires to deliver “the ultimate
display”
To me, VR means sensory immersion

3. latrobe.edu.au
Slide 3
 Display as a window into a virtual world
 Improve image generation until the picture looks
real
 Computer maintains world model in real time
 User directly manipulates virtual objects
 Manipulated objects move realistically
 Immersion in virtual world via head-mounted
display
 Virtual world also sounds real, feels real
“The Ultimate Display”
Sutherland, 1968

4. latrobe.edu.au
Slide 4
The Sword of Damocles

5. latrobe.edu.au
Slide 5
 2 slides ago, I said “VR systems try, to varying extents and with varying
levels of success, to replace the sensory input we normally receive from the
real world with computer-generated sensory input”
– Basically, immersion is the “…varying extents and with varying levels of
success…”
• You can think of it as the answer to the question, “How close does this VR system
get to the Ultimate Display?”
(Sensory) Immersion

6. latrobe.edu.au
Slide 6
 More precisely, I prefer to define immersion as the set of all valid
sensorimotor and effective actions supported by a virtual experience
– Valid: Having the intended effect
– Sensorimotor actions: Actions which change a user’s perception of the virtual
environment
• I.e., turning your head, taking a step
– Effective actions: Actions which change the state of the virtual environment
• I.e., picking up an object, flipping a switch
Immersion

7. latrobe.edu.au
Slide 7
 Immersion is an objective characteristic of a given VR system
– It should not vary with different users
 It is determined by the input and output devices and interface techniques of
a VR system
– ”This system has so-and-so field of view, and such-and-such resolution, and
supports locomotion by real walking…”
 It is NOT determined by the actual virtual environment being displayed
– Whether it’s a detailed environment or a sparse one, a big one or a small one, a
pretty one or an ugly one, makes no difference for immersion
Immersion

8. latrobe.edu.au
Slide 8
Immersion
Model
Computer
System
Rendering
Software
Processing
Software
Display
device(s)
Input
device(s)
USER
This is the realm of
immersion.
“How ‘good’ are
these components?”

9. latrobe.edu.au
Slide 9
What else is there?
Model
Computer
System
Rendering
Software
Processing
Software
Display
device(s)
Input
device(s)
USER
This is the realm of
immersion.
We haven’t talked at
all yet about this…
…or this.

10. latrobe.edu.au
Slide 10
 Back at the beginning of this talk, I argued that a high level of immersion is
needed for me to consider an experience virtual reality
 VR is just a medium, a way to communicate
 The choice of medium doesn’t inherently make what is being
communicated good or bad
– There are good video games and bad video games, there are good movies and bad
movies, there are good novels and bad novels, etc.
What do you need for good VR?

11. latrobe.edu.au
Slide 11
 Interesting/believable characters
 Interesting plot
 Beautiful imagery
– Characters, locations, cinematography,
computer generated imagery, etc.
 Thought-provoking
– Give you a new way of thinking about
things
 Emotion-inducing
– Make you feel something
 Good acting
 Enjoyable game mechanics
 Good music
 Sense of accomplishment
 An experience shared with friends
 Novelty
 “Winning”
An incomplete list of characteristics of
enjoyable media experiences
Q: How many of these depend on
immersion?
A: Arguably, none.

12. latrobe.edu.au
Slide 12
 You need all these sorts of squishy, “intangible” aspects of the experience
to be good
 So we won’t talk too much more about these aspects of designing virtual
reality in this talk
 However, there are other goals for VR than having a “great experience”
– VR is not always supposed to be fun. Often, it is supposed to be useful.
How to make a truly great VR experience
is, sadly, beyond the scope of this talk

13. latrobe.edu.au
Slide 13
 One way of evaluating virtual reality experiences is by asking, “Do users
react in the same way to the VR as they would having the same experience
in the real world?”
– This doesn’t apply to every possible VR experience, but it does apply often enough
to be useful
React-as-if-real (RAIR)

14. latrobe.edu.au
Slide 14
 Let’s assume that we are dealing with an application where you want a user
to react-as-if-real (RAIR)
– What does that take?
 Slater theorizes that a user will react to a VR environment as if it is real if
– The user believes that they are in the VR environment, and
– The user believes that the events happening in VR are real
 “If you are there…and what appears to be happening is really
happening…then this is happening to you!”
What does it take to RAIR?

15. latrobe.edu.au
Slide 15
 Let’s continue to assume that we are dealing with an application where you
want a user to react-as-if-real (RAIR)
 Now we have two goals:
– Convince the user that they are somewhere else
– Convince the user that what they are seeing makes sense
Goals for VR design and development

16. latrobe.edu.au
Slide 16
 Slater: Place Illusion (or PI) is “The illusion of being in a place in spite of the
sure knowledge that you are not there.”
 Why might someone feel PI in VR?
– Because of immersion
• When I turn my head, do I see my living room, or do I just see more of the virtual
world? When I take a step, do I move in my living room, or in the virtual world? If I
raise my hand, do I see my hand raise in the virtual world, or is there no change?
 “Ultimate Display” = perfect immersion ⇒ guaranteed Place Illusion
– However, in the real world, we don’t have the ultimate display, immersion is never
perfect, and a given user may or may not experience PI in our virtual environments
Place Illusion (PI)

17. latrobe.edu.au
Slide 17
 So, if we had the Ultimate Display, we would expect every user to feel PI
every time
 But, practically, we don’t, and so for any VE, some users will feel PI, and
some won’t. Why?
PI and individual users

18. latrobe.edu.au
Slide 18
 Familiarity/comfort with interface techniques: Objectively, tilting a joystick
to move your viewpoint is much less natural than moving your head. But it is
much more natural for a user who plays a lot of console video games than
one who doesn’t.
 Amount of exposure to the interface: Imagine two users, A and B. A is
fascinated by their environment, and looks around constantly, while B
almost never moves their head. Now imagine we put A and B in a virtual
reality system that does not have head tracking. A is likely to notice this
immediately, while B might never notice.
– Alternatively, just consider one user who has half an hour to look around, and
another that has 30 seconds. Who is likely to discover the limits of the system?
Why some users feel more PI

19. latrobe.edu.au
Slide 19
 Inherent individual differences: Some people get more motion sick than
others, for example.
Why some users feel more PI

20. latrobe.edu.au
Slide 20
 As humans, our assumption is usually that we are experiencing perfect
immersion until proven otherwise
 So, most users of VR will start out feeling Place Illusion, until they try to do
something that is not supported by the system
– This results in a so-called break in PI
 Our goal as designers should be to minimize breaks in PI
– How?
• Build systems with higher levels of immersion
• Employ interaction techniques that keep users “safe”
Breaks in PI

21. latrobe.edu.au
Slide 21
 Slater: Plausibility Illusion (or Psi) is “The illusion that what is apparently
happening is really happening (even though you know for sure that it is not)”
 Why might someone feel Psi?
Plausibility Illusion (Psi)

22. latrobe.edu.au
Slide 22
 On October 30, 1938, an American radio
broadcaster aired a radio play by Orson Welles,
“War of the Worlds.”
 This radio play, presented in the form of a news
broadcast, depicted a Martian invasion of planet
Earth.
 Reportedly*, millions of people believed that the
invasion was real, setting off a panic in the US.
– Exactly what we mean by Psi!
”War of the Worlds”
* This almost certainly didn’t actually happen, but it’s too
good an illustration to turn down.
https://tinyurl.com/myvgs9p

23. latrobe.edu.au
Slide 23
 So, what is it about a media experience that makes it believable (or not)?
– Events make sense given prior knowledge and expectations
• War of the Worlds “worked” because it presented itself as a news broadcast
• VR often “works” because it presents itself as the real world
 I call the extent to which events make sense given prior knowledge and
expectation coherence
 Much like someone feels PI because of immersion, they might feel Psi
because of coherence
 Unlike immersion, coherence is NOT a function of the physical
implementation of the VR
– It is a function of what is being displayed in VR, and user expectations
Coherence

24. latrobe.edu.au
Slide 24
 Like the more precise definition of immersion offered earlier, I similarly
propose to define coherence as the set of objectively reasonable
circumstances that can be demonstrated by the scenario without
introducing objectively unreasonable circumstances
– Objectively reasonable circumstances: states of affairs that are self-evident given
prior knowledge provided in the context of the virtual experience
• For better or worse, we don’t have control (or knowledge) of users’ prior life
experiences, so this is the best we can do
- Or is it?
Coherence

25. latrobe.edu.au
Slide 25
 When designing VR experiences, we generally cannot control
every aspect of user expectations
– People have different life experiences, after all
 But think about what theme parks do
– At right is Disney’s ”Expedition Everest”
– From the moment you get in line, they’re showing
you things to convince you you’re not in Florida
waiting to ride a rollercoaster, but in Nepal waiting
to go on an adventure to the “forbidden mountain”
 What you show and tell users before they put on
the headset can impact their experience in VR!
Managing user expectations

26. latrobe.edu.au
Slide 26
Context matters!

27. latrobe.edu.au
Slide 27
Context matters!

28. latrobe.edu.au
Slide 28
Immersion and coherence, illustrated
Low Medium High
Low
Medium
High Playing tennis
Immersion
Low Medium High
Low PONG
Medium
High Playing tennis
Low Medium High
Low PONG
Medium Wii Tennis
High Playing tennis
Low Medium High
Low PONG Watching on TV
Medium Wii Tennis
High Playing tennis
Low Medium High
Low PONG Watching on TV
Medium Wii Tennis
High “Tennis Simulator 1” Playing tennis
Low Medium High
Low PONG Watching on TV
Medium Wii Tennis “Tennis Simulator 2”
High “Tennis Simulator 1” Playing tennis
Low Medium High
Low PONG Watching on TV
Medium Wii Tennis “Tennis Simulator 2”
High Whatever this is “Tennis Simulator 1” Playing tennis
Coherence

29. latrobe.edu.au
Slide 29
 I’ve run several experiments investigating the relationship between
immersion and coherence
– Consistently, there is an interaction between the two, such that users have the best
experience when BOTH are high, more than can be predicted by the sum of the two
effects
– More enjoyment, more presence, etc.
Immersion AND coherence is best

30. latrobe.edu.au
Slide 30
 So far, I’ve treated immersion and coherence as totally separate constructs
– This may not be totally accurate
 One example: Having a body can be an aspect of both
– A body that moves as I move implies very sophisticated interaction techniques
(immersion)
– A body that moves as I move also implies that the environment behaves as I expect
(coherence)
The body: Where immersion and
coherence meet

31. latrobe.edu.au
Slide 31
 If we want users to react-as-if-real, we should maximize immersion and
coherence
– Immersion: Based primarily on hardware and interface techniques
– Coherence: Based primarily on software and user expectations
 If we have high immersion and high coherence, users probably have PI and
Psi
– Not guaranteed though!
• Every user is different
 And if they feel PI and Psi, they will probably react-as-if-real
Our best shot at RAIR

32. latrobe.edu.au
Presence?

33. latrobe.edu.au
Slide 33
You’ve probably heard the term
“presence”
AMD Oculus
(Facebook)
Leap Motion
Sony

34. latrobe.edu.au
Slide 34
 Sense of “being there”
– “The subjective experience of being in one place or environment, even
when one is physically situated in another”
 Suspension of disbelief
– “(Suspension of dis-)belief that [one] is in a world other than where [one's]
body is located”
 Not noticing the intervening technology
– “the perceptual illusion of nonmediation”
What do we mean by presence

35. latrobe.edu.au
Slide 35
The ”official” definition of presence
Presence (a shortened version of the term telepresence) is a psychological state or
subjective perception in which even though part or all of an individual's current
experience is generated by and/or filtered through human-made technology, part or all
of the individual's perception fails to accurately acknowledge the role of the technology
in the experience. Except in the most extreme cases, the individual can indicate
correctly that s/he is using the technology, but at *some level* and to *some degree*,
her/his perceptions overlook that knowledge and objects, events, entities, and
environments are perceived as if the technology was not involved in the experience.
Experience is defined as a person's observation of and/or interaction with objects,
entities, and/or events in her/his environment; perception, the result of perceiving, is
defined as a meaningful interpretation of experience.
(International Society for Presence Research, 2000)
Books? Atari?
Phone calls? Skype?
The Holodeck?

36. latrobe.edu.au
Slide 36
 It has a bunch of definitions
 These definitions are not necessarily useful
 These definitions don’t necessarily agree
 Presence is “a buzzword for what virtual reality can offer”
– So, let’s use it like that
One man’s opinion on the term “presence”

37. latrobe.edu.au
Slide 37
 Let presence be, “The perceived realness of a virtual experience”
– And use other terms (like Place Illusion and Plausibility Illusion) when you
need to be clear and specific
 Note the focus on perceived realness
– Actual realness, on the other hand, would be a function of a system’s
ability to provide stimuli that match reality
• That is, it would be a function of immersion and coherence
My definition of presence

38. latrobe.edu.au
Slide 38
 Plausible interaction with virtual characters is an important special case of
Psi
– Copresence Illusion
• The sense of being together with another or others in a virtual space
– Social Presence Illusion
• The moment-by-moment awareness of the copresence of another
sentient being combined with a sense of engagement with them
– Company
• The existence of a (seemingly) sentient other in an environment
Sidebar: Social experiences in VR

39. latrobe.edu.au
Slide 39
Putting it all
together
Quite a rich history of
research here (Most
people who historically
studied “presence”
actually focused on what
we call Place Illusion
Lots of research here as
well – many social
psychologists and media
theorists have done
research in this area
To the best of my
knowledge, there has
been almost no research
in this area - into what it
takes for people to
believe a mediated reality
And there has been even
less research here – how
these different
components of
“realness” add up to an
overall experience

40. latrobe.edu.au
Fidelity and realism

41. latrobe.edu.au
Slide 41
 Fidelity (for the purposes of VR) is defined as “the extent to which the
virtual environment emulates the real world”
 Can talk about:
– Physical fidelity: The simulation looks, sounds, and feels like the real environment
– Functional fidelity: The simulation acts like the real environment in reacting to the
user
– Psychological fidelity: The simulation replicates the psychological factors
experienced in the real environment
 Comes from the world of training, e.g. flight simulators
Fidelity

42. latrobe.edu.au
Slide 42
 Physical fidelity ≈ immersion
 Functional fidelity ≈ coherence
 Psychological fidelity ≈ presence (?)
– I am confident in the first two, but the third is somewhat of an open question, in my
opinion
• If, in fact, presence ≈ psychological fidelity, that would be the best argument to
date for the widespread adoption of VR training
Fidelity

43. latrobe.edu.au
Slide 43
 Despite the fact that there are clear analogies between
fidelity and immersion, coherence, PI, Psi, presence, etc., I
prefer to use the latter terms rather than the former
 The definitions of fidelity talk about fidelity of the
simulation with respect to the real environment
– VR doesn’t have to model a real environment!
– It’s “a looking glass into a mathematical wonderland”!
 So, while a VR experience might be fantastical, it can (and
should) still be coherent. But it cannot have functional
fidelity.
One quibble about fidelity w/r/t VR

44. latrobe.edu.au
Slide 44
 I would argue that for a VR experience, more immersion and more
coherence are always better than less of either
 But what about realism?
– In the sense of “realistic graphics” or “realistic physics”, not in the sense of “Does
this VR experience attempt to accurately portray an experience in the real world?”.
Is realism necessary?

45. latrobe.edu.au
Slide 45
VR can succeed without (much) realism
DisneyQuest Aladdin:
Successfully fun
UNC Pit:
Successfully stressful
Virtual Vietnam:
Successfully therapeutic

46. latrobe.edu.au
Slide 46
 Short answer: We don’t know
– This is an open research question
 Longer answer: It may not matter; we’re getting more realism, whether we
need it or not
– Every year, processors and graphics cards get faster, models have more triangles,
textures have higher detail, etc.
• If you want more realism, you just have to wait
• But VR works without super-high realism
 So maybe the better question would be, is more realism ever worse?
Is realism necessary? Is realism better?

47. latrobe.edu.au
Slide 47
 Cost (actual money, developer time, artist time)
 Complexity (model sizes, texture sizes, algorithms, CPU/GPU time)
– At any given time, getting more realism is not free. There are real costs associated with it.
 Can be a distraction as a designer/developer
– You might be familiar with user stories, a process from agile development. You’ll never see anything
like, “As a user, I need my avatar to have at least 1.5 million triangles because something
something.”
 Can be a distraction to a user
– Do I need to see every leaf on every tree if my task is to learn chainsaw safety?
 The “uncanny valley” (maybe)
Downsides of realism

48. latrobe.edu.au
Slide 48
 Originally from robotics
 As a [humanoid thing] gets more realistic,
generally speaking, we like it more. But, there
comes a point where it looks almost-but-not-
quite real, and at that point, we do not like it
much at all.
– Not totally clear if this is sometimes true, or
always true, or rarely true. But that’s the
hypothesis, roughly speaking.
The uncanny valley hypothesis
≈

49. latrobe.edu.au
Slide 49
The uncanny valley hypothesis

50. latrobe.edu.au
Less engineering, more imagineering

51. latrobe.edu.au
Slide 51
 As developers, we often think of ourselves
as engineers, builders, or toolsmiths
– And in many (most?) cases, that is exactly right
 But sometimes, it can be helpful to take a different
perspective
 When we’re building virtual reality applications, we’re really building experiences
– They may also be tools, as in training applications, but they are certainly experiences
 From this perspective, I think we can learn some things from other experience designers
– In particular, designers of theme park attractions
Engineering vs. “Imagineering”

52. latrobe.edu.au
Slide 52
1. Know your audience
– Present information at an appropriate level
2. Wear your guest's shoes
– Try the experiences yourself, and don’t forget the human factors!
3. Organize the flow of people and ideas
– Ensure experiences tell a story that is organized and logically laid out
4. Create a weenie (Walt Disney's term for a "visual magnet")
– Lead your users from one part of the experience to the next
5. Communicate with visual literacy
The ten commandments of imagineering

53. latrobe.edu.au
Slide 53
6. Avoid overload
– Don’t feel the need to show everything at once
7. Tell one story at a time
– Each virtual experience should be about just one ”big idea”
8. Avoid contradictions—maintain identity
– Following from (7). If it’s not what your VE “is about”, leave it out
9. For every ounce of treatment, provide a ton of treat
– Take advantage of the distinction of virtual reality, which is that it encourages active participation,
compared to passive entertainment
10. Keep it up!
The ten commandments of imagineering

54. latrobe.edu.au
Slide 54
 Immersion : Place Illusion :: Coherence : Plausibility Illusion
– I prefer to use these terms instead of “presence”
 Lots of research into immersion, very little into coherence
 Immersion and coherence TOGETHER is best
– They naturally come together in the virtual body
 When building VEs, think imagineering before engineering
TL; DR

55. La Trobe University
CRICOS Provider Code Number 00115M © Copyright La Trobe University 2018
latrobe.edu.au
Thank you. Be well.