1. Semantics in Visual Perception:
Methodological Remarks,
Issues and Perspectives
Jurgis Skilters
Center for Cognitive Sciences and Semantics
University of Latvia, Latvia
Thanks to my collaborators and colleagues:
Baingio Pinna (Sassari, Italy),
4-5.11.2010, Aarhus,
Jānis Pencis (CCSS, Riga)
Methods in Language&Cognition 1
Denmark

2. The aim of my talk
• to provide some methodological
preliminaries and some parameters for
research on the interaction between visual
perception and semantics.
4-5.11.2010, Aarhus, Methods in Language&Cognition 2
Denmark

3. interference effects between
language and visual perception:
• Language is an access to human
cognition.
• Visual perception is another access to
human cognition
• ....
4-5.11.2010, Aarhus, Methods in Language&Cognition 3
Denmark

4. interference effects between
language and visual perception:
• we do have representations of linguistic
stimuli
• we do have certain representations of
visual stimuli.
• are they the same representations?
• What does happen when we look at visual
stimuli and when we process (either
produce or comprehend) language?
4-5.11.2010, Aarhus, Methods in Language&Cognition 4
Denmark

5. Outline:
1. Visual perception and the role of psychophysics
2. How does semantics arise in perception?
3. Invariants
4. Affordances
5. Events
6. Experience
7. Objects: the basic principles of their perception
8. Perceived causality, motion perception
9. Semantics in Vision: Multimodality effects
10. Remarks on spatial cognition
11. Methods from semantics and perception (brief overview)
12. Conclusion: Constraints
4- Methods in Language&Cognition 5
5.11.2010, Aarhus, Den

6. 1. Visual perception and the
role of psychophysics
4-5.11.2010, Aarhus, Methods in Language&Cognition 6
Denmark

7. • The methods we chose depend on our
research motivation.
• Relationships between stimuli and
physiological response Φ
• Relationships between stimuli and
perceptual reaction ψ
• linking relationship (L) between perceptual
and physiological response.
4- Methods in Language&Cognition 7
5.11.2010, Aarhus, Den

8. Cp. Goldstein, 2005
Stimuli
Φ ψ
Physiological reaction L perceptual reaction
4-5.11.2010, Aarhus, Methods in Language&Cognition 8
Denmark

9. Stimuli
Φ ψ
Physiological reaction L perceptual reaction
4-5.11.2010, Aarhus, Methods in Language&Cognition 9
Denmark

10. • How are relations ψ studied and explored?
• methods formulated by Fechner (1860) to
determine thresholds,
• methods for magnitude estimation (above the
threshold formulated by Stevens (1961)),
• phenomenological observations
(Katz, 1935, Metzger 2006/1936);
• also identification
measures, recognition, reaction time research….
Also overlapping with Φ
4-5.11.2010, Aarhus, Methods in Language&Cognition 10
Denmark

11. • there is no direct correspondence between
stimuli and perceptual result:
– different stimuli can cause the same perception and
(physically) the same stimulus in different contexts
can cause different perceptual results.
– we can perceive something without the sensorial
basis for it,
• e.g., we can perceive movement / causality / motion where
there is no movement at the level of physical stimuli.
And this has nothing to do with semantics but with
the principles of our visual perception.
4-5.11.2010, Aarhus, Methods in Language&Cognition 11
Denmark

12. the binding problem:
• how is a unitary perceptual experience
generated that combines qualities such
as color, shape, location, orientation
(Goldstein, 2005, 12)?
4-5.11.2010, Aarhus, Methods in Language&Cognition 12
Denmark

13. Most of the models are hierarchical:
• top level encodes the whole / figural representation;
• subsequent / lower levels encode parts and relations
between them (Palmer, 1975, 1977, 1978, Carlson-
Radvansky & Irwin, 1995)
• 1. Grouping
• 2. shape assignment
• 3. meaning assignment
• there are also parallel models assuming that
object memories critically determine object recognition
and also grouping and shape assignment factors.
• NB: grouping and shape assignment processes function
without attention, intention and awareness.
4-5.11.2010, Aarhus, Methods in Language&Cognition 13
Denmark

14. a. Classical serial hierarchical theory (Marr) (picture from Peterson,
2005)
4-5.11.2010, Aarhus, Methods in Language&Cognition 14
Denmark

15. b. interactive hierarchy model (picture from Peterson, 2005)
4-5.11.2010, Aarhus, Methods in Language&Cognition 15
Denmark

16. c. Parallel interactive model (picture from Peterson, 2005)
4-5.11.2010, Aarhus, Methods in Language&Cognition 16
Denmark

17. experience
stimuli Grouping Shape
meaning
assignment
situation
4-5.11.2010, Aarhus, Methods in Language&Cognition 17
Denmark

18. 2. How does semantics
arise in perception?
4-5.11.2010, Aarhus, Methods in Language&Cognition 18
Denmark

19. 1. visual field is segmented, the components are grouped
together;
2. shape assignment takes place
3. semantics is assigned.
• And finally….. the result is lexicalized.
• NB all these processes are interactive and there is not
one way of bottom-up or top-down determination.
• NB we „translate‟ continuous perceptual material into
discontinuous language. As soon as we lexicalize we
make the continuous discontinuous.
4-5.11.2010, Aarhus, Methods in Language&Cognition 19
Denmark

20. Gouping Shape
Concrete stimuli
assignment Concepts language
Meaning
assignment
meanings
Visual processing
Continuous Discontinuous /
4-5.11.2010, Aarhus, Methods in Language&Cognition discrete 20
Denmark

21. Grouping
Rows vs. columns and the rectangle illusion.
4-5.11.2010, Aarhus, Methods in Language&Cognition 21
Denmark

22. The role of directional symmetry in forming
the shape
The inverted rod and frame illusion
4-5.11.2010, Aarhus, Methods in Language&Cognition 22
Denmark

23. The directional symmetry influences the shape
The pointing illusion.
4-5.11.2010, Aarhus, Methods in Language&Cognition 23
Denmark

24. • Further, we assign border / contour to the
figural regions. We discriminate figure
from ground.
4-5.11.2010, Aarhus, Methods in Language&Cognition 24
Denmark

25. From grouping to shape
4-5.11.2010, Aarhus, Methods in Language&Cognition 25
Denmark

26. • And finally we assign meaning to what we
see.
– We simply do not stop the perceptual activity
after grouping and shape-building
4-5.11.2010, Aarhus, Methods in Language&Cognition 26
Denmark

27. Form of meaning: More complex results
The illusion of meaning: A square showing different kinds of happenings
4-5.11.2010, Aarhus, Methods in Language&Cognition 27
Denmark

28. 3. Invariants
4-5.11.2010, Aarhus, Methods in Language&Cognition 28
Denmark

29. There is consensus among vision
scientists:
• at least some structures remain invariant across
changes (cp. Carlson-Radvansky & Irwin, 1995, Pinna,
Tanca, & Skilters, 2010).
• Every type belongs to a different level of perceptual
processing and every level contains invariants of a
corresponding type.
– According to certain invariants from type „grouping‟ the
perceptual material (from zero-level objects) is transformed to
higher level and processed according to the type „shape‟ and,
finally, to meaning.
• when one unit (structural invariant in our case)
determines another, the one must belong to the higher
level (type) than the other (cp. Jeffrey 1981).
4-5.11.2010, Aarhus, Methods in Language&Cognition 29
Denmark

30. • invariants itself are not objects of the
particular levels; they are principles that
structure the organization of particular
visual material.
4-5.11.2010, Aarhus, Methods in Language&Cognition 30
Denmark

31. • Every level of perception contains structural
invariants determining the processing of lower
level.
• The third-level invariants – meaning – are the
strongest
• the structural invariants at the level of meaning
assignment are the strongest
• dissimilarities and multiplicities in lower level
structures become units and their dependencies
become stronger with every higher level of
processing.
• strength of functional dependencies increases
from lower to higher level
4-5.11.2010, Aarhus, Methods in Language&Cognition 31
Denmark

32. 4. Affordances
4-5.11.2010, Aarhus, Methods in Language&Cognition 32
Denmark

33. • meaning as an affordance-sensitive
structure.
• affordances are dispositional properties of
objects and events.
• „Affordances are the acts or behaviors
permitted by objects, places, and
events.“ (Michaels & Carello, 1981, 42).
• affordances differs in different organisms
4-5.11.2010, Aarhus, Methods in Language&Cognition 33
Denmark

34. • two mutually connected aspects: action system
and environment.
– „The action system (effectivity structure) and the environment
(affordance structure) are in a relationship of mutual
constraint.“ (Michaels & Carello, 1981, 54, cp. also. Shaw,
Turvey, & Mace, 1982, 209)
• Affordances are processed to a large part un-
intentionally: objects potentiate a range of
actions associated to them but irrespective of
intentions (Ellis &Tucker, 2000).
• the affordances change from one moment or
situation to another.
– more permanent and more time-dependant („episodic“, cp.
Glenberg et al. 2009) affordances can be gradually distinguished.
4- Methods in Language&Cognition 34
5.11.2010, Aarhus, Den

35. • Meaning “consists of the set of actions the individual can
undertake in that situation. The set of actions is
determined by the goal-directed mesh of affordances.”
(Glenberg et. al., 2007, 223).
– In a particular situation an individual modifies the meaning
according to his/her experience and bodily configuration.
• Affordances constrain meaning but affordances are
constrained by the actual situation and experiential
factors of the agent as well.
• the situation determines meaning; differences in
affordances result in differences in meaning.
4-5.11.2010, Aarhus, Methods in Language&Cognition 35
Denmark

36. 5. Events
4-5.11.2010, Aarhus, Methods in Language&Cognition 36
Denmark

37. Preliminary
• Have to be distinguished:
• optic flow: when observer moves through
environment, where entire field of view is
determining,
• and
• perception of object motion: looking at a
point in visual field, where only limited
segments of visual field are co-involved
(cp. Shiffrar, 2005).
4-5.11.2010, Aarhus, Methods in Language&Cognition 37
Denmark

38. • Visual perception takes place according to
events:
• Events are time-space-linked structures
• “If events are the significant units of the
world, the world must be described in a
way that preserves their integrity. The
world must be described in terms of both
time and space.” (Michaels & Carello,
1981, 10)
4-5.11.2010, Aarhus, Methods in Language&Cognition 38
Denmark

39. • Events are cognitively perceived as goal-
oriented patronymic hierarchies (Zacks,
Tversky, & Iyer, 2001)
• and later lexicalized in object/individual-
predicate (object-action) structure (guiding
hypothesis of Pencis, Pinna, & Skilters,
2010).
4-5.11.2010, Aarhus, Methods in Language&Cognition 39
Denmark

40. • events as segments of time at a given location,
perceived as having a beginning and an end, cp.
Zacks, Tversky, & Iyer, 2001;
– based on partonymic hierarchies.
• Even material presented in pictures is perceived
in time and according to hierarchic event-
structure.
– recognition of intentional action on objects is needed
(Tversky, Zacks, & Hard, 2008, 461).
4-5.11.2010, Aarhus, Methods in Language&Cognition 40
Denmark

41. Correlation of three processes
(1) decomposition of visual material in spatial parts
(2) decomposition of events in temporal parts
(3) lexicalization.
functional dependencies and relations are generated.
hierarchical bias hypothesis: people are spontaneously
disposed to actively encode ongoing activity in terms of a
hierarchical part-whole structure; Zacks, Tversky, & Iyer,
2001
4-5.11.2010, Aarhus, Methods in Language&Cognition 41
Denmark

42. • Goal/sub-goal relationship
• a single goal in sense of motion, causality is assigned to
a visual representation even in case of minimal stimuli
(even in case where there is no objective goal or motion
or causality at all).
• In a narrative we can observe hierarchic event
perception more explicitly (cp. also Abbott, Black, &
Smith, 1985, Tversky, Zacks, & Hard, 2008). Faster
narrative production is if there is one higher level event
(Foss & Bower, 1986, Zacks, Tversky, & Iyer, 2001)
• parts as visual segments and parts as sub-goals.
• parts are functionally different
• fast inference from visual appearance to function
• link between perception and function is always present
even in case of minimal stimuli
– (for the perception-to-function hypothesis cp. Tversky, Zacks, &
Hard, 2008)
4-5.11.2010, Aarhus, Methods in Language&Cognition 42
Denmark

43. • Objects are integral and crucial to perceive
actions and vice versa (Baldwin & Baird, 1999)
• actions and objects are comprehended together
(Zacks, Tversky, & Iyer, 2001);
• we always perceive object-activity-
constellations.
• actions and objects are mutually constraining
(Tversky, Zacks, & Hard, 2008)
– Object-action matrices are occurring in events and
allow only certain combinations of substituted
instances.
4-5.11.2010, Aarhus, Methods in Language&Cognition 43
Denmark

44. 4-5.11.2010, Aarhus, Methods in Language&Cognition 44
Denmark

45. 6. Experience
4-5.11.2010, Aarhus, Methods in Language&Cognition 45
Denmark

46. (a) interrelations between visual perception and
experience vary according to the material under
consideration
(b) and more crucially – experience effects are not
atomistic / summative, simplistic but rather
holistic.
the question is not whether but rather how
experience influences visual perception.
• Meaning assignment as resonation of visual
stimulus with experiential structures.
4-5.11.2010, Aarhus, Methods in Language&Cognition 46
Denmark

47. 7. Objects: the basic principles
of their perception
4-5.11.2010, Aarhus, Methods in Language&Cognition 47
Denmark

48. • what counts as an object?
• Conceptions of object recognition
1. we recognize objects by components. I.e., there are
some core elements of visual objects (geons) which are
basic for building together any complex object.
– research tradition by Biederman, Marr
– One consequence from Recognition by Components Theory is
that object recognition is viewpoint invariant. It might be true but
if this is the case we would perhaps see no difference between
pigs and dogs.
2. Multiple Views Theory
– tradition of Buelthoff, Edelman and Tarr
• there are multiple two-dimensional views that enable
recognition of object. Object recognition is view-
dependant because objects observed in new views have
to undergo some time-consuming process before they
are matched to stored views and recognized.
4-5.11.2010, Aarhus, Methods in Language&Cognition 48
Denmark

49. 8. Perceived
causality, motion perception
4-5.11.2010, Aarhus, Methods in Language&Cognition 49
Denmark

50. • Even when there is not causality at the level of
stimuli
there is causality at the level of perception.
• Integration of motion
• there is a tendency to assign intentionality to
„moving‟ objects even if they are simple
geometric stimuli. (For a single stimulus
approach cp. Pinna & Skilters, for a classic study
of moving geometric objects cp. Heider &
Simmel, 1944.)
4-5.11.2010, Aarhus, Methods in Language&Cognition 50
Denmark

51. • In case of animated objects
• Heider, F. & Simmel, M. (1944). An
experimental study of apparent behaviour.
American Journal of Psychology, 13,
1944.
• http://www.all-about-
psychology.com/fritz-heider.htm or
• http://www.youtube.com/watch?v=sZB
Ker6PMtM
4-5.11.2010, Aarhus, Methods in Language&Cognition 51
Denmark

52. Unanimated / single stimuli
4-5.11.2010, Aarhus, Methods in Language&Cognition 52
Denmark

53. 4- Methods in Language&Cognition 53
5.11.2010, Aarhus, Den

54. 4-5.11.2010, Aarhus, Methods in Language&Cognition 54
Denmark

55. 4-5.11.2010, Aarhus, Methods in Language&Cognition 55
Denmark

56. • One object according to the reports of naïve
perceivers is causing changes in another object.
• Our memory for the spatial location of an object
is biased towards objects motion even if there is
no motion or even if the object is static or
geometric (cp. Shiffrar, 2005, 257),
• moreover, the motion assigned to non-animated
objects is frequently like human-movement.
4-5.11.2010, Aarhus, Methods in Language&Cognition 56
Denmark

57. 9. Semantics in Vision:
Multimodality effects
4-5.11.2010, Aarhus, Methods in Language&Cognition 57
Denmark

58. • Vision is influenced by other expressions
of cognition
– Vestibular system
– Motor system
– Auditory system
Also purely visual stimulus may induce some
e.g. quasi-auditory perception in simulating
collision
4-5.11.2010, Aarhus, Methods in Language&Cognition 58
Denmark

59. 4-5.11.2010, Aarhus, Methods in Language&Cognition 59
Denmark

60. 4-5.11.2010, Aarhus, Methods in Language&Cognition 60
Denmark

61. cognition
Visual Vestibular Motor Auditory
perception system system system
4- Methods in Language&Cognition 61
5.11.2010, Aarhus, Den

62. cognition
Visual Vestibular Motor Auditory
perception system system system
4-5.11.2010, Aarhus, Methods in Language&Cognition 62
Denmark

63. 10. Remarks on spatial
cognition
4- Methods in Language&Cognition 63
5.11.2010, Aarhus, Den

64. • language provides strong constraints – selectivity and
enrichment – modulating the expression of spatial
cognition
1. Language with the basic components – symbolic units –
noun and verbs and configurations of these give rise to
semantic and syntactic functions of agent, patient,
subject, object.
2. In contrast spatial system contain primitives such as
shapes, objects, locations, landmarks, geometric
layouts, angles and directions – represented in different
spatial reference systems
• (Landau, Dessalegn, & Goldberg, 2010)
4-5.11.2010, Aarhus, Methods in Language&Cognition 64
Denmark

65. 11. Minimal semantics
generated in visual perception
4-5.11.2010, Aarhus, Methods in Language&Cognition 65
Denmark

66. • parallelism between visual and language
processing
• First, we have to build a complementary
methodology, using both psychophysical
methods and semantic methods.
• Second – what to start with?
• Primitive perceptual semantics. Very
rudimentary stimuli can generate a lot of
semantics
4-5.11.2010, Aarhus, Methods in Language&Cognition 66
Denmark

67. Object and predicate / thing and
action structure
4-5.11.2010, Aarhus, Methods in Language&Cognition 67
Denmark

68. action-resonance effects
- we see not separate
objects but action-
resonated objects.
- we do recognize events
(instead of separate
objects) as soon as we
assign meaning (even is
case of relatively minimal
and poor stimuli).
4-5.11.2010, Aarhus, Methods in Language&Cognition 68
Denmark

69. 11. Methods from semantics
Different biased and non-biased tasks (e.g.,
sentence completion tasks, priming
experiments, self-report measures). (Cp.
Hasson, U. & Giora, R. 2007).
4-5.11.2010, Aarhus, Methods in Language&Cognition 69
Denmark

70. • Lexical decision tasks are used to determine
whether a letter string on the screen makes a
word.
• prime and target.
4-5.11.2010, Aarhus, Methods in Language&Cognition 70
Denmark

71. faster
Related prime target
Unrelated control target
prime
4-5.11.2010, Aarhus, Methods in Language&Cognition 71
Denmark

72. • Memory measures
• learning stage and a test stage
• free recall method.
• old / new recognition task
• Self report measures: Listing features.
The task is to write down properties that
capture the meaning of an expression (or
visual stimulus).
4-5.11.2010, Aarhus, Methods in Language&Cognition 72
Denmark

73. • Effects of comprehension on subsequent tasks.
Priming paradigms. How comprehension of one stimulus
affects comprehension of another stimuli.
• word / sentence fragment completion task
• Negative priming (Christie & Klein, 2008, May, Kane,
& Hasher, 1995, MacLeod, Chiappe, & Fox, 2002)
• Different modifications: some distractor determines the
target and decreases activation time.
• Unconscious / masked priming (Dehaene, S.,
Naccache, L., Le Clec'H, G.., Koechlin, E., Mueller, M.,
Dehaene-Lambertz, G., van de Moortele, P. F., &
LeBihan, D., 1998)
4-5.11.2010, Aarhus, Methods in Language&Cognition 73
Denmark

74. Perception
4-5.11.2010, Aarhus, Methods in Language&Cognition 74
Denmark

75. • Central assumption: stimulus and sensory
are not directly corresponding and
proportional
4-5.11.2010, Aarhus, Methods in Language&Cognition 75
Denmark

76. Classically: Threshold
determination
• Method of average error
• Method of minimal changes
• Constant stimuli method
4-5.11.2010, Aarhus, Methods in Language&Cognition 76
Denmark

77. Classically: Magnitude
estimation
• Magnitude estimation with standard:
• A stimulus is presented (standard) and a number is
assigned to it (modulus, e.g., 30).
• The following stimuli are evaluated by subjects in that
they report the intensity relative to the standard in
assigning the stimuli numerical evaluation. The ratio
between numerical estimated and sensations.
• Magnitude estimation without standard:
• Subjects can choose their own standard (any number)
and assign it to the first stimulus. All following are
assigned by subject; preserving the ratio between
sensations and numbers
4-5.11.2010, Aarhus, Methods in Language&Cognition 77
Denmark

78. • Cross-modality matching:
• Indirect methods to differentiate
between stimuli
• Direct scaling methods
• Methods for stimulus comparison
• Some remarks regarding the use of
semantic differential in
4-5.11.2010, Aarhus, Methods in Language&Cognition 78
Denmark

79. • research on attitudes:
• Semantic differential is a fast way how to
measure attitudes. (The crucial point is that no
more than attitudes can be measured in using
semantic differential methodology.)
• Visual categorization and the use of
quantitative methods as complementary
tools in research of semantics in visual
perception.
• basic level prominence effects (Rosch, 1978,
Tversky, Zacks, & Hard, 2008).
4- Methods in Language&Cognition 79
5.11.2010, Aarhus, Den

80. 12. Conclusion: Constraints
4-5.11.2010, Aarhus, Methods in Language&Cognition 80
Denmark

81. • Lexicalization itself / dinscontiuty generation
• Selectivity: hierarchic – both in object encoding and in
encoding objects in events
• Attention
• How much can be actually perceived at a moment
– Saccades and fixations are mediated and correlated with
transsacadic memory (some information retained from one eye
fixation is invoked in processing that occurs during the next
fixation).
• Invariants
• Experiential constraints
• Situational constraints
4- Methods in Language&Cognition 81
5.11.2010, Aarhus, Den