PapePresentation

1. Jacek Gwizdka & Michael J. Cole
Dept. of Library and Information Science, Rutgers University
New Brunswick, NJ, USA
Workshop on Inferring Cognitive and Emotional States from Multimodal Measures – MMCogEmS’2011
November 17, 2011

2. !! Overall research goal: infer and predict mental
states and context of a person engaged in
interactive information search (e.g., Web search)
!! Completed projects: measures derived from eye-
gaze patterns
!! eye-movement patterns and interaction logs to infer
"! task characteristics
"! dynamic user states (such as cognitive load/effort)
"! persistent user characteristics (such as domain knowledge)
!! On-going projects: multi-modal measures
!! eye-tracking + EEG + GSR
!! cognitive load + timing of relevance decisions

3. !! Methodology: Using eye-gaze patterns
!! Higher-order patterns: Reading Models
!! Measures of cognitive effort in reading
!! Results:
!! User study I: journalistic search tasks
"! task characteristics
"! cognitive effort
!! User study II: genomics search tasks
"! cognitive effort (& learning)
"! domain knowledge
!! On-going work
3

5. !! Eye-tracking research have
frequently analyzed eye-
gaze position aggregates
('hot spots’)
!! spatiotemporal-intensity –
heat maps
!! also sequential – scan paths
! Higher-order patterns:
reading models & derived measures
5

6. !! We have developed a new methodology to analyze
eye-gaze patterns:
!! Model the reading process to represent (textual) information
acquisition in search
!! Measure the cognitive effort due to (textual) information
acquisition
!! Use both to correlate / infer higher-level constructs (task
characteristics, user knowledge, etc.)

7. Can be represented as units of reading experience:
((F F F) F (F F F) F F F F (F F F F F F) F)
F = fixation 7

8. 1.! Eye movements are cognitively controlled (Findlay &
Gilchrist, 2003)
2.! Eyes fixate until cognitive processing is completed
(Rayner, 1998)
Eye gaze pattern analysis is powerful:
!! Eye gaze is only way to acquire (textual)
information
!! 1. + 2. ! Direct causal connection between
observable (text) information search behavior and
user’s mental state

9. !! We implemented the E-Z Reader reading model
(Reichle et al., 2006)
!! Fixation duration >113 ms – threshold for lexical processing
(Reingold & Rayner, 2006)
!! The algorithm distinguishes reading fixation sequences from
isolated fixations, called 'scanning' fixations
!! Each lexical fixation is classified to (S,R) that is (Scan,
Reading)
!! Inputs: eye gaze location, duration
!! Add fixation to reading sequence if next saccade:
!! on the same line of text
!! and less than 120 pixels to the right
!! or is a regression on the same line of text
9

11. !! Two states: reading and scanning
!! transition probabilities
!! each state characterized by the number of lexical fixations
and duration
q
p
Read Scan
1-p 1-q
11

12. Can be represented as units (Fixations) of reading experience:
(F F F) F (F F F) F F F F (F F F F F F) F
Using the reading model :
Reading state – R (green); Scanning state – S:
RSRSSSSRS
12

13. 13

14. !! Eyes fixate until cognitive processing is completed
(Rayner 1998)
!! While reading, words already understood in the
parafoveal region are skipped (Reichle, et al., 2006)
!! Eye gaze patterns depend on cognitive processing
of information that is being acquired
!! Hypothesis: Analysis of reading fixation patterns
reveal some aspects of cognitive effort
14

15. text acquired
!! Reading Speed = -----------------
processing time
!! Perceptual Span - Average spacing of fixations
!! Lexical Fixation Duration Excess (LFDE):
!! Time needed to acquire meaning above the minimum for
lexical access
!! Fixation Regressions - Number of regression
fixations in the reading sequence
15

16. 1o (70px) foveal region
!! Reading speed will be slower for:
!! hard to read text (Rayner & Pollatsek, 1989),
!! unfamiliar words (Williams & Morris, 2004),
!! words used in less frequent senses (Sereno, O’Donnell, &
Rayner, 2006),
!! more complex concepts (Morris, 1994)
16

17. Perceptual span is the spacing of fixations
Perceptual span reflects a human limitation on the number
and difficulty of concepts that can be processed (e.g. Pollatsek
et al. 1986).
17

18. !! 10-15% of fixations are regressions
!! Reading goal affects reading regressions
!! More regressions when:
!! greater reader domain expertise,
!! conceptually complex & difficult text passages,
!! resolution of ambiguous (sense) words
18

19. !! Greater LFDE indicates less familiar words &
greater conceptual complexity
!! LFDE is also correlated with establishing word
meaning in context
example
19

20. 20

21. !! 32 journalism students
!! 4 journalistic tasks (realistic, created by journalism faculty
and journalists)
!! Journalism tasks can be about any topic, but few
task types.
!! Tasks designed to vary in ways that affect search
behavior (Li, 2009)
!! Task difficulty was post-self-rated by participants (7-
point Likert scale: ’very easy’ to ’extremely difficult’)
21

22. !! Complexity - number of steps needed (ex: identify an expert, get
contact information)
!! Task Product (factual vs. intellectual, e.g., fact checking vs.
production of a document)
!! Named - is actual search target specified?
!! Level - the information object to process (a complete document vs.
a document segment)
!! Task Goal - the nature of the task goal (specific vs. amorphous)
Task Product Level Named Goal Complexity
Background BIC mixed Document No Specific High
Copy Editing CPE factual Segment Yes Specific Low
Interview Preparation INT mixed Document No Mixed A,S Low
Advance Obituary OBI factual Document No Amorphous High
!! Note: Copy Editing CPE & Advance Obituary OBI are most dissimilar
!! Copy Editing is expected to be easiest, Advance Obituary most difficult

23. !! User task characteristics
!! Can we detect task characteristics from eye-gaze patterns ?
!! Cognitive effort
!! Do the cognitive effort measures correlate with:
"! task properties expected to contribute to task difficulty?
"! the effort needed to complete the task?
"! user judgment of task difficulty?
23

24. 24

25. !! Task effects on transition probabilities S!R & R!S
(all subjects & pages) OBI: advanced obituary
INT: interview preparation
CPE: copy editing
BIC: background information
•! For OBI, INT searchers
biased to continue
reading
•! For CPE to continue
scanning
Searchers are adopting
different reading
strategies for different
task types
(Cole, Gwizdka, Liu, Bierig, Belkin & Zhang, ECCE 2010; IwC 2011) 25

26. OBI: advanced obituary
INT: interview preparation
!! For highly attended pages CPE: copy editing
BIC: background information
Total Text Acquired on Total Text Acquired on
SERPs and Content SERPs and Content
per page 26

27. OBI: advanced obituary
INT: interview preparation
!! For highly attended pages CPE: copy editing
BIC: background information
Read ! Scan! Read
Read ! Scan! Read Scan
Scan
State Transitions State Transitions on
on SERPs per page Content pages per page
27

28. Task Product Level Named Goal Complexity
Background BIC mixed Document No Specific High
Copy Editing CPE factual Segment Yes Specific Low
Interview Preparation INT mixed Document No Mixed A,S Low
Advance Obituary OBI factual Document No Amorphous High
Measure Related Task Characteristics
level: document; goal other
bias to read Task level than specific (OBI & INT)
Number of For all
and task
state transitions level: segment and task pages
bias to scan goal
goal: specific (CPE)
Task complexity: More text acquired in
Total text acquired on SERPs
BIC and OBI
Text acquired and number of
Task level: segment and task product:
state transitions per page on
factual (CPE) For
content pages
highly
attended
pages
Cole, Gwizdka, Liu, Bierig, Belkin & Zhang. (2011). Task and User Effects on Reading Patterns in Information28
Search Interacting with Computers 23(4), 346 – 362.

30. (self-rated after the task)
BIC CPE INT OBI

31. !! Search
effort: task time, pages visited, queries
entered
!! Copy Editing (CPE) required the least effort of all tasks
!! Advance Obituary (OBI) required overall most effort (although not
the greatest effort of the tasks for every effort measure)
!! Forall tasks, for both greater perceived difficulty (self-
ratings) and search task effort:
!! higher median LFDE (Kruskal-Wallis chi-squared =125.02, p = 0.03)
!! slower reading speed (ANOVA F-value=5.5 p=0.02)
!! Strongest correlations obtained when considering only the
single longest reading sequence on a page

32. 32

33. !! …
33

34. !! …
34

35. !! …
35

36. !! Cognitive effort measures seem valid
!! Eye gaze pattern cognitive effort measures match
with subjective task difficulty
!! Cognitive effort measure results correlate with task
characteristics related to task effort
!! e.g. Complex tasks, amorphous goals
36

37. 37
37

38. !! Words are indicative of concepts and concept
features
!! Reading involves:
!! knowledge used to understand words,
!! processing concepts expressed in the content, and
!! acquisition of information (and concepts) from the content
!! User knowledge controls interaction during search:
!! selects the words to read, and
!! imposes cognitive processing demands to understand the
concepts associated with the words

39. !! Does user’s knowledge influence information search
behavior?
!! Is cognitive effort related to domain knowledge?
39

40. !! 40 undergraduate and graduate students
!! Rated 409 genetics and genomics MeSH terms
!! 1: No knowledge, ... to 5: Can explain to others
!! Five tasks from 2004 TREC Genomics track
!! Tasks were hard!
!! We use the same methodology to create reading
models and calculate cognitive effort measures as in
study I

41. !! Participants’
domain knowledge (PDK) was
represented by sum of term ratings
!! participants rated MeSH terms
!! normalized by a hypothetical expert
•! ki is the term knowledge rating (1-5)
•! i ranges over all terms
•! ti is 1 if rated or 0 if not
•! m number of terms rated by a participant
•! The sum is normalized by a hypothetical expert
who rated all terms as 'can explain to others'

43. These cognitive effort measures were individually
correlated with level of domain knowledge.
For all reading sequences:
!! higher domain knowledge ~ lower cognitive effort
!! perceptual span (Kruskal-Wallis "2 = 4734.254, p < 2.2e-16)
!! LFDE (Kruskal-Wallis "2 = 5570.103, p < 2.2e-16)
!! reading speed (Kruskal-Wallis "2 = 5570.103, p < 2.2e-16)
Similar correlations found for long reading sequences
Long reading sequences might better reflect concept use by participants
during information acquisition because of the attention allocated to acquiring
that text.

44. !! For long reading sequences
!! We used random forests to construct regression
models from the cognitive effort measures
!! Regression results were clustered
(agglomerate hierarchical clustering)
!! Random forest model gave us relative importance of
cognitive effort measures as contributing variables
in a predictive model
!! high importance: reading length (px), LDFE, total duration
of reading sequences (sum of lexical fix), perceptual span
!! less important: number of regressions

45. Random forest model classification errors
all participants PDKgroups! low! inter! high!
low! 8! 0! 0!
intermediate! 1! 23! 0!
high! 0! 0! 6!
only native English speakers PDKgroups! low! inter! high!
low! 3! 0! 0!
intermediate! 0! "#! 0!
high! 0! 0! $!
Random forest model cog effort !
domain knowledge correlation
with MeSH based domain knowledge

46. !! Ability to detect knowledge levels indicates a
possibility of real-time detection of learning of a
new material (new domain)
!! Task “phase” analysis: beginning, middle, end
!! same random forest model across the three phases
!! significant difference : LFDE drops from beg to mid to end
phase, while
!! numFix -- not significantly different between phases
!! and readingLength increased from middle to end (sig: Kruskal-
Wallis chi^2 = 885.2262, df = 817, p < 0.05)
!! Possible evidence for learning ?

47. !! Eye tracking enables high resolution analysis of
searcher’s activity during interactions with
information systems
!! There is more beyond eye-gaze locations with
timestamps
!! Eye-tracking data:
!! can be used to identification of task characteristics
!! … cognitive effort
!! … domain knowledge
!! High potential for implicit detection of a searcher’s
states
47

48. !! The reading model methodology and cognitive effort
measures are based on many years of empirical
research.
!! Eye movements have a direct causal connection to the
information acquisition process.
!! This connection is not mediated!
!! Domain independent
!! Document content is not involved
!! Culturally and individually independent
!! Method represents the user's experience of the information
acquisition process
!! Real-time modeling of user domain knowledge is possible

49. !! Processing requirements are low - just need fixation
location and duration.
!! Only recent eye movements are needed to calculate
cognitive effort.
!! Real-time assessment of cognitive effort
!! Early task session detection of user properties, e.g. domain
knowledge and perception of task difficulty
!! Soon enough for a system to make a difference in
providing user support

50. 50

51. •! Implicit characterization of Information Search Process using
physiological devices
•! Can we detect when searchers make information relevance
decisions?
!! Start with eye- Eye
Emotiv EPOC
wireless EEG
tracking: tracking headset
pupillometry EEG
!! info relevance (Oliveria,
Russell, Aula, 2009)
!! low-level decision timing
(Einhäuser, et al. 2010)
Tobii T-60
!! Adds EEG, GSR eye-tracker
!! Funded by Google
Research Award
GSR
51

52. !! Jacek Gwizdka http://jsg.tel
!! Acknowledgements
!! Funding: IMLS Google
!! Collaborators:
"! Dr. Nicholas J. Belkin, Dr. Xiangmin Zhang
"! Post-Doc: Dr. Ralf Bierig
"! Collaborator & PhD student: Michael Cole
"! PhD students: Chang Liu, Jingjing Liu
"! Master students
52

53. !! Eye tracking technology is declining in price and in
2-3 years could be part of standard displays.
!! Already in luxury cars and semi-trucks (sleep detection)
!! Computers with built in eye-tracking
Tobii / Lenovo
proof of concept eye-tracking
laptop - March 2011
53