1. Visual Analytics as a Cognitive
Science
Brian Fisher
SFU School of Interactive Arts & Technology and
Program in Cognitive Science
UBC Media & Graphics Interdisciplinary Centre (MAGIC)
2. My Background
• UG Biology, Medical Biophysics tech at CWRU Med
• Scientific Programmer, Varian
• Ph.D Experimental Psychology, UCSC
• UWO & Rutgers Centres for Cognitive Science
• Human-Information Interaction AKA “Cyberpsychology”
• Institute for Robotics and Intelligent Systems Networks of Centres of
Excellence “a Cognitive Basis for the Design of Intelligent User
Interfaces to Complex Systems” (SFU-UWO)
• 1999- Associate Director, UBC Media And Graphics
Interdisciplinary Centre, Computer Science, Psychology,
Institute for Computing, Information and Cognitive Systems
• 2004- SFU School of Interactive Arts and Technology and
Program in Cognitive Science
3. My double life
• Ph.D in Experimental Psych
• Postdoc w Cognitive
Science society founder &
president
• Psychonomics Fellow
• VIS-related symposia at
Cogsci, & APS, papers on
cogsci of interaction
• Fuzzy-logic/Bayes models
• Postdoc funded by Inst. for
Robotics and AI
• VAST SC, VEC, VACCINE
• Led Dagstuhl “Interaction with
Information for Visual
Reasoning”, Cogsci-based
papers at VIS, CHI, BELIV.
Participants
4. “Big Data”: Volume, Velocity &
Variety
• In 2011, data expected to be
about 1.8 zettabytes (1021).
• In 2013, Internet traffic to reach
667 exabytes (1018)/yr.
• Comparison: US Library of
Congress
is ~10 petabytes (1015).
• “By 2018, the United States alone could face a shortage
of 140,000 to 190,000 people with deep analytical skills
as well as 1.5 million managers and analysts with the
know-how to use the analysis of big data to make
effective decisions.” (McKinsey Global Institute 2011)
5. Challenges for computational
approaches
• “3 Vs” challenge
• Relevance, validity, reliability of data uncertain
• Insufficient time to reach solution
• Model challenges
• Multiple models to chose from
• Assumptions may or may not hold
• “Wicked problems” challenge (Rittel)
• Lack criteria to evaluate solution
• Each problem is unique (no population)
• Problem is not understood until solution is found
The information needed to understand the problem depends upon
one’s idea for solving it. -- Rittel & Webber 1973
6. The ultimate challenge
In business, government & the professions
specific people are responsible for the
decisions that are made and how they are
executed. These people are derelict in their
duties if they only accept what the model tells
them. Either we re-engineer society to accept
a computer model as the ultimate authority or
we find a way that human decision makers
can exercise due diligence for computational
as well as informational aspects of the
problem.
8. Visualization history
• NSF meeting: “Visualization
in Scientific Computing”
• Nov. 1987 Computer
Graphics
• First IEEE Visualization (now
SciVis) conference in 1990
“The purpose of [scientific] computing is insight, not numbers.”
Richard Hamming
“Visualization is a method of computing.” Authors of report
9. Information visualization
• 1990 Conference
on diagrammatic
reasoning
• 1995 InfoVis
Conference
• “Information
Visualization: Wings
for the Mind”
Keynote by Stuart
Card
10. Stuart Card’s view
•Increase the memory & processing
resources available to users
•Reduce the search for information by using
visual representations to enhance the
detection of patterns
•Engage perceptual inference operations
•Use perceptual attention mechanisms for
monitoring
•Support manipulation of information
12. Visualization literacy
• Build a “language” of
collections of images
that support thinking
• Diagrammatic
reasoning science of
how we understand
complex diagrams
13. Thinking as “smart seeing and Projecting”
Actively looking at external representations and
projecting onto them makes us more powerful thinkers
than thinking in our heads alone.
David Kirsh example
14. Design Approach:
Bertin
6
Bertin Semiologie
Graphique (1967)
Cartographer, built
description of how data
should be represented
visually
Jock Mackinlay, Stanford
Ph.D. dissertation
Tableau software, used for
our work with Boeing etc.
20. Role of psychology in VIS
• Design based on theories
(but no effective eval of
those theories)
• Adaptation of methods
from cogsci (but original
methods not well
understood)
• Rarely, ongoing
collaboration with
cognitive scientists
21. On the Death of Visualization
(Lorensen 2004)
Can It Survive Without Customers?
• Visualization, alone, is not a solution.
• Visualization is a critical part of many applications.
• Visualization, the Community, lacks application
domain knowledge.
• Visualization has become a commodity.
• Visualization is not having an impact in applications.
23. Battelle PNNL R&D Agenda
Panel
• In US, Panel meeting in 2004
•Brown, GMU, Georgia Tech, OSU, Penn State, Purdue,
SFU , Stanford, UC, UI, UM, UNC, UU, WPI
•Boeing, Microsoft, PARC, Sandia Labs
•CIA, DHS, FBI, NIST, NSA, unspecified
• Gave rise to
•Industry Consortium
•DHS Centre of Excellence
• Ccicada (Rutgers DIMACS)
• VACCINE (Purdue et al)
• In Europe, EU Vismaster Coordination action
• DFG Scalable Visual Analytics Priority program
24. Visual analytics
“This science must be built on integrated perceptual
and cognitive theories that embrace the dynamic
interaction between cognition, perception, and action.
It must provide insight on fundamental cognitive
concepts such as attention and memory. It must build
basic knowledge about the psychological foundations
of concepts such as ‘meaning,’ ‘flow,’ ‘confidence,’
and ‘abstraction.’ “
“Illuminating the Path” (IEEE Press)
“The science of analytical reasoning
facilitated by interactive visual interfaces”
25. How are VA Information
systems different?
• Development based on understanding of
expert cognition in situ
• Informed by current cognitive & social science
• Engagement with community of experts
• Emergent cognitive science of expert reasoning
• Clear support for analytical processes--
reasoning, collaboration & interaction
• Graphical analog for analytic processes
• Support “Human-information discourse”
• Integrated across roles in the community
26.
27. Visual analytics
Pg. 4 in Thomas, J., Cook, K., Institute of Electrical and Electronics Engineers, Dept
of Homeland Security, & United States. (2008). Illuminating the Path: The Research
and Development Agenda for Visual Analytics. IEEE Press. Retrieved from http://
www.osti.gov/energycitations/product.biblio.jsp?osti_id=912515
33. How to bridge informatics &
psychology?
• VIS offers:
• Implementations
• Funding
• Awesome
Research
Questions
• Psych offers
• Methodology
• Theory
• Phenomena
• Cheap talent
Challenges: defining boundary
objects, culture clash,
publication venues, academic
jobs for Cogs grads…
34. My approach: start in the middle!
• Develop bridging
“Cyberpsychology”
theory & methods
• Hope is that…
• They are building
from each shore
• Somehow we will be
able to align things
http://www.magic.ubc.ca
http://www.icics.ubc.ca
http://interaction-science.iat.sfu.ca
35. Bridging ideas from D-Cog
• Visualization literacy is a form of “Smart seeing
and projecting” w external representation
• We propose 2 additional D-cog perspectives:
• Agent-machine coupling: coordination of thought and
action in dynamic artificial environments
•Cognitive architecture modeling and characterizing
“personal equation” of individual differences (e.g.
perceptual expertise)
• Socially-distributed cognition
•Grounded theory in Clark’s Joint Activity Theory
framework analysis of pair/group collaborative decision
making
36. Agent-machine coupling in air
traffic control
• Cognitive architecture from psychology
• Extend to expert human performance
• Cognitive expertise
• Visual expertise
• Visuomotor expertise
• Multimodality & modularity
• Test human capabilities in dynamic display
environments
37. Controller/display systems in air
traffic control
• NextGen ATC
“fishtank”
projection
• Change camera
position for better
view
• How will global
motion affect
tracking?
Liu, G.Austen, E. L., Booth, K.S. Fisher, B., Argue, R. Rempel, M.I., & Enns, J. (2005)
Multiple Object Tracking Is Based On Scene, Not Retinal, Coordinates. Journal of
Experimental Psychology: Human Perception and Performance. 31(2),Apr 2005,
235-247.
http://www.youtube.com/watch?v=tKJVB4id_TY
45. Conclusion: We track in allocentric
space
• Retinal speed of targets does not determine
performance
• Motion of targets relative to each other does
• But only if motion preserves good metric
characteristics of space
• Explanation is at the level of a human -
display cognitive system
46. • “Pair analytics” sessions
•Student visual analyst & trained
domain expert collaborate on
analytic task
•Student “drives”, expert
“navigates”
•Video session & capture screen
Social D-Cog in safety analysis
47. Joint Activity Theory (Clark)
• Language is an essentially collaborative
activity, like playing duet or paddling canoe
• We work to build common ground so as to
communicate effectively and efficiently
• Clark’s theory:
• Defines kinds of common ground
• Formalizes the notion of activity as a “joint action”
• Describes the processes by which common ground is
developed through joint action
54. Wade Internship
• Video recorded and screen captured over 10
Paired Analysis sessions using both Tableau
and IN-SPIRE
• Influenced design decisions on:
• 777
• P8-A
• 787
• 747-8
• Changes to pilot training manual
55. Wade Internship
• Presented work to:
• 787 Engineers
• Aviation Safety Community of Practice
• Aerodynamics, Performance, Stability and Control
flight data recorder analysis group
• Advanced Analytics group
• UW Aeronautics and Astronautics students
• Boeing Educational Network webcast (400+)
• 500+ people exposed to Visual Analytics,
Paired Analysis for Aviation Safety
56. Joint activity analysis
• Observe joint attention management by
project markers
• Multimodal markers — mouse gesture & verbal
(content & prosody)
• Vertical and horizontal markers
• We define event structure for analysis based
on markers & actions
• Social constraints seen in self-talk incidents,
provides effective verbal protocols for protocol
analysis methods
57. Metrics include:
• Fluidity of the process.
• Management of joint attention.
• Situational awareness.
• Cognitive economics.
58. Publications
• Kaastra, L. T. and Fisher, B. (2014) Tracking Joint Activities in Visual Analyses.
Proceedings of the Psychonomics Society meeting
• Kaastra, L. T. and Fisher, B. (2014) Pair Analytics as a Field Experiment Methodology.. In
Proceedings of the 2014 BELIV Workshop: Beyond Time and Errors-Novel Evaluation
Methods for Visualization
• Kaastra, L. T., Arias-Hernandez, R., & Fisher, B. (2012). Evaluating analytic performance.
In Proceedings of the 2012 BELIV Workshop: Beyond Time and Errors-Novel Evaluation
Methods for Visualization
• Arias-Hernandez, R., Green, T. M., & Fisher, B. (2012). From Cognitive Amplifiers to
Cognitive Prostheses: Understandings of the Material Basis of Cognition in Visual
Analytics. In Interdisciplinary Science Reviews, Vol. 37 No. 1.
• Arias-Hernandez, R., Kaastra, L. T., & Fisher, B. (2011). Joint action theory and pair
analytics: In- vivo studies of cognition and social interaction in collaborative visual analytics
In Proceedings of the 33rd Annual Conference of the Cognitive Science Society
• Arias-Hernandez, R., Kaastra, L. T., Green, T. M., & Fisher, B. (2011). Pair Analytics:
Capturing Reasoning Processes in Collaborative Visual Analytics. In (HICSS), 2011 44th
Hawaii International Conference On System Sciences.
60. How to do it
• Institutes/centres/programs that combine:
• Application experts
• VIS people
• Cognitive scientists
• Designers
• Publication venues for translational research
• Two-part conference approach
• Outreach to application venues
• Return to VIS community
61. Thanks to SCIENCElab!
• Dr Richard Arias-
Hernández
• Dr. Nathalie Prevost
• Dr. Linda Kaastra
• Dr. Payam Rahmdel
• Samar Al-Hajj
• Nadya Calderón
• Tera Marie Green
• Ethan Soutar-Rau
• Ali Khalili
• Barry Po
• Aaron Smith
• Andrew Wade
• Doug Mackenzie
62. Cogs/IAT 885 Visually Enabled
Reasoning
• Cognitive theory
• Psychology of human reasoning
• Modelling causality
• Alternative (modal/hybrid) logics
• Analytic practice
• Problems and datasets from SEMVAST, Boeing
• Teams w paper, IN-SPIRE, Tableau Jigsaw
• Outcomes
• Learn reflective analytic practice
• Prepare for internship as analyst
63. Textbooks
• How We Reason. Philip Johnson-Laird,
Oxford Press
• Causal Models: How People Think about the
World and Its Alternatives. S. Sloman, Oxford
Press R
• Human Reasoning and Cognitive Science.
Keith Stenning & Michiel van Lambalgen, MIT
Press
