Detection of and Response to Online Users' Emotion

Beverly Park Woolf
College of Information and Computer Sciences,
University of Massachusetts-Amherst
bev@cs.umass.edu
Detection of and Response to
Online Users’ Emotion
Department of Quantitative Health Sciences
University of Massachusetts Medical School
March 24, 2017

Bounce between two systems . . .
Mathematics Tutor: Exists and contains
most of the described features.
Patient Care Tutor: Proposed and can
be built with these features.

Introduction and Care for the Patient
Model the Patient
Model Emotion (Sensors & Computer Vision)
Model the Medical Domain
Provide Interventions
Assess Patient Learning
Agenda

Caring: A Definition
Caring implies actions based on another person’s
wants and desires (Noddings, 1986).
Entities that care consider the other person’s point
of view, her needs and what she expects of us
(Cooper, 2003).
Caring includes the ability to empathize with others
and take responsibility for their needs.

Caring Impacts Learning
Findings from neuroscience suggest that all learning is
affective in nature (Damasio, 1994).
Students learn most effectively in a climate in which
people care about them (Cooper, 2003).
A major weakness of traditional psychology is to separate
the intellect and affect (Vygotsky, 1986). Every person’s
idea contains an affective attitude.

When Caring is Present . .
Positive emotions and interactions create the ambience
for learning and enable a student’s brain to remain
curious and open (Cooper, 2002; 2003).
A continuing, positive sense of self produces a constant
positive feeling throughout the body, which leads to
greater openness and willingness to engage in
interactions (Winkley, 1996). Babies brains grow when
they feel cared for.

When Caring is NOT Present . . .
Students flounder in internal confusion (Cooper, 2003).
Negative affect tends to produce a shutting down of one’s
self, a withdrawal, stimulating protection and defense.

Research Questions about Emotion
How are users’ emotions evidenced and measured?
How do emotions predict learning?
How accurate are emotion models (e.g., Markov or
Bayesian Models) at predicting future emotions from
student behaviors?
How effective are interventions at responding to
emotion??

Introduction and Care for the Patient
Model the Patient
Model Emotion (Sensors & Computer Vision)
Model the Medical Domain
Provide Interventions
Propose Health Care Tutor
Agenda

Model the Patient
Model the Disease
Personalize Tutoring
Assess Learning
Intelligent
Tutoring
Systems

Adele explains the importance of palpating the patient’s abdomen.
Background Research: Shaw, Johnson , & Ganeshan (1999). In
Proceedings of the third annual conference on Autonomous Agents (pp. 283-290). ACM.

Adele advising in a critical care scenario on the World Wide Web.
Pedagogical Agents on the Web
Erin Shaw, W. Lewis Johnson, and Rajaram Ganeshan

Adele instructs a student to answer a quiz when the student selects a urine
dipstick test. Pedagogical Agents on the Web
Erin Shaw, W. Lewis Johnson, and Rajaram Ganeshan

Background Research: Taking the Time to Care:
Empowering Low Health Literacy Hospital Patients
with Virtual Nurse Agents
Timothy W. Bickmore, Laura M. Pfeifer
College of Computer & Information Science Northeastern University
{bickmore,laurap}@ccs.neu.edu
• Ninety million Americans have inadequate health literacy,
resulting in a reduced ability to read and follow directions in
the healthcare environment.
• Animated, empathic virtual nurse interface for educating and
counseling hospital patients with inadequate health literacy in
their hospital beds at the time of discharge.
• Results indicate that hospital patients with low health literacy
found the system easy to use, reported high levels of
satisfaction, and most said they preferred receiving the
discharge information from the agent over their doctor or

Patient Interacting with Virtual Nurse
Taking the Time to Care
Chi 2009

Sample dialogue - relational aspects highlighted Taking the Time to Care
Chi 2009

Chi 2009

Self-report Ratings of the Virtual Nurse (mean (SD))
Chi 2009

MathSpring
An intelligent tutor for mathematics;
Contains hundreds of math problem, grades 5-10;
Used by tens of thousands of students;
Aligned with Common Core standards;
Detects and responds to student emotion;
Improves student passing of state standardized tests;
Uses multimedia characters to support student emotion;
Built by Woolf & Arroyo at UMass-Amherst.

Pedagogical Agents
Adult Users Require Adult Companions
Help to develop a positive emotional
environment; Agents offers advice and
encouragement.
Empathize with patient
Agents express full sentences of
cognitive, meta-cognitive and emotional
feedback.
Build patients’ self-esteem and self-worth
Agents are gendered and multicultural:
White, Hispanic, African-American

Mass Statewide Standardized Tests
Passing grades for experimental (dark grey) and control group
(light grey); same grade, same school and same teacher.

Students represented by the yellow/green polygon used the Math Tutor;
those represented by the blue polygon did not. The distribution of students
using the tutor is shifted towards the right, towards more proficiency and
above proficiency. Students were matched in terms of teacher; all were 7th
grade students.

Sensors to Detect
Student Emotion
David Cooper, Ph.D. Dissertation

Sensors
(Clockwise): mental state camera, skin conductance bracelet, pressure
sensitive mouse, pressure sensitive chair.

32
The Students
Rural-Area High School in MA (35 students)
Geometry and Algebra classes
UMASS 114 (29 students)
Math for Elementary School Teachers

33
Students Self-Report Emotions
Four bipolar emotional axes
Ekman’s
Categorization
Cognitive-Affective
Term
Emotion Scale
High excitement
Joy
Low excitement
“I am very excited.”
. . .
“This is not fun.”
Frustration
Anger
Low-frustration
“I am very frustrated.”
. .
“I am not frustrated at all.”
High interest
Interest
Low interest
“I am very interested.”
. . .
“I am bored.”
Anxiety
Fear
Confidence
“I feel anxious.”
.. . .
“I feel confident.”
Table 1. Cognitive-affective terms based on human face studies (Ekman et al., 1972; Ekman 1999)

How are you feeling? Please rate
your level of interest in this

Using Sensors to Measure Emotion:
A Linear Regression Model
Tutor
context
only
Camera +
Tutor
Seat +
Tutor
Wrist +
Tutor
Mouse +
Tutor
Confident R=0.49,
N=62
R=0.72,
N=20 -- a
R=0.35,
N=32
R=0.55,
N=28
Frustrated R=0.53,
N=69
R=0.63,
N=25
R=0.68,
N=25 -- d
R=0.56,
N=45
R=0.54,
N=44
Excite d R=0.43,
N=66
R=0.83,
N=21 -- b
R=0.65,
N=39
R=0.42,
N=37
R=0.57,
N=37
Interested R=0.37,
N=94
R=0.54,
N=36 -- c
R=0.28,
N=51
R=0.33,
N=51
a --( Solved? ConcentratingMax); b -- (concentratingMax; unsureMin; HintsSeen; LearningCompanion?);
c -- (InterestedMin LearningCompanion?); d -- (IncAttempts; SecsToFirstAtt; TimeInSession; sitForwardStdev
R represents the fit of the model
N is the number of cases available, for each emotion self-report and each sensor
Significant
results
Student
self-report
Cooper et al., UMAP 2009;
Arroyo et al., AIED 2009
David Cooper, Ph.D. Dissertation

36
Models of Emotions with Sensors
From Tutor-Context Variables and Sensors
Linear Models to Predict Emotions
Variables Entered in Stepwise Regression
Confidence InterestFrustration Excitement
SitForward
Stdev
“Concentrating”
Max. Probability
Camera Facial
Detection
Software
SitForward
Mean
Seat Sensor
# Hints
Seen
Solved?
1st Attempt
# Incorrect
attempts
Gender
Ped. Agent
Seconds to
1st Attempt
Time in
Tutor
Seconds
To Solve
Tutor Context Variables (for the last problem)Tutor Only All Sensors+Tutor
R=0.53 R=0.43 R=0.37R=0.49
R=0.72 R=0.70 R=NAR=0.82
Sensor Variables (Mean, Min, Max, Stdev for the last problem)

Results of Emotion Prediction Studies
Sensors predict self-report, accuracy 75-80%.
Sensors improve student emotion self-reports.
Self-reports predicts post-tutor math
performance, attitudes and perceptions of
tutoring software.

Computer Vision to Monitor &
Measure Grit
• Several written Instruments are used to measure
non-cognitive skills, primarily self-report surveys.
– Self-regulation
– Motivation strategies (MSLQ)
• Students with grit & persistence have good self-
regulation skills:
– Goal setting, self-monitoring, and self-instruction;
– They manage their emotions and waning motivation

Head Position
The right mouth corner is tracked during student-
computer interactions with significant changes of
head position and orientation

Low Level Features
Extraction of low-level features from face, (a) furrows, (b)
iris, and (c) mouth, for FACS-based facial expression
analysis, yielding the interpretation “student exhibits
anger.”

Trace Student Attention
The Mathspring GUI overlaid with a “heat
map” of student attention, as measured
by computer-vision. In red, screen
regions the student particularly focused
on

Engaged while reading and solving the
problem

Starting to get annoyed with the
learning companion
• Student was actually fine with the
learning companion at first but seems to
get annoyed later on
• Seem to be offended when the learning
companion states “See, I told you that
hints are really helpful”. He never took
any hints and did everything correctly on
every first try
• Finally closes the learning companion as
soon as it pops up after doing six
Looking to the right side of the screen
means he is looking at learning
companion

Closing learning companion
Companion
removed by
student.

Summary for GritCamera6
• Student is very engaged
• Student takes time to answer every question
– Made no mistake
– Took no hint
– Got it right in every first try
• Takes time to answer survey
– Only two survey shows up

Cardiac Tutor, Built at UMass-Amherst
The Cardiac Tutor teaches advanced
cardiac support techniques to medical
personnel. It presents cardiac problems
and, using a variety of steps, students
select various interventions. It provides
clues, verbal advice, and feedback to
personalize and optimize the learning..
Simulated patient in the Cardiac Tutor.
The intravenous line has been installed (
“IV in ”); chest compressions are in
progress, ventilation has not yet begun,
and the electronic shock system is
discharged. Eliot, Williams & Woolf, 1996, An intelligent learning
environment for advanced cardiac life support.

Rashi Case Description
Built at UMass-Amherst
Dragon et al., 2006 Coaching Within a Domain
Independent Inquiry Environment

Rashi Interview Tool

Rashi Exam Tool

Rashi Inquiry Notebook

Rashi Argument Editor

Background Research:
Medical Knowledge
Bioworld patient scenario and evidence palette. Susanne Lajoie 2001
Constructing knowledge in the context of BioWorld

Model Medical Knowledge
(Left) Conducting Diagnostic Tests; (Right)
constructing the closing argument Lajoie 2001; Constructing knowledge in the context of BioWorld
Lajoie & Greer, 2003 Establishing an Argumentation Environment to Foster
Scientific Reasoning With Bio-World

Research Questions
• What to do in the moment when users are
frustrated, bored, etc.?
– Increase Challenge? Decrease Challenge?
– Provide extra scaffolds?
– Provide “affective” scaffolds? What are those?
– Encourage users to stop and think about what is going
on?
• How to measure changes in student affect, or to
capture micro-changes in student affective
states?

Interventions have a strong
impact on Learning
• Average effect size of metacognitive
instruction across 20 studies was .72, a very
large effect.
– Teach a skill; track the student, measure the skill
before and after use of the software.
– Measure a student’s aptitude for learners trained
in a skill against students who are not trained in
that skill.

Many people:
• relate to computers in the same way they relate to
humans (Nass, 2010);
• continue to engage in frustrating tasks significantly
longer after an empathic digital response (Picard);
• have lowered stress levels after receiving an empathetic
message from a digital character (Arroyo et al., 2009);
• recalled more information when interacting with an
artist agent compared to scientist agent;
• report reduced frustration and more general interest
when working with gender-matched characters.
People and Agents

Affirmation Theory Messages
Affirmation Theory: Propose that users’ motivation is
rooted in their belief about why they succeed or fail. Students
can be taught to understand that failure is the result of a lack
of effort instead of a lack of ability.
Example Messages “People have myths about math, like,
that only some people are good in
math. The truth is that we can all be
successful in math if we give it a try”
“We will learn new skills only if we
are persistent. If we are very stuck,
let's call the teacher, or ask for a
hint!”
“When we realize we don't know why
the answer was wrong, it helps us
understand better what we need to
practice.”

Effort Interaction Messages
Effort Interactions: Acknowledge effort and incorrect
answers. The goal is to make students realize that praise is not
always appropriate and that effort is the primary goal.
Example Messages
“That was too easy for you. Let's
hope the next one is more
challenging so that we can learn
something.”!”
“Good job! See how taking your time
to work through these questions can
make you get the right answer?”

Incorrect ResponseStudent effort shown/
correct response
Student effort shown /
incorrect response
Agent Emotion
Frustrated students are supported by helpful companions.
 Arroyo et al., AIED2009

• Our current Gold standard/Truth: self-reports
• During Experiment:
– Ask students about their “interest” level and their
“excitement” level every 7 minutes, on average.
Gathering Student Affect “During”
Low
High
Neutral / Middle

Large Data Sets
EventLog Table of a Math Tutoring System. 571,776 rows, just in a year time.

The Student Progress Page
73
Dovan Rai, PhD, WPI

Three Experimental Conditions
74
• No access to the Student Progress Page
• “My Progress” button was present
(student choice)
• Prompt invitation to see “my progress” upon
bored (disinterested) or unexcited
• Force student to see my progress when student
bored (disinterested) or unexcited

How to analyze changes in student
affect, from moment to moment?
MARKOV CHAIN MODELS

Student Interest Ten thousand Data Points N~230.
 SPP
Absent
 SPP
Present
SPP 
Prompted
SPP 
Forced

How to compare Markov Chain
Models, quantitatively?
• Probability of student following a specific path
• What is the probability that a student will end
up excited, after 3 transitions?
• Given that they started in a specific state?

Health Care Research Questions
Which interventions improve patients’ affect and
learning?
Which interventions are best for which patients?
What is the impact of deeper companion
characteristics?
Do affective characters make users “feel” better?
How does the gender (ethnicity) of companions
impact patients’ attitudes/emotions/learning?

 Detect patient emotion, using Computer Vision or sensors
(camera, wrist band, pressure mouse, seat sensors).
 Dynamically collect data streams of students’ physiological
activity and self-reports of emotions.
 Apply interventions once emotion is detected,
e.g., animated embodied agents
 Integrate computer vision techniques to improve
detection of emotion.
 Train researchers and educators in techniques to recognize an
respond to patients’ emotion and make predictions over larg
data sets.
Proposed Health Care System

Study Design
Patients
Control
Experimental
Intervention
Control
Experimental
Intervention
Visit # 1 Visit # 2 Visit # 3 Visit # 4
Randomized
• Survey
• Audio tape with
physician
• Survey
• Audio tape with physician
• Survey
• Audio tape with physician • Survey
• Survey • Survey
• Survey
Baseline data
Greenfield et al., 1986; Patients’ participation in Medical Care General
Internal Medicine

Emotion is vital to learning and using technology to
recognize users’ emotion has led to powerful performance
results.
Detecting and responding to users’ emotion while they
work with online environments improves learning.
Summary

Thank You !
Any Questions?
Detection of and Response to
Online Users’ Emotion
Department of Quantitative Health Sciences
University of Massachusetts Medical School
March 24, 2017

Detection of and Response to Online Users' Emotion

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (19)

Ähnlich wie Detection of and Response to Online Users' Emotion

Ähnlich wie Detection of and Response to Online Users' Emotion (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Detection of and Response to Online Users' Emotion