Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Detection of and response to Online Users' Emotion

131 Aufrufe

Veröffentlicht am

This presentation describes the importance of detecting and responding to users emotion while they work with online environments. Emotion is vital to learning and using technology to recognize users’ emotion has led to powerful performance results. First, we describe how to detect emotion, using sensors (camera, wrist band, pressure mouse, seat sensors). Computational tutors dynamically collected data streams of students’ physiological activity and self-reports of emotions. Summaries of student physiological activity helped predict more than 80% of the variance of students’ emotional states. Second, we describe responses or interventions that we used once emotion was detected, i.e., we evaluated the impact of animated embodied agents on user motivation and achievement. Results showed that women and students with disabilities, while using agents reported increased math value, self-concept and mastery orientation and reduced frustration. Third, we describe the integration of computer vision techniques to improve detection of emotion.

Veröffentlicht in: Bildung
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Detection of and response to Online Users' Emotion

  1. 1. Beverly Park Woolf College of Information and Computer Sciences, University of Massachusetts-Amherst bev@cs.umass.edu Detection of and Response to Online Users’ Emotion Department of Quantitative Health Sciences University of Massachusetts Medical School March 24, 2017
  2. 2. Bounce between two systems . . . Mathematics Tutor: Exists and contains most of the described features. Patient Care Tutor: Proposed and can be built with these features.
  3. 3. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Assess Patient Learning Agenda
  4. 4. Caring: A Definition Caring implies actions based on another person’s wants and desires (Noddings, 1986). Entities that care consider the other person’s point of view, her needs and what she expects of us (Cooper, 2003). Caring includes the ability to empathize with others and take responsibility for their needs.
  5. 5. Caring Impacts Learning Findings from neuroscience suggest that all learning is affective in nature (Damasio, 1994). Students learn most effectively in a climate in which people care about them (Cooper, 2003). A major weakness of traditional psychology is to separate the intellect and affect (Vygotsky, 1986). Every person’s idea contains an affective attitude.
  6. 6. When Caring is Present . . Positive emotions and interactions create the ambience for learning and enable a student’s brain to remain curious and open (Cooper, 2002; 2003). A continuing, positive sense of self produces a constant positive feeling throughout the body, which leads to greater openness and willingness to engage in interactions (Winkley, 1996). Babies brains grow when they feel cared for.
  7. 7. When Caring is NOT Present . . . Students flounder in internal confusion (Cooper, 2003). Negative affect tends to produce a shutting down of one’s self, a withdrawal, stimulating protection and defense.
  8. 8. Research Questions about Emotion How are users’ emotions evidenced and measured? How do emotions predict learning? How accurate are emotion models (e.g., Markov or Bayesian Models) at predicting future emotions from student behaviors? How effective are interventions at responding to emotion??
  9. 9. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
  10. 10. Model the Patient Model the Disease Personalize Tutoring Assess Learning Intelligent Tutoring Systems
  11. 11. Adele explains the importance of palpating the patient’s abdomen. Background Research: Shaw, Johnson , & Ganeshan (1999). In Proceedings of the third annual conference on Autonomous Agents (pp. 283-290). ACM.
  12. 12. Adele advising in a critical care scenario on the World Wide Web. Pedagogical Agents on the Web Erin Shaw, W. Lewis Johnson, and Rajaram Ganeshan
  13. 13. Adele instructs a student to answer a quiz when the student selects a urine dipstick test. Pedagogical Agents on the Web Erin Shaw, W. Lewis Johnson, and Rajaram Ganeshan
  14. 14. Background Research: Taking the Time to Care: Empowering Low Health Literacy Hospital Patients with Virtual Nurse Agents Timothy W. Bickmore, Laura M. Pfeifer College of Computer & Information Science Northeastern University {bickmore,laurap}@ccs.neu.edu • Ninety million Americans have inadequate health literacy, resulting in a reduced ability to read and follow directions in the healthcare environment. • Animated, empathic virtual nurse interface for educating and counseling hospital patients with inadequate health literacy in their hospital beds at the time of discharge. • Results indicate that hospital patients with low health literacy found the system easy to use, reported high levels of satisfaction, and most said they preferred receiving the discharge information from the agent over their doctor or
  15. 15. Patient Interacting with Virtual Nurse Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer Chi 2009
  16. 16. Sample dialogue - relational aspects highlighted Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer Chi 2009
  17. 17. Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer Chi 2009
  18. 18. Self-report Ratings of the Virtual Nurse (mean (SD)) Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer Chi 2009
  19. 19. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
  20. 20. MathSpring An intelligent tutor for mathematics; Contains hundreds of math problem, grades 5-10; Used by tens of thousands of students; Aligned with Common Core standards; Detects and responds to student emotion; Improves student passing of state standardized tests; Uses multimedia characters to support student emotion; Built by Woolf & Arroyo at UMass-Amherst.
  21. 21. 23
  22. 22. The MathSpring System
  23. 23. Pedagogical Agents Adult Users Require Adult Companions Help to develop a positive emotional environment; Agents offers advice and encouragement. Empathize with patient Agents express full sentences of cognitive, meta-cognitive and emotional feedback. Build patients’ self-esteem and self-worth Agents are gendered and multicultural: White, Hispanic, African-American
  24. 24. Mass Statewide Standardized Tests Passing grades for experimental (dark grey) and control group (light grey); same grade, same school and same teacher.
  25. 25. Students represented by the yellow/green polygon used the Math Tutor; those represented by the blue polygon did not. The distribution of students using the tutor is shifted towards the right, towards more proficiency and above proficiency. Students were matched in terms of teacher; all were 7th grade students.
  26. 26. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
  27. 27. Sensors to Detect Student Emotion David Cooper, Ph.D. Dissertation
  28. 28. Sensors (Clockwise): mental state camera, skin conductance bracelet, pressure sensitive mouse, pressure sensitive chair.
  29. 29. 31 Use Sensors
  30. 30. 32 The Students Rural-Area High School in MA (35 students) Geometry and Algebra classes UMASS 114 (29 students) Math for Elementary School Teachers
  31. 31. 33 Students Self-Report Emotions Four bipolar emotional axes Ekman’s Categorization Cognitive-Affective Term Emotion Scale High excitement Joy Low excitement “I am very excited.” . . . “This is not fun.” Frustration Anger Low-frustration “I am very frustrated.” . . “I am not frustrated at all.” High interest Interest Low interest “I am very interested.” . . . “I am bored.” Anxiety Fear Confidence “I feel anxious.” .. . . “I feel confident.” Table 1. Cognitive-affective terms based on human face studies (Ekman et al., 1972; Ekman 1999)
  32. 32. How are you feeling? Please rate your level of interest in this
  33. 33. Using Sensors to Measure Emotion: A Linear Regression Model Tutor context only Camera + Tutor Seat + Tutor Wrist + Tutor Mouse + Tutor Confident R=0.49, N=62 R=0.72, N=20 -- a R=0.35, N=32 R=0.55, N=28 Frustrated R=0.53, N=69 R=0.63, N=25 R=0.68, N=25 -- d R=0.56, N=45 R=0.54, N=44 Excite d R=0.43, N=66 R=0.83, N=21 -- b R=0.65, N=39 R=0.42, N=37 R=0.57, N=37 Interested R=0.37, N=94 R=0.54, N=36 -- c R=0.28, N=51 R=0.33, N=51 a --( Solved? ConcentratingMax); b -- (concentratingMax; unsureMin; HintsSeen; LearningCompanion?); c -- (InterestedMin LearningCompanion?); d -- (IncAttempts; SecsToFirstAtt; TimeInSession; sitForwardStdev R represents the fit of the model N is the number of cases available, for each emotion self-report and each sensor Significant results Student self-report Cooper et al., UMAP 2009; Arroyo et al., AIED 2009 David Cooper, Ph.D. Dissertation
  34. 34. 36 Models of Emotions with Sensors From Tutor-Context Variables and Sensors Linear Models to Predict Emotions Variables Entered in Stepwise Regression Confidence InterestFrustration Excitement SitForward Stdev “Concentrating” Max. Probability Camera Facial Detection Software SitForward Mean Seat Sensor # Hints Seen Solved? 1st Attempt # Incorrect attempts Gender Ped. Agent Seconds to 1st Attempt Time in Tutor Seconds To Solve Tutor Context Variables (for the last problem)Tutor Only All Sensors+Tutor R=0.53 R=0.43 R=0.37R=0.49 R=0.72 R=0.70 R=NAR=0.82 Sensor Variables (Mean, Min, Max, Stdev for the last problem)
  35. 35. Results of Emotion Prediction Studies Sensors predict self-report, accuracy 75-80%. Sensors improve student emotion self-reports. Self-reports predicts post-tutor math performance, attitudes and perceptions of tutoring software.
  36. 36. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
  37. 37. Computer Vision to Monitor & Measure Grit • Several written Instruments are used to measure non-cognitive skills, primarily self-report surveys. – Self-regulation – Motivation strategies (MSLQ) • Students with grit & persistence have good self- regulation skills: – Goal setting, self-monitoring, and self-instruction; – They manage their emotions and waning motivation
  38. 38. Head Position The right mouth corner is tracked during student- computer interactions with significant changes of head position and orientation
  39. 39. Low Level Features Extraction of low-level features from face, (a) furrows, (b) iris, and (c) mouth, for FACS-based facial expression analysis, yielding the interpretation “student exhibits anger.”
  40. 40. Trace Student Attention The Mathspring GUI overlaid with a “heat map” of student attention, as measured by computer-vision. In red, screen regions the student particularly focused on
  41. 41. Engaged while reading and solving the problem
  42. 42. Head tilts
  43. 43. Starting to get annoyed with the learning companion • Student was actually fine with the learning companion at first but seems to get annoyed later on • Seem to be offended when the learning companion states “See, I told you that hints are really helpful”. He never took any hints and did everything correctly on every first try • Finally closes the learning companion as soon as it pops up after doing six Looking to the right side of the screen means he is looking at learning companion
  44. 44. Closing learning companion Companion removed by student.
  45. 45. Summary for GritCamera6 • Student is very engaged • Student takes time to answer every question – Made no mistake – Took no hint – Got it right in every first try • Takes time to answer survey – Only two survey shows up
  46. 46. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
  47. 47. Cardiac Tutor, Built at UMass-Amherst The Cardiac Tutor teaches advanced cardiac support techniques to medical personnel. It presents cardiac problems and, using a variety of steps, students select various interventions. It provides clues, verbal advice, and feedback to personalize and optimize the learning.. Simulated patient in the Cardiac Tutor. The intravenous line has been installed ( “IV in ”); chest compressions are in progress, ventilation has not yet begun, and the electronic shock system is discharged. Eliot, Williams & Woolf, 1996, An intelligent learning environment for advanced cardiac life support.
  48. 48. Rashi Case Description Built at UMass-Amherst Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
  49. 49. Rashi Interview Tool Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
  50. 50. Rashi Exam Tool Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
  51. 51. Rashi Lab Test Tool
  52. 52. Rashi Inquiry Notebook Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
  53. 53. Rashi Argument Editor Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
  54. 54. Background Research: Medical Knowledge Bioworld patient scenario and evidence palette. Susanne Lajoie 2001 Constructing knowledge in the context of BioWorld
  55. 55. Model Medical Knowledge (Left) Conducting Diagnostic Tests; (Right) constructing the closing argument Lajoie 2001; Constructing knowledge in the context of BioWorld Lajoie & Greer, 2003 Establishing an Argumentation Environment to Foster Scientific Reasoning With Bio-World
  56. 56. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
  57. 57. Research Questions • What to do in the moment when users are frustrated, bored, etc.? – Increase Challenge? Decrease Challenge? – Provide extra scaffolds? – Provide “affective” scaffolds? What are those? – Encourage users to stop and think about what is going on? • How to measure changes in student affect, or to capture micro-changes in student affective states?
  58. 58. Interventions have a strong impact on Learning • Average effect size of metacognitive instruction across 20 studies was .72, a very large effect. – Teach a skill; track the student, measure the skill before and after use of the software. – Measure a student’s aptitude for learners trained in a skill against students who are not trained in that skill.
  59. 59. Many people: • relate to computers in the same way they relate to humans (Nass, 2010); • continue to engage in frustrating tasks significantly longer after an empathic digital response (Picard); • have lowered stress levels after receiving an empathetic message from a digital character (Arroyo et al., 2009); • recalled more information when interacting with an artist agent compared to scientist agent; • report reduced frustration and more general interest when working with gender-matched characters. People and Agents
  60. 60. Affirmation Theory Messages Affirmation Theory: Propose that users’ motivation is rooted in their belief about why they succeed or fail. Students can be taught to understand that failure is the result of a lack of effort instead of a lack of ability. Example Messages “People have myths about math, like, that only some people are good in math. The truth is that we can all be successful in math if we give it a try” “We will learn new skills only if we are persistent. If we are very stuck, let's call the teacher, or ask for a hint!” “When we realize we don't know why the answer was wrong, it helps us understand better what we need to practice.”
  61. 61. Effort Interaction Messages Effort Interactions: Acknowledge effort and incorrect answers. The goal is to make students realize that praise is not always appropriate and that effort is the primary goal. Example Messages “That was too easy for you. Let's hope the next one is more challenging so that we can learn something.”!” “Good job! See how taking your time to work through these questions can make you get the right answer?”
  62. 62. Incorrect ResponseStudent effort shown/ correct response Student effort shown / incorrect response Agent Emotion Frustrated students are supported by helpful companions.  Arroyo et al., AIED2009
  63. 63. • Our current Gold standard/Truth: self-reports • During Experiment: – Ask students about their “interest” level and their “excitement” level every 7 minutes, on average. Gathering Student Affect “During” Low High Neutral / Middle
  64. 64. Large Data Sets EventLog Table of a Math Tutoring System. 571,776 rows, just in a year time.
  65. 65. The Student Progress Page 73 Dovan Rai, PhD, WPI
  66. 66. Three Experimental Conditions 74 • No access to the Student Progress Page • “My Progress” button was present (student choice) • Prompt invitation to see “my progress” upon bored (disinterested) or unexcited • Force student to see my progress when student bored (disinterested) or unexcited
  67. 67. How to analyze changes in student affect, from moment to moment? MARKOV CHAIN MODELS
  68. 68. Student Interest Ten thousand Data Points N~230.  SPP Absent  SPP Present SPP  Prompted SPP  Forced
  69. 69. How to compare Markov Chain Models, quantitatively? • Probability of student following a specific path • What is the probability that a student will end up excited, after 3 transitions? • Given that they started in a specific state?
  70. 70. Resulting Data
  71. 71. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
  72. 72. Health Care Research Questions Which interventions improve patients’ affect and learning? Which interventions are best for which patients? What is the impact of deeper companion characteristics? Do affective characters make users “feel” better? How does the gender (ethnicity) of companions impact patients’ attitudes/emotions/learning?
  73. 73.  Detect patient emotion, using Computer Vision or sensors (camera, wrist band, pressure mouse, seat sensors).  Dynamically collect data streams of students’ physiological activity and self-reports of emotions.  Apply interventions once emotion is detected, e.g., animated embodied agents  Integrate computer vision techniques to improve detection of emotion.  Train researchers and educators in techniques to recognize an respond to patients’ emotion and make predictions over larg data sets. Proposed Health Care System
  74. 74. Study Design Patients Control Experimental Intervention Control Experimental Intervention Visit # 1 Visit # 2 Visit # 3 Visit # 4 Randomized • Survey • Audio tape with physician • Survey • Audio tape with physician • Survey • Audio tape with physician • Survey • Survey • Survey • Survey Baseline data Greenfield et al., 1986; Patients’ participation in Medical Care General Internal Medicine
  75. 75. Emotion is vital to learning and using technology to recognize users’ emotion has led to powerful performance results. Detecting and responding to users’ emotion while they work with online environments improves learning. Summary
  76. 76. Thank You ! Any Questions? Detection of and Response to Online Users’ Emotion Department of Quantitative Health Sciences University of Massachusetts Medical School March 24, 2017

×