Affective Computing by Michal Shmueli-Scheuer, PhD, IBM Haifa Research Lab presented on Cognitive Systems Institute Group Speaker Series call 28 May 2015.

1. © 2015 IBM Corporation
Affective Computing
Michal Shmueli-Scheuer, David Konopnicki – IBM Research, Haifa
Research Lab
May 28, 2015

2. © 2013 IBM Corporation
The ABC that makes up a Human
Being
2
Affect
(feel)
Behavior
(act)
Cognition
(think)

3. What is Affective Computing
Affective computing is the study and
development of systems and devices that
can recognize, interpret, process, and
simulate human affects (Wikipedia)
3

4. A Computer that has EQ
• monitors (one's
own?) and other
people's emotions
• uses emotional
information to
guide thinking
and behavior
4
Crowd Sensing Better Selling
Product Test Affective CRM

5. Emotions can be measured

6. Some Scenarios

7. - “we had an accident. Please come fast.”
Sensing the stress in his voice, the
computer replied with a calm, soothing
voice
- “Sir, please calm down, we are on our
way”
and dispatched emergency forces to the
scene

8. Real-time emotion sensing from voice
Affect-aware Persona adaptation
Affect-aware Customized response
- “we had an accident. Please come fast.”
Sensing the stress in his voice, the
computer replied with a calm, soothing
voice
- “Sir, please calm down, we are on our
way”
and dispatched emergency forces to the
scene

9. - “… those refrigerators have been been
voted ‘Best Design’ of 2014” said the
computer
The customer frowned.
“… and they are environmentally
friendly”…

10. Real-time emotion sensing from video
Affect-aware Dynamic Argument adaptation
- “… those refrigerators have been been
voted ‘Best Design’ of 2014” said the
computer
The customer frowned.
“… and they are environmentally
friendly”…

11. Scenarios
Emotions in CRM Conversational agents Social Media Analytics Emotions in
Organizations
Emotions in Email, IM
Media and
Entertainment: focus
groups
Retail: store behavior Digital analytics Consumer Products:
market research, focus
groups
Automotive: mood
recognition
Education: attention
management
Healthcare Deception prevention

12. © 2013 IBM Corporation© 2015 IBM Corporation
Key roles of Affective Computing
 Making cognitive agents convincing
– Monitoring users’ emotions and being adaptive to them
– Conveying affect in the communication interfaces
 Helping users become aware of their own emotions or
emotions of others
– Supporting better decision making
 Making cognitive agents easier to understand and
follow through regulation of the information flow
– Adapting the flow to the user’s attention level and
emotional state.
12

13. © 2013 IBM Corporation© 2015 IBM Corporation
Analysis Process
Extraction
•Raw emotion
recognizers from
text, speech,
face/body &
sensors
Interpretation
•In-context
emotional state
modeling
Analysis
•Use of user
models in
predictive and
prescriptive
scenarios
Ouput
•Expressive
Communication
with humans
13
Text
Speech
Video
Wearable
Predictive
Prescriptive
Expressive
Speech
Avatars
Application
Context
Modifications
Expressive
Text
Fusion
Primary
Secondary
and
Social
Context
User Model

14. © 2013 IBM Corporation© 2015 IBM Corporation
What we work on
14
Virtual Agents with Emotional
Intelligence
• Develop virtual agents that have
emotional intelligence.
• Improve customer satisfaction
during a conversation by
communicating in an empathetic
manner.
Personal Emotion
Measurement in Mobile using
Video
• Large-scale measurement of
emotional response to content
presentation and content
optimization
Emotion Speller For Email
• Personal assistant for managing
emotions in incoming and outgoing
email
Analysis of Emotional Text in
Enterprise
• Sociological analysis of emotions
expressed in enterprise social text
in order to evaluate mood and
etiquette

15. © 2013 IBM Corporation© 2015 IBM Corporation
Emotions in email
 In-coming email

16. © 2013 IBM Corporation© 2015 IBM Corporation
Emotions in email
 Out-going email

17. © 2013 IBM Corporation17
 Modern workplaces involve a lot of emotions
 Emotions play an integral role in employee motivation, personal
commitment and social interactions at work
 On the other hand, emotional labor inhibits workers from expressing
authentic emotions
 Potential to improve the effectiveness and overall health of large
organizations
 Large scale organizational data and analyzes multiple data sources
with six Ekman’s emotions
 Demographics & temporal effects
 Deep analytics:
– analyzed the evolution of users’ emotion along time and their
characteristics
– groups analysis
– effect of external events on the happiness of employees
Contributions
Emotions in internal social network communications

18. © 2013 IBM Corporation© 2015 IBM Corporation
Emotions in internal social network communications
Time of day:
Happy and Surprised
0.33
0.35
0.37
0.39
0.41
0.43
0.45
0.47
0.49
0.51
0.53
Jan-12M
ar-12M
ay-12
Jul-12
Sep-12Nov-12
Jan-13M
ar-13M
ay-13
Jul-13
Sep-13Nov-13
Jan-14M
ar-14M
ay-14
Jul-14
Averageemotionintensity
175
180
185
190
195
200
205
210
Averagestockvalue
Happiness
Stock
0.09
0.095
0.1
0.105
0.11
0.115
0.12
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Hapiness
0.29
0.295
0.3
0.305
0.31
0.315
0.32
0.325
0.33
0.335
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Emotion display rules

19. © 2013 IBM Corporation
Cognitive Virtual Agents
 Many enterprises use self-service agents that automatically deliver
answer to customers’ queries across a variety of interaction channels.
 These agents have no emotional intelligence:
 They can’t sense emotions
 They can’t express themselves in an emotional manner (e.g.
express empathy)

20. © 2013 IBM Corporation© 2015 IBM Corporation
Personal Emotion Measurement in Mobile using Video
Demographics
and
psychology
questionnaire
Content
Presentation
and User
Capture
User Emotion
Analysis
Reporting
Content
optimization

21. © 2013 IBM Corporation© 2015 IBM Corporation
Thank You!
21