This document provides an introduction to building self-adaptive software through an affective-driven software factory approach. It discusses (1) how affective states can be sensed using various modalities like facial expressions and physiological sensors, (2) how software can be made self-adaptive by incorporating machine learning to change behavior based on sensed affective states, and (3) how a software factory framework that utilizes design patterns, tools, and reusable components can be used to develop such affective-driven, self-adaptive software.
How to Troubleshoot Apps for the Modern Connected Worker
201209 An Introduction to Building Affective-Driven Self-Adaptive Software
1. An Introduction to Building
Affective-Driven Self-Adaptive Software
through a Software Factory Approach
Javier Gonzalez-Sanchez
javiergs@asu.edu
www.javiergs.com
4. Affective-Driven
(B) Emotions are generally understood as representing a
synthesis of a subjective experience, an expressive
behavior, and a neurochemical activity.
(A) Feeling
Emotion
Mood
Affective State
Affect
(C) facilitation of social communication
9. Face-Based
Face-based emotion recognition systems
These systems infer affective states by capturing images of the users’ facial
expressions and head movements.
We are going to show the capabilities of face-based emotion recognition systems
using a simple 30 fps USB webcam and software from MIT Media Lab [8].
[8] R. E. Kaliouby and P. Robinson, “Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures,” Proc. Conference on
Computer Vision and Pattern Recognition Workshop (CVPRW ‘04), IEEE Computer Society, June 2004, Volume 10, p. 154.
15. Other Sensors…
S. Mota, and R. W. Picard, "Automated Posture Analysis for Detecting Learners Interest Level," Proc. Computer Vision
and Pattern Recognition Workshop (CVPRW 03), IEEE Press, June 2003, vol. 5, pp. 49, doi:10.1109/CVPRW.2003.10047.
Y. Qi, and R. W. Picard, "Context-Sensitive Bayesian Classifiers and Application to Mouse Pressure Pattern
Classification," Proc. International Conference on Pattern Recognition (ICPR 02), Aug. 2002, vol 3, pp. 30448, doi:
10.1109/ICPR.2002.1047973.
M. Strauss, C. Reynolds, S. Hughes, K. Park, G. McDarby, and R.W. Picard, “The HandWave Bluetooth Skin Conductance
Sensor,” Proc. First International Conference on Affective Computing and Intelligent Interaction (ACII 05), Springer-
Verlang, Oct. 2005, pp. 699-706, doi:10.1007/11573548_90.
16. Machine Learning
I. Arroyo, D. G. Cooper, W. Burleson, F. P. Woolf, K. Muldner,
and R. Christopherson, “Emotion Sensors Go to School,” Proc.
Artificial Intelligence in Education; IOS Press, July 2009, vol.
Frontiers in Artificial Intelligence and Applications 200, pp.
17-24.
28. Not One… But Many
Process, Rules, and
Regulations
software FACTORY
Manufacturing using TOOLS…
industrial production… COMPONENTS
are transformed or assembled…
29. Not One… But Many
IMPLEMENTATION:
VALIDATION:
Frameworks
ATAM Tools
Empirical Software Methods
software ARCHITECTURE
MODELING:
Software Design Patterns
Pattern Languages
Model-Driven Design
Component-Based Engineering
32. A Pattern Language
Gonzalez-Sanchez, J., Chavez-Echeagaray, M.E., Atkinson, R. and Burleson, W. (2011) Affective Computing Meets Design
Patterns: A Pattern-Based Model of A Multimodal Emotion Recognition Framework. Proceedings of the16th European
Conference on Pattern Languages of Programs. Irsee, Germany. July 2011.
Gonzalez-Sanchez, J., Chavez-Echeagaray, M.E., Atkinson, R. and Burleson, W. (2012) Towards a Pattern Language for
Affective Systems. Proceedings of the19th Conference on Pattern Languages of Programs. Tucson, Arizona, USA. October
2012. In press.
36. Videogames
Bernays, R., Mone, J., Yau, P., Murcia, M., Gonzalez-Sanchez, J., Chavez-Echeagaray, M.E., Christopherson, C., Atkinson, R., (2012), "Lost in the Dark:
Emotion Adaption", ACM Symposium on User Interface Software and Technology 2012. Cambridge, MA USA. October 2012.
43. Acknowledgements
This research was supported by
Office of Naval Research under Grant N00014-10-1-0143
awarded to Dr. Robert Atkinson,
and by
National Science Foundation, Award 0705554, IIS/HCC
Affective Learning Companions: Modeling and Supporting Emotion During
Teaching awarded to Dr. Beverly Woolf and Dr. Winslow Burleson.