Bringing characters to life for immersive storytelling discusses techniques for animating virtual characters including behavior trees, finite state machines, and the Behavior Markup Language (BML). BML allows for synchronization of behaviors, animation, speech, and other signals. Behavior trees provide a way to organize complex character behaviors hierarchically and handle parallelism and priorities. The document references many tools and techniques for facial animation, speech recognition, and autonomous character systems that can be used to create believable immersive characters.

1. Bringing characters to life
for immersive storytelling
Dioselin Gonzalez
Lead VR Engineer, Unity Labs
VR With The Best - March 19, 2016

2. Relevant background

3. Since Oct/2016
VR authoring tools
labs.unity.com

4. Character animation
&
Partially autonomous character system

5. Networked VR simulation for
dismounted soldiers
CAVE™
Omnidirectional treadmill

6. M.S. Computer Graphics Technology
Collaborative virtual reality

7. InterPlay: Loose Minds in a Box (2005) - anotherlanguage.org/interplay/003_lmib/

8. Scenario

9. ©2014-2016 AlloyRabbit - http://alloyrabbit.deviantart.com/

10. Sense-Think-Act-Communicate
M. Siegel, "The sense-think-act paradigm revisited," Robotic Sensing, 2003. ROSE' 03. 1st International Workshop on, 2003, pp. 5 pp.-.
doi: 10.1109/ROSE.2003.1218700

11. Facial recognition

12. Action Units and the Facial Action Coding System
Shichuan Du et al. Compound facial expressions of emotion. PNAS 2014;111:E1454-E1462. doi: 10.1073/pnas.1322355111

13. Shichuan Du et al. Compound facial expressions of emotion. PNAS 2014;111:E1454-E1462. doi: 10.1073/pnas.1322355111

14. MPEG-4 Face and
Body Animation (MPEG-4 FBA) [ISO14496]
International Standard
Facial Feature Points (FP)
https://visagetechnologies.com/uploads/2012/08/MPEG-4FBAOverview.pdf

15. Shichuan Du et al. Compound facial expressions of emotion. PNAS 2014;111:E1454-E1462. doi: 10.1073/pnas.1322355111

16. Facial Feature Points (FP)
Fully Automatic Facial Feature Point
Detection Using Gabor Feature Based
Boosted Classifiers
Danijela Vukadinovic and Maja Pantic
Delft University of Technology
2005 IEEE International Conference on Systems, Man and
Cybernetics
http://ibug.doc.ic.ac.uk/media/uploads/documents/VukadinovicPantic-SMC05-FINAL.pdf

17. Open Source Computer Vision Library
http://docs.opencv.org/3.1.0/d7/d8b/tutorial_py_face_detection.html#gsc.tab=0

18. OpenCV face Detection using Haar Cascades
http://docs.opencv.org/3.1.0/d7/d8b/tutorial_py_face_detection.html#gsc.tab=0

19. visagetechnologies.com
visage|SDK 8.0

20. www.ulsee.com

21. https://facerig.com/

22. Other facial features
Using Cr-Y Components to Detect Tongue
Protrusion Gesturess
Crawford, C.; Bailey, S.; Badea, C.; Gilbert, J.
CHI EA '15 Proceedings of the 33rd Annual ACM Conference
Extended Abstracts on Human Factors in Computing Systems
http://dl.acm.org/citation.cfm?id=2732916

23. Speech recognition

24. CMUSphinx toolkit
Pocketsphinx — lightweight recognizer library written in C.
Sphinxbase — support library required by Pocketsphinx
Sphinx4 — adjustable, modifiable recognizer written in Java
Sphinxtrain — acoustic model training tools
http://cmusphinx.sourceforge.net/

25. OpenEars®
Offline iPhone Voice
Recognition and Text-To-
Speech
http://www.politepix.com/openears/

26. #import <OpenEars/OELanguageModelGenerator.h>
#import <OpenEars/OEAcousticModel.h>
- (void)viewDidLoad
{
OELanguageModelGenerator *lmGenerator = [[OELanguageModelGenerator alloc] init];
NSArray *words = [NSArray arrayWithObjects:@"WORD", @"STATEMENT", @"OTHER WORD", @"A
PHRASE", nil];
NSString *name = @"NameIWantForMyLanguageModelFiles";
NSError *err = [lmGenerator generateLanguageModelFromArray:words withFilesNamed:name
forAcousticModelAtPath:[OEAcousticModel pathToModel:@"AcousticModelEnglish"]];
NSString *lmPath = nil;
NSString *dicPath = nil;
if(err == nil) {
lmPath = [lmGenerator pathToSuccessfullyGeneratedLanguageModelWithRequestedName:@"
NameIWantForMyLanguageModelFiles"];
dicPath = [lmGenerator pathToSuccessfullyGeneratedDictionaryWithRequestedName:@"
NameIWantForMyLanguageModelFiles"];
} else {
NSLog(@"Error: %@",[err localizedDescription]);
}
}
OpenEars language model

27. The characters’ “brain”

28. Finite state machines
http://gamedevelopment.tutsplus.com/tutorials/finite-state-machines-squad-pattern-using-steering-behaviors--gamedev-13638

29. boost::statechart
http://www.boost.org/doc/libs/release/libs/statechart/doc/tutorial.html

30. http://www.boost.org/doc/libs/release/libs/statechart/doc/tutorial.html
namespace sc = boost::statechart;
struct EvShutterHalf : sc::event< EvShutterHalf > {};
struct EvShutterFull : sc::event< EvShutterFull > {};
struct EvShutterRelease : sc::event< EvShutterRelease > {};
struct EvConfig : sc::event< EvConfig > {};
struct NotShooting;
struct Camera : sc::state_machine< Camera, NotShooting >
{ /* ... */};
struct Idle;
struct NotShooting : sc::simple_state<NotShooting, Camera, Idle >
{
typedef sc::custom_reaction< EvShutterHalf > reactions;
sc::result react( const EvShutterHalf & );
};
struct Idle : sc::simple_state< Idle, NotShooting >
{
typedef sc::custom_reaction< EvConfig > reactions;
// ...
sc::result react( const EvConfig & );
};

31. boost::msm
http://www.boost.org/doc/libs/release/libs/msm/doc/HTML/ch03s02.html

32. http://www.boost.org/doc/libs/release/libs/msm/doc/HTML/ch03s02.html
struct transition_table : mpl::vector<
// Start Event Target Action Guard
// +---------+------------+-----------+---------------------------+----------------------------+
a_row< Stopped , play , Playing , &player_::start_playback >,
a_row< Stopped , open_close , Open , &player_::open_drawer >,
_row< Stopped , stop , Stopped >,
// +---------+------------+-----------+---------------------------+----------------------------+
a_row< Open , open_close , Empty , &player_::close_drawer >,
// +---------+------------+-----------+---------------------------+----------------------------+
a_row< Empty , open_close , Open , &player_::open_drawer >,
row< Empty , cd_detected, Stopped , &player_::store_cd_info , &player_::good_disk_format >,
row< Empty , cd_detected, Playing , &player_::store_cd_info , &player_::auto_start >,
// +---------+------------+-----------+---------------------------+----------------------------+
a_row< Playing , stop , Stopped , &player_::stop_playback >,
a_row< Playing , pause , Paused , &player_::pause_playback >,
a_row< Playing , open_close , Open , &player_::stop_and_open >,
// +---------+------------+-----------+---------------------------+----------------------------+
a_row< Paused , end_pause , Playing , &player_::resume_playback >,
a_row< Paused , stop , Stopped , &player_::stop_playback >,
a_row< Paused , open_close , Open , &player_::stop_and_open >
// +---------+------------+-----------+---------------------------+----------------------------+
> {};

33. Behavior trees
Image from http://charliepark.tumblr.com/post/97667486/a-sample-behavior-tree-from-spore-from-chris

34. Behavior trees

35. Behavior trees
HALO 2 (2004)
GDC 2005 - Handling Complexity in the Halo 2 AI
Image from http://en.wikipedia.org

36. Behavior trees
Façade: a one-act interactive drama (2005)
GDC 2005 - AI and Interactive Storytelling: How We Can Help Each Other
Image from http://www.macobserver.com/

37. Behavior trees
SPORE
GDC 2010 - Behavior Trees: Three Ways of Cultivating Strong AI
Image from http://en.wikipedia.org

38. BT_root
decider
decider
decider
decider
decider
behavior(s)
behavior(s)
behavior(s)
behavior(s)decider
behavior(s)decider

39. Group decider
Behavior decider
BT_root
decider
decider
decider
decider
decider
behavior(s)
behavior(s)
behavior(s)
behavior(s)decider
behavior(s)decider

40. id type data
... ... ...
Blackboard
BT_root
decider
decider
decider
decider
decider
behavior(s)
behavior(s)
behavior(s)
behavior(s)decider
behavior(s)decider

41. bool GroupDecider::decide(Blackboard &);
BT_root
decider
decider
decider
decider
decider
behavior(s)
behavior(s)
behavior(s)
behavior(s)decider
behavior(s)decider

42. enum Status {RUNNING, SUCCESS, FAILURE};
bool BehaviorDecider::activate(Blackboard &);
bool BehaviorDecider::deactivate(Blackboard &);
Status BehaviorDecider::tick(Blackboard &);
BT_root
decider
decider
decider
decider
decider
behavior(s)
behavior(s)
behavior(s)
behavior(s)decider
behavior(s)decider

43. Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)

44. Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)
? ?

45. Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)
12:00 pm
? ?

46. Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)
12:00 pm
? ?

47. Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)
? ?

48. Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)
? ?

49. Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)
? ?

50. Group decider types - Sequence
BT_root
Mix_ingredients
Bake
behavior(s)
behavior(s)
behavior(s)
Put_food_in
Preheat_oven
Decorate behavior(s)
→ →

51. Group decider types - Parallel
BT_root
Sing
Dance
behavior(s)
behavior(s)
behavior(s)
Jump
Wave_arms
Idle behavior(s)
→
→
?
→
→

52. More group decider types (and decorators)
random
repeat
<x>
times
loop_seq

53. Behavior Designer @ Unity Asset Store
http://www.opsive.com/assets/BehaviorDesigner/

54. Concurrency: Resource allocators
http://aigamedev.com/open/article/popular-behavior-tree-design/

55. Aborting/Enabling subtrees
http://aigamedev.com/open/article/popular-behavior-tree-design/

56. Hybrid architectures
Horizontal layering
Sensor input
Action output
http://www.inf.ed.ac.uk/teaching/courses/abs/slides/abs05-reactivehybrid.pdf

57. Hybrid architectures
Horizontal layering Vertical layering
Sensor input
Action output
Sensor input
Action output
One-pass control
http://www.inf.ed.ac.uk/teaching/courses/abs/slides/abs05-reactivehybrid.pdf

58. Hybrid architectures
Horizontal layering Vertical layering
Sensor input
Action output
Sensor input
Action output
Sensor input Action output
One-pass control Two-pass control
http://www.inf.ed.ac.uk/teaching/courses/abs/slides/abs05-reactivehybrid.pdf

59. InteRRaP
http://www.sharprobotica.com/2011/01/interrap-hybrid-architecture-for-robotic-multi-agent-systems/

60. Behavior realization language
(i.e. messaging protocol)

61. (some) Issues
AI Sound
Animation
Physics

62. (some) Issues
AI Sound
Animation
Physics
Behavior
expression

63. (some) Issues
AI Sound
Animation
Physics
Timing

64. (some) Issues
AI Sound
Animation
Physics
Synchronization

65. (some) Issues
AI Sound
Animation
Physics
Interruption

66. Behavior Markup Language (BML)
Intent planning Behavior planning Behavior realization
SAIBA framework (Situation, Agent, Intention, Behavior, Animation)
FML BML
Towards a Common Framework for Multimodal Generation: The Behavior Markup Language
http://xenia.media.mit.edu/~kris/ftp/BML-IVA-06-KoppEtAl.pdf

67. Behavior Markup Language (BML)
http://www.mindmakers.org/projects/bml-1-0/wiki

68. BML: synchronization
<bml id="bml1" xmlns="http://www.bml-initiative.org/bml/bml-1.0" character="Alice">
<pointing id="behavior1" target="blueBox" mode="RIGHT_HAND" start="speech1:start"/>
<speech id="speech1"><text>Look there!</text></speech>
...
<constraint>
<synchronize>
<sync ref="speech1:sync4"/>
<sync ref="beat1:stroke:2"/>
<sync ref="nod1:stroke"/>
</synchronize>
</constraint>
http://www.mindmakers.org/projects/bml-1-0/wiki

69. BML: behavior elements
Towards a Common Framework for Multimodal Generation: The Behavior Markup Language
http://xenia.media.mit.edu/~kris/ftp/BML-IVA-06-KoppEtAl.pdf

70. What is interactive storytelling anyway?

71. What is interactive storytelling anyway?
Image from http://ragreynolds.net/review-the-last-of-us-remastered/

72. More references
GDC 2005 Handling Complexity in the Halo 2 AI - Gamasutra, GDC Vault.
AiGameDev.com Behavior Trees for Next-Gen Game AI - Part 1, Part 2, Part 3.
University of Denver Video Game AI: Behavior Trees & Blackboards
GDC AI Summit 2010 Behavior Trees: Three Ways of Cultivating Strong AI - GDC Vault, AiGameDev.com
Chatting Up Façade’s AI: 23 Ideas to Talk Your Game Into - AiGameDev.com
Popular Approaches to Behavior Tree Design - AiGameDev.com