ICES 2015 in Jeju

1. Fairy Tale Directed Training Game
Platform with Brainwaves, Motion
Sensing Cognitive Technologies for
Children with ADHD
November 20, 2015
Kyungeun Park
ICES 2015
The International Conference on Emotion and Sensibility 2015
November 18 (Wed) – 21 (Sat), 2015, Jeju, Republic of Korea
Maryland, U.S.A

2. Contents
• Research Objective
• ADHD Facts
• ADHD Treatment Strategy
• Serious Game and its Effects
• Narrative Therapy
• Behavior Training Game Platform
• Experiments and Analysis
• Demo
• Review and Future Research
2

3. Research Objective
• Fairy Tale Directed Behavior Training Game Platform
– Brain-Computer Interface (BCI) with NeuroSky’s MindWave
– Skeleton and face tracking with Microsoft’s Kinect V2
– Narrative fairy tale contents with characters and contextual instructions
• This Training Game Platform:
– Keeps track of user’s brainwaves and associated behaviors responding to
the given narrative contents or implicit contextual instructions
– Works as a treatment framework for children with ADHD
• to improve the children’s sustained attention
• to control their hyperactive behavior
• to promote reading comprehension capability
• to give them an opportunity of self-examination
• to get them treated emotionally
• to encourage them to feel self-confidence
3

4. ADHD Facts
• ADHD (Attention-Deficit/Hyperactivity Disorder)
– 5% to 15% children in the U.S. have been diagnosed
• American Psychiatric Association (APA): Avg. 5%
• Centers for Disease Control and Prevention (CDC) : Avg. 11%
– Typical Symptoms include:
• Deficits in behavioral inhibition, sustained attention, and
resistance to distraction
• Impaired behavioral inhibition and self-control needed as a
critical foundation for the performance of any task
• Weak social skills
• Low level reading capabilities
4

5. ADHD by the Numbers
5
http://www.healthline.com/health/adhd/facts-statistics-infographic

6. Types of ADHD
• ADD: Attention Deficit Disorder
• HD: Hyperactivity Disorder
• Mixed ADHD: Attention Deficit/Hyperactivity
Disorder
6
Rief, S.F. 2005. How to reach and teach children with
ADD/ADHD: practical techniques, strategies and
interventions, 2nd ed. San Francisco: Jossey-bass.

7. Attention Deficit Disorder (ADD)
7
https://www.pinterest.com/thebluebunyip/adhd/

8. Hyperactivity Disorder (HD)
8
http://www.theatlantic.com/health/archive/2012/02/a-
western-diet-high-in-sugars-and-fat-could-contribute-to-
adhd/252853/

9. ADHD Affects Children:
9
• The more severe, the earlier the diagnosis
– 8 years old: with mild ADHD
– 7 years old: with moderate ADHD
– 5 years old: with severe ADHD
http://www.healthline.com/health/adhd/facts-statistics-infographic

10. ADHD & Co-Existing Conditions
• Children might have co-existing conditions
– learning disabilities
– conduct disorders and difficulties: antisocial behavior,
fighting, and oppositional defiant disorder
– anxiety disorder
– depression
– sleep disorders, etc.
• Boys and girls display very different ADHD symptoms.
– Boys: more likely the attention disorder
– Girls: more internal and less obvious
10

11. ADHD Behavioral Treatment
Strategy
• Develop relatively long-term sustained attention to
successfully complete the instruction (ADD)
• Keep high-level attention to figure out complicate
and boring story or discussion (ADD/HD)
• Encourage children to stay motionless for a while
and behave appropriately in a certain situation (HD)
11

12. Serious Games
12
• Any form of interactive computer-based game
software for one or multiple players to be used
on any platform and that has been developed
with the intention to be more than
entertainment
• Primary purpose beyond pure entertainment
– To train or educate users
– To achieve a desired progress
Serious Games: Mechanisms and Effects, by Ritterfield, Cody, & Vorderer, 2009

13. Categories of Serious Games
• Educational Games
• Social Change & Public Welfare Games
• Occupational Games
• Healthcare Games
• Military Games
• Marketing Games
13

14. Expectations via Serious Games
• As a training environment,
– Draw problem solving strategies
– Resolve the problems
– Experience the processes
14
Serious Games: Mechanisms and Effects, by Ute Ritterfield,
Michael Cody, and Peter Vorderer, 2009

15. Narrative Therapy
• Indirect Narrative Experiences
– A fairy tale story with pain and frustration
• Narrative Experiences 
Catharsis, Empathy 
Change in perspective, Pair effect,
Self-examination, Generalization 
Relief, Problem-solving, Therapy
15

16. Types of Narratives
• Dialogue
• Personal Experience Narrative
• Fairy tale or Fiction
16

17. Dialogue
• Do you want to talk about … ?
• How about school?
17

18. Personal Experience Narrative
• Self-Experienced Story
• Dreams
– Psychoanalytic approach uses interpretation of
dreams and presents solution to psychological
problems.
18
http://www.seedboxpress.com/the-interpretation-of-dreams/

19. Fairy tale or Fiction
• Hansel and Gretel,
Snow White, Cinderella,
Little Red Riding Hood,
Jack and Beanstalk, etc.
– Through painful threats and
failure, a childish character
changes into a matured
character, “Hero.”
19
http://news.nationalgeographic.com/news/2013/11/131129-
little-red-riding-hood-folktale-tehrani-anthropology-science/

20. Structure of Narratives
• Story (plain contents): Character, Action, Space,
Item, etc.
• Discourse (representation): Point of view,
Perspective, Focalization, Tone, Style, Order,
Duration, Frequency, Plot, etc.
20
Scholes, R., Phelan, J., and Kellogg, R., 2006, The Nature of
Narrative, Oxford University Press.

21. Application of Narratives
to the Behavioral Treatment
• Reading narratives requires complete
understanding of the components of a story
and its development processing.
• Children with ADHD can be effectively treated
with well-known, structured narratives (fairy
tales) to improve reading skill, control
hyperactive behavior, and treat emotional
problem.
21

22. Behavior Training Platform
Unity3D
Game Contents Manager
Contents
Kinect V2
MindWave
Logger Replayer
Screen
Capture
MPEG
Processor
Middleware
Event Logger Video Producer
Server
ZooKeeper
Kafka
ElasticSearch DB
Video File Server
Video File Storage
API
Web App Dashboard
User List Experiment Analysis

23. Interactive Data Collection
• Monitoring Concentration with Brain-
Computer Interface (BCI)
• Monitoring Behavior with Motion Sensors
• Interactive Narratives and Intervention Data
• User’s Responses
23

24. Brainwaves via MindWave
• eSense meter of Brainwaves
– Attention
– Meditation
• Saved to the list of Brainwaves
• After finishing game, the data sent to Data Sever
24
Brainwaves
MindWave
Interface
Collect and
Save
Brainwaves
ElasticSearch Collect Event
Logs
Send to
Data Server

25. Kinect Skeleton Structure
25
00_Hip_center 10_Wrist_Right
01_Spine 11_Hand_Right
02_Shoulder_Center 12_Hip_Left
03_Head 13_Knee_Left
04_Shoulder_Left 14_Ankle_Left
05_Elbow_Left 15_Foot_Left
06_Wrist_Left 16_Hip_Right
07_Hand_Left 17_Knee_Right
08_Shoulder_Right 18_Ankle_Right
09_Elbow_Right 19_Foot_Right
Joints Twenty joints

26. Motion and Face Tracking
Skeletal tracking
with Kinect:
recognizing 22
different
motions
Face tracking
with Kinect:
recognizing 6
different
motions
26
• Skeleton Tracking
• Face Tracking

27. Kinect: Motion Recognition
27

28. Interactive Intervention
28Normal playing with the player sitting in place Paused upon recognizing the player’s motion
• Controls the game according to the recognized brainwaves,
skeleton, and face tracking results.
• The main idea of the game is to make the player stay
motionless, sitting still in place for a while.
• To prohibit a game character from proceeding when the
player is not concentrating on the narratives
(video clip to show how it works)

29. Interactive Intervention
29

30. Fairy Tale Contents
• Dynamically controlled fairy tale contents
affect cyberspace with real-world events
• While playing the game, a child with his/her
own identity influences the story and the
discourse of the fairy tale narratives.
30

31. Fisher man and Genie
• 3 Characters
– Fisher man: main character
– Genie: threat then friend character
– Child: outside helper
• Design Strategy
– 6 Sequences
– Real-time events collection and intervention
– Reproduction of past games for post evaluation
and self-examination
31

32. Reading Flow
32

33. Processes
33
Sequence 1 Sequence 2 Sequence 3 Sequence 4 Sequence 5 Sequence 6
Reading
Comprehe
nsion
Fishermanis
preparingfor
fishing.
Fishermanis
headingfor
fishingplace.
Fishermanis
fishinga
bottle.
Genieis
threatening
fishermanto
kill.
Fisherman is
escaping out
ofthelife-
threatening
situation
using
wisdom.
Fisherman
andreader
(child)are
savingGenie
together.
Contextual
Instruction
Greeting Keep high-
level
attention
without
hyper
action
Keep high-
level
attention
according
to
instructions
Comprehen
d complex
dialogue
Comprehen
d complex
dialogue
according
to
instructions
Consider
others’
situation

34. Sequence 1
34
Order
Story
Interface
Comprehension and Contextual
Instruction
Intervention
Event Character Space
1. Greeting Fisher man Indoor Text Greeting
Kinect Mutual interaction
2. Preparation for fi
shing out
Fisher man Indoor Text Preparing fishing
Kinect Response according to the
instruction
Sequence 1
agree request
Yes
Object
search
No
compliment
requestNo
Yes
Leaving more
than 5 minutes
Finish
No
Yes Login

35. 35
Sequence 2
No movement for
15 seconds
Remember
details
test
request
Yes
No
requestNo
compliment
Yes
Leaving more
than 5 minutes
FinishYes
No
Login
Order
Story
Interface
Comprehension and Contextual
Instruction
Intervention
Event Character Space
1. Go to sea shore Fisher man Path to the fishi
ng place
Text Go to the Fishing place Fisherman is falling
down.Kinect Stay motionless
MindWave Sustained attention is needed
2. Quiz Fisher man Path to the fishi
ng place
Text Reading comprehension Repetition
Kinect Answering
3. Go to sea shore Fisher man Path to the fishi
ng place
Text Figuring out the reason why fisher
man is continuously walking to the
fishing place.
Fisherman is falling
down.
Kinect Maintain the sustained attention wit
h the understanding of the boring sit
uation
MindWave Sustained attention is needed.
Sequence 2

36. Sequence 3
36
Sequence 3
No movement for
10 seconds
Help cycle
requestNo
Yes
No movement for
5 seconds
Yes
Concentration
more than 40%
Object
acquisition
empty
Yes
No
Leaving more
than 5 minutes
Finish
No
Yes Login
No
Order
Story
Interface
Comprehension and Contextual
Instruction
Intervention
Event Character Space
1. Fishing Fisher man Sea shore Text Catching a donkey
Recognize this is not wat the fisher
man wanted
Fisherman is falling
down.
Kinect Stay motionless
MindWave Sustained attention according to the
instruction

37. Sequence 4
37
Order
Story
Interface
Comprehension and Contextual
Instruction
Intervention
Event Character Space
1. Opening the g
ourd bottle
Fisher man Sea shore Text Genie is out of the gourd bottle Game temporarily s
tops and waits for t
he child follow the i
nstruction
Kinect Keep motionless to perform the
instruction
MindWave Sustained attention according to the
instruction
2. Dialogue
between
Genie and the
fisher man
Genie
Fisher man
Sea shore Text Genie threats the fisher man to kill Game temporarily s
tops and waits for t
he child follow the i
nstruction
Kinect Keep motionless and read the text
Sequence 4
Concentration
more than 40%
request
Genie appears
No
Yes
Leaving more
than 5 minutes
FinishYes
No
Login

38. Sequence 5
38
Sequence 5
Concentration
more than 40%
request
Genie
disappears
No
Yes
Leaving more
than 5 minutes
Finish
No
Yes Login
Order
Story
Interface
Comprehension and Contextual
Instruction
Intervention
Event Character Space
1. Dialogue
between
Genie and the
fisher man
Genie
Fisher man
Sea shore Text The fisher man is seeking a way
to escape from the life threating
crisis by Genie.
Game temporarily s
tops and waits for t
he child follow the i
nstructionKinect Keep motionless and read the text
2. Closing the
gourd bottle
Genie
Fisher man
Sea shore Text Shutting up Genie in the bottle Game temporarily s
tops and waits for t
he child follow the i
nstruction
Kinect Keep motionless to perform the
instruction
MindWave Sustained attention according to the
instruction
3. Dialogue
between
Genie and the
fisher man
Genie
Fisher man
Sea shore Text Genie asks for saving Game temporarily s
tops and waits for t
he child follow the i
nstruction
Kinect Keep motionless and read the text

39. Sequence 6
39
Order
Story
Interface
Comprehension and Contextual
Instruction
Intervention
Event Character Space
1. Dialogue
between the
fisher man
and the child
Child
Fisher man
Sea shore Text The fisher man asks the child if he
saves Genie.
Game temporarily s
tops and waits for t
he child follow the i
nstruction
Kinect Keep motionless and read the text
2. Opening the g
ourd bottle
Genie
Fisher man
Sea shore Text Saving Genie Game temporarily s
tops and waits for t
he child follow the i
nstruction
Kinect Keep motionless to perform the
instruction
MindWave Sustained attention according to the
instruction
3. Dialogue
among Genie,
the fisher
man, and the
child
Genie
Fisher man
Sea shore Text Genie thanks to the fisher man and
becomes friends each other.
Game temporarily s
tops and waits for t
he child follow the i
nstruction
Kinect Keep motionless and read the text
Sequence 6
Concentration
more than 40%
request
Genie appears
Leaving more
than 5 minutes
Finish
No
Yes Login

40. Load Balancing for
Reading and Behavior Requirements
40
0
50
100
150
200
250
sequence1 sequence2 sequence3 sequence4 sequence5 sequence6
text(/word) interaction(/sec)
content Sequence 1 Sequence 2 Sequence 3 Sequence 4 Sequence 5 Sequence 6
Text (words) 109 97 186 229 203 57
BCI Attention Req.
Duration (sec)
0 105 90 30 30 30
Text (words)
BCI Attention Req.
Duration (sec)

41. Expected Effects via Complicate
Reading with Intervention Scheme
• Reading Skill Improvement: By figuring out complicat
e dialogue in Sequence 4 and 5 with sustained high- le
vel attention
• Behavior Treatment: By training high-level attention,
keeping motionless in each scene, and behaving accor
dingly in a certain situation
• Emotional Treatment: By helping the child experience
feeling of sympathy for Genie especially in Scene 6.
41

42. Treatment Effects via Self-Examination
with Reproduction of the Game
• Visualization of the Child’s Experience with
the Game for Post Evaluation and Self-
Examination
– The child recognizes how Cyberspace was
affected by his/her behavior.
– As an Object Watcher, self-examination may
stimulate changes in his/her behavior
42

43. H/W Configuration
• Windows 8.1, 64 bit
• Ram: 8GB
• USB 3.0
• CPU: Intel Core i5-4590m
• HDD: 500GB SATA3
• GPU: GeForce 705
43

44. Experiments and Analysis
1) Motion Tracking Test
– To see if brainwaves and motion tracking data are
successfully collected while playing the narrative game
– The result shows the motion tracking capabilities
recognize 22 different motions successfully
2) Game Tracking Test
– Skeletal and face tracking have been tested while playing
the game.
– At the same time, the tester’s brainwaves were measured
and collected.
44

45. Game Tracking Results
45
Body
Category
Motion
No. Motions
Tester 1
No. Motions
Tester 2
No. Motions
Tester 3
No. Motions
Tester 4
No. Motions
Tester 5
UpperBody
UBLeft 2 4 1 3 0
UBRight 2 3 0 3 0
UBFront 0 5 2 1 1
UBBack 0 2 1 0 0
UBStand 0 2 0 1 1
UBSit 0 2 0 1 1
Left Hand
LHLeft 3 2 1 3 0
LHRight 2 1 1 3 0
LHFront 0 1 1 0 1
LHBack 0 1 0 0 0
LHUp 2 3 3 4 1
LHDown 0 0 0 0 1
Right Hand
RHLeft 2 2 4 3 0
RHRight 1 1 1 3 0
RHFront 0 1 1 1 0
RHBack 0 1 0 0 0
RHUp 2 1 3 3 1
RHDown 0 0 0 0 1
Left Foot
LFLeft 1 1 1 0 0
LFRight 1 1 1 0 0
LFFront 1 1 2 5 1
LFBack 0 1 0 0 0
LFUp 0 1 0 0 0
Right Foot
RFLeft 1 1 0 0 0
RFRight 1 1 0 0 0
RFFront 1 1 2 5 1
RFBack 0 1 0 1 0
RFUp 0 1 1 0 1

46. Game Tracking Results (cont.)
46
Body
Category
Motion
No. Motions
Tester 1
No. Motions
Tester 2
No. Motions
Tester 3
No. Motions
Tester 4
No. Motions
Tester 5
Head
HDLeft 1 2 1 2 1
HDRight 1 2 0 2 1
HDUp 0 2 0 1 0
HDDown 1 2 0 1 0
HDLeftTilt 0 2 1 1 1
HDRightTilt 0 1 0 1 1
Brainwaves
Avg. eSense
Attention 75.8759 62.7829 51.4263 53.7829 29.0852
Meditation 67.2558 63.0921 61.2523 55.2282 34.1012
Clear Time
(sec)
Total 65 112 67 101 61
Scene1 28 40 26 51 24
Scene2 37 59 41 50 37
Total No. Motion 25 53 28 48 15

47. Web App Dashboard
(Statistics Data)
• User List
• Experiment List
• Players Interaction Log:
– Motion Sensing Data Log: Body and face tracking
– Brainwaves: Attention and meditation tracking
• Web App Dashboard Demo
47

48. User List
48
Click “admin” User

49. Experiment List
49
Experiment List of
“admin” User
Click this event

50. Kinect Motion Tracking and Interaction
50

51. MindWave Brainwaves Tracking
51
Attention Tracking
Meditation Tracking

52. Demo
Fisher man and Genie
52

53. • Themaincontributionistheimplementationofthebehaviortraining
platformwhichcombinesBCIandmotionsensingtechnologywith
fairytalenarrativecontentsandtreatchildrenwithADHDinterms
ofthebehavioraltreatmentandreadingcomprehension.
• Weareexpectingthistrainingplatformtohelpthechildrenimprove
theirattentionandcontrol hyperactivebehavior basedonnarrative
therapystrategy.
• Inadditiontothecurrentnarrativecontent,“FishermanandGenie,”
wehopetoprovidediversecontentswithwide-rangeoffairytales.
• Wewouldliketotestthisplatformasthetreatmentframeworkfor
childrenwithADHDbycollaboratingwithrelatedfieldsandmedical
professionals.
Review and Future Research

54. Q&A
Thank you for your attention!
54
Kyungeun Park, D.Sc.
Dept. of Computer and Information Sciences
Towson University
Towson, Maryland, USA
kpark@towson.edu
kepark@gmail.com