A technique that allows blind users to pla

1. Real-time Sensory Substitution to Enable
Players who are Blind to Play Video games
using Whole-Body Gestures
Tony Morelli, Eelke Folmer - FDG’11, Bordeaux
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

2. Gesture Based Interaction
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

3. Reliance on Visual Cues
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

4. Reliance on Visual Cues
dodge
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

5. Reliance on Visual Cues
dodge
♫
♫
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

6. visual impairment & obesity
60,000 Blind Children in US More obese
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

7. Barriers to physical activity
Rely on sighted guide Safety
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

8. Exercise games
★ Moderate to Vigorous PA (MVPA) ★Independently
★60 min of MVPA daily (CDC) ★Safer
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

9. Research Question
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

10. Gestures w/o visual cues?
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

11. Gestures w/o visual cues?
?
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

12. Sensory Substitution
Visual
information
Haptic Audio
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

13. Sensory Substitution
closed captioning
Visual
information
Haptic Audio
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

14. Sensory Substitution
closed captioning
Visual
information
Haptic Audio
braille speech synthesis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

15. SS with Gestures?
Wii Sports VI Tennis
serve ♫
bounce ♫ Bu
zz
return
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

16. User studies
vigorous
moderate
light
VI Tennis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

17. User studies
vigorous
moderate
light
VI Tennis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

18. User studies
vigorous
moderate
Upper Body
light
VI Tennis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

19. User studies
vigorous
moderate
Upper Body
light
VI Tennis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

20. User studies
vigorous
moderate
light
VI Tennis
Whole Body
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

21. Whole Body Exergames
Yield Vigorous PA [Biddis & Irwin 2010]
Track Gestures using Computer Vision
Problems with Sensory Substitution
»Systems are Controller less
»Access to Source code Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

22. Real Time Sensory Substitution
Le*!!
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

23. Video Analysis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

24. Instrumentation
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

25. Real time Video Analysis
<SECTION>
<XMIN>100</XMIN>
<XMAX>300</XMAX>
<YMIN>150</YMIN>
<YMAX>300</YMAX>
<MINR>230</MINR>
<MAXR>999</MAXR>
<MING>230</MING>
<MAXG>999</MAXG>
<MINB>0</MINB>
<MAXB>100</MAXB>
<ACTION
<WIIMOTE>
<ID>0</ID>
<DURATION>-1</DURATION>
</WIIMOTE>
<AUDIO>
<ID>alert.wav</ID>
<LOOP>0</LOOP>
</AUDIO>
</ACTION>
</SECTION>
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

26. Real time Video Analysis
<SECTION>
<XMIN>100</XMIN>
<XMAX>300</XMAX>
<YMIN>150</YMIN>
<YMAX>300</YMAX>
<MINR>230</MINR>
<MAXR>999</MAXR>
<MING>230</MING>
<MAXG>999</MAXG>
<MINB>0</MINB>
<MAXB>100</MAXB>
<ACTION
<WIIMOTE>
<ID>0</ID>
<DURATION>-1</DURATION>
</WIIMOTE>
<AUDIO>
<ID>alert.wav</ID>
<LOOP>0</LOOP>
</AUDIO>
</ACTION>
</SECTION>
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

27. Real time Video Analysis
<SECTION>
<XMIN>100</XMIN>
<XMAX>300</XMAX>
<YMIN>150</YMIN>
<YMAX>300</YMAX>
<MINR>230</MINR>
<MAXR>999</MAXR>
<MING>230</MING>
<MAXG>999</MAXG>
<MINB>0</MINB>
<MAXB>100</MAXB>
<ACTION
<WIIMOTE>
<ID>0</ID>
<DURATION>-1</DURATION>
</WIIMOTE>
<AUDIO>
<ID>alert.wav</ID>
<LOOP>0</LOOP>
</AUDIO>
</ACTION>
</SECTION>
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

28. Real time Video Analysis
<SECTION>
<XMIN>100</XMIN>
<XMAX>300</XMAX>
<YMIN>150</YMIN>
<YMAX>300</YMAX>
<MINR>230</MINR>
<MAXR>999</MAXR>
<MING>230</MING>
<MAXG>999</MAXG>
<MINB>0</MINB>
<MAXB>100</MAXB>
<ACTION
<WIIMOTE>
<ID>0</ID>
<DURATION>-1</DURATION>
</WIIMOTE>
<AUDIO>
<ID>alert.wav</ID>
<LOOP>0</LOOP>
</AUDIO>
</ACTION>
</SECTION>
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

29. Real time Video Analysis
<SECTION>
<XMIN>100</XMIN>
<XMAX>300</XMAX>
<YMIN>150</YMIN>
<YMAX>300</YMAX>
<MINR>230</MINR>
<MAXR>999</MAXR>
<MING>230</MING>
<MAXG>999</MAXG>
<MINB>0</MINB>
<MAXB>100</MAXB>
<ACTION
<WIIMOTE>
<ID>0</ID>
<DURATION>-1</DURATION>
</WIIMOTE>
<AUDIO>
<ID>alert.wav</ID>
<LOOP>0</LOOP>
</AUDIO>
</ACTION>
</SECTION>
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

30. Real time Video Analysis
<SECTION>
<XMIN>100</XMIN>
<XMAX>300</XMAX>
<YMIN>150</YMIN>
<YMAX>300</YMAX>
<MINR>230</MINR>
<MAXR>999</MAXR>
<MING>230</MING>
<MAXG>999</MAXG>
<MINB>0</MINB>
<MAXB>100</MAXB>
<ACTION
<WIIMOTE>
<ID>0</ID>
<DURATION>-1</DURATION>
</WIIMOTE>
<AUDIO>
<ID>alert.wav</ID>
<LOOP>0</LOOP>
</AUDIO>
</ACTION>
</SECTION>
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

31. Real time Video Analysis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

32. Evaluate
How efficient is RTSS?
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

33. Thursday, June 30, 2011

34. Thursday, June 30, 2011

35. User Study
Anonymously recruited and tested
Evaluated 2 versions of the hurlde game
»Regular (Audio + Video)
»RTSS ( Audio + Haptic)
28 sighted participants
Novice players
Switch between versions a*er 3 players
12 people played Regular <-- data loss
16 people played RTSS
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

36. Results
version N success time
(seconds)
RTSS 12 0.39
(δ=0.26) 28.67
(δ=8.41)
Regular 16 0.37
(δ=0.32) 26.76
(δ=4.76)
No Significant Difference p>.05
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

37. Jump Timing Accuracy
Significant Difference p<0.05
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

38. Observations
Game Restarts
»False Starts
‣Regular: 2 players started too early
‣RTSS: None
»Out of Range
‣Regular: None
‣RTSS: 3 players out of range
Successful jump depends on pace
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

39. Limitations of RTSS
Kinect Sports Kinect Adventures
★ Predefined gestures ★ Different gestures
★ Fixed target ★ Random targets
★ 1 haptic cue ★ (directional) haptic cues?
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

40. Future Work
Techniques for keeping
players in range and
facing the camera.
Multiplayer
Detecting cues/objects
Test with Blind Kids & Measure Energy Expenditure
Integrate RTSS in Console OS
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

41. Questions
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011

42. Questions
?
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011