1. Android and OpenNI - NUI Application
Treasure Hunter Robot
Hirotaka Niisato
@hirotakaster
1
©SIProp Project, 2006-2012
1

2. About Myself
Special
Server/Network Technology
Community (OSS)
SIProp.org
Company
GMO Internet, Inc
©SIProp Project, 2006-2012 2

3. NUI - Natural User Interface
©SIProp Project, 2006-2012 3

4. NUI Device
©SIProp Project, 2006-2012 4

5. Device Specifications
Kinect Xtion Pro Live
(Windows)
RGB Camera ○ ○
Depth Sensor ○ ○
Audio ○ ○
Commercial Use ○ ○
Library Kinect for OpenNI
Windows SDK (GPL/LPGL)
©SIProp Project, 2006-2012 5

6. Shop
©SIProp Project, 2006-2012 6

7. Robotics
©SIProp Project, 2006-2012 7

8. others
©SIProp Project, 2006-2012 8

9. OpenNI
Open Source
Environment
●
OS:Windows/Linux(+ARM)/MacOS/Android
●
Java/C++/C#
Modules
●
OpenNI
●
Sensor
●
Middleware(NITE)
©SIProp Project, 2006-2012 9

10. OpenNI with Android Application Pattern.1
USB
©SIProp Project, 2006-2012 10

11. OpenNI with Android Application Pattern.2
USB NW
Android : Toshiba T-01C(Regza Phone) ©SIProp Project, 2006-2012 11

12. Treasure Hunting Robot
©SIProp Project, 2006-2012 12

13. Summary
This is an
"AR(augmented reality) Treasure Hunting Game“
You get virtual treasures by controlling real robot!
©SIProp Project, 2006-2012 13

14. Manual
Look at radar window like dragon radar.
Show some treasure on radar.
Center is a place in which a robot is present.
N(North) is a robot's front.
If you move the robot to a treasure place, you g
et some point.
Controlling the robot is by brain wave.
©SIProp Project, 2006-2012 14

15. How to control
Keep 3 plots/5 plots in same area.
Upper(exciting) area
Turn left
Middle(active) area
Go toward
Lower(relax) area
Turn right
©SIProp Project, 2006-2012 15

16. How to make
©SIProp Project, 2006-2012 16

17. NUI Devices
©SIProp Project, 2006-2012 17

18. Hardwares & Softwares
Hardwares Softwares
Base computer Ubuntu & Android
Pandaboard Linaro11.11
Brain Wave Sensor Depth Sensor
MindWave OpenNI
NeuroSky Bone skeleton tracker
Depth Sensor NITE for ARM
Xtion pro live Only binary…
ASUS UI Framework
Display openFrameworks
AiRscoter Android
Brother Industries
Walking Robot
KHR-3WL
Kondo science
©SIProp Project, 2006-2008 18

19. Hardwares’ photo
©SIProp Project, 2006-2012 19

20. Softwares’ photo
©SIProp Project, 2006-2012 20

21. Calculate robot position
Recognizing a robot by bone skeleton.
Using OpenNI
Calculate direction & position of robot.
Calculate by moving robot.
Using depth camera.
©SIProp Project, 2006-2012 21

22. Recognizing bone skeleton
NITE is NOT OSS!
Only binary for ARM/Ubuntu.
Want to use Android&openFrameworks UI.
Made dual boot OS for Ubuntu&Android.
UserLand: Linaro/Android (Run on Ubuntu by chroot)
UserLand: Linaro/Ubuntu (OpenNI runs on this userland)
Linaro Kernel for Ubuntu&Android on Pandaboard
©SIProp Project, 2006-2012 22

23. Recognizing bone skeleton
NITE can NOT recognize our robot…
Head is too SMALL as human.
Legs & arms are too THIN as human.
Upright pose is not like human.
©SIProp Project, 2006-2012 23

24. Calculate robot position
Calculate a current center position.
Moving distance is calculated by using depth camera.
Calculate a robot's front.
This robot has just 3 ways action.
Go toward, Turn left, Turn right
Direction of “Go toward” is the robot's front.
©SIProp Project, 2006-2012 24

25. How to control
Keep 3 plots in same area.
Upper(exciting) area
Turn left
Middle(active) area
Go toward
Lower(relax) area
Turn right
©SIProp Project, 2006-2012 25

26. Brain wave 1/2
What wave is What feeling?
γ(gamma) wave is exciting.
β(beta) wave is active.
α(alpha) wave is relax.
θ(theta ) wave is sleep.
δ(delta) wave is deep sleep.
δ θ α β γ
Relax Exciting
©SIProp Project, 2006-2012 26

27. Brain wave 2/2
Upper area
γ(gamma) wave is exciting.
Middle area
β(beta) wave is active.
Lower area
α(alpha) wave is relax.
δ θ α β γ
Relax Exciting
©SIProp Project, 2006-2012 27

28. Download source code
We release all source code on our site.
http://www.siprop.org/ja/2.0/index.php?product%2FTreasureHuntingRobot
Do It Yourself ⇒Do It With Others!
©SIProp Project, 2006-2012 28

29. Thank you!
©SIProp Project, 2006-2012 29