Дмитрий Тетерюков (Сколковский институт науки и технологий) NurseSim: Virtual Reality Simulator for Nurse Training with Haptic Feedback

1. Dzmitry Tsetserukou
Assistant Professor
Intelligent Space Robotics Laboratory
NurseSim:
Virtual Reality Simulator for Nurse Training
with Haptic Feedback
The best way to predict the future is to
invent it.
Steve Jobs
Dzmitry Tsetserukou, Yuuki Nakagawa and Kazuhiko Terashima

2. EIIRIS: Electronics-Inspired Interdisciplinary Research Institute
Professor K.Terashima Professor J. Miura
M.S. Y. Nakagawa M.S. S. HosokawaPhD J. Sugiyama M.S. H. Sugiura

3. Introduction
Background: Serious understaffing at the care person
– Japan needs twice amount of care person next 20 years.
– It’s necessary to raise international human resources having nursing care skill.
– Language difference causes a barrier of providing a technical assistance and training.
Purpose: Develop novel nurse training system with haptic technology
– We develop nurse training simulator with “sense of force” but without a language.
– Simulator helps nurse training and achieve shortening of training period.
– Reduction personal costs by using this simulator.
Haptic: Tactile feedback technology which takes
advantage of the sense of touch.
Care person from Indonesia

4. NurseSim: Nurse Training Simulator
• Using this haptic interface, we achieve effective and low cost training
system compared to using a human manikin or a robot.
• It shows an optimal motion with sense of force, effective and easy to
understand teaching is realized.
User wearing ExoInterface Virtual Care Receiver
NurseSim：Nursing 3D Simulator as Education System
ExoInterface: Wearable Haptic Interface
Haptic Interaction
Virtual weight, 3D graphics
Posture information
3D Glass
3D Projector
IR Emitter
Xtion: Motion Capture Camera

5. Wearable Haptic Interface
Elbow Unit
Wrist Unit
Fixing Plate
Flexible Belt
Belt Shaft
DC Motor
Unit Base
Timing Belt
Pulley
Bearing Motor Holder
DC Motor
Bearing
Connector
Unit Base
Motor Shaft
Elbow Unit Wrist Unit
• We developed wearable haptic interface for an interaction in VR environment.
• The flexible belt is connected from upper arm to wrist, and from wrist to hand.
• DC motors embedded in the interface pull flexible belts, then joint torques are generated.
• This device reproduces the human muscle structure by generating extensor and flexor muscle
torques at the user‘s arm joint.
• Doesn’t limit user’s movement and can be used all over the place.

6. 0 20 40 60 80 100 120 140 160 180
0.2
0.4
0.6
0.8
1
Elbow Angle[deg]
BeltLength[m]
0 20 40 60 80 100 120 140 160 180
0
0.5
1
1.5
2
2.5
Elbow Angle[deg]
JointTorque[Nm]
Interface Geometrical Modeling
• It is necessary to prepare not only devices but also geometric models of that for generating
the arbitrary joint torques in human arm.
• From this interface model, we calculated belt length and joint torque by MATLAB.
• This model helps for dynamic computation or simulation in virtual environments like
bringing sense of physical interaction.
e
eu
bul
eml
fal
wuF
uwuF sin
eud
Upper Motor
Belt
u
Wrist Joint
Elbow JointHand
Upper Arm
e
Elbow Unit Geometry Model







 



eufa
bueuemfa
wufaeu
bl
ldll
Fl
2
cos
sin
2222
1


Elbow Joint Torque is
Belt Length and Joint Torque

7. AR Application Experiment
D/AA/D
USB
PC
Sensor
Circuit
Motor Driver Motor
Flex
Sensor
WireWire
Camera
HMD
VGA
Wire
AR Experiment Configuration
• We designed an application using AR technology.
• The user wears a haptic interface and HMD with a camera.
• When camera recognizes AR marker, virtual object appears on it.
• When a virtual ball touches surface of the hand, the interface reproduces force feedback to elbow
and wrist.
AR (Augmented Reality): Technology which combines real world
and virtual world, and shows its mixed image to the user.

8. Dynamics Calculation Experiment
Human Model
Foot
Lower
Leg
Upper
Leg
Waist
Upper
Body
Hand
Lower
Arm
Upper
Arm
Head
• To create virtual nurse training simulator, we developed the dynamics calculation space and 3D
human body model with ODE.
• Using ODE, accurate dynamics calculation and good visualization are implemented.
• The model has 9 DOF in its arm joint and waist and it’s controlled by the user’s posture.
Xtion
ScreenOpenCV
OpenNI
Angle
Calculation
Algorithm
P control
RGB, Depth(m) Joint Point(x,y,z) Joint Angle(rad)
Angular
Velocity
(rad/s)
Joint Torque(Nm)
Current(A)
Virtual
Object
Load(N)
Human
Model
ODE Space
Interface
Model
Interface
RGB
ODE System Block Diagram
ODE(Open Dynamics Engine): Open source physical engine which is used for
simulating the dynamic interactions between bodies in space.

9. Lifting Rigid Box and Multijoint Object
• The user can transform posture to optimal joint angle referring to ODE’s visual information.
• We can design a care receiver body model with the same method and realize interaction
between human and human.

10. NurseSim
Wrist
Elbow
Whole Body
• The high precision tracking by optical motion capture
• Nurse care scenario can be changed easily in Unity
• Learning by the force feedback interface
Effective nurse training can be realized
PC1 PC2
Camera
Hub
Display
Motor
Driver
Trainer
with
Markers
Skeleton data Voltage
Current
OptiTrack Camera
Image
Force
Feedback
Image
Interface
Motors

11. Psychophysical experiment
• Experimental results y = d -
a- d( )
1+ x / c( )
b
a = -0.0020187 b = 3.05279
c = 4.30725 d = 1.05594
a = -0.0020187 b = 3.05279
c = 4.30725 d = 1.05594
a = 0.046811 b = 4.69180
c = 4.48232 d = 0.97842
Elbow unit only Wrist unit only Both units
Sigmoid Curve
(4-parameter logistic curve)
Elbow Unit Wrist Unit Both Units
Upper DL [g] 135.66 139.26 133.92
Lower DL [g] 78.86 94.78 88.28
0
0.25
0.5
0.75
1
0 20 40 60 80 100
Rateof"heavy"response
Weight [g]
Rate of heavy Sigmoid Curve
20 60 100 140 180 220
0
0.25
0.5
0.75
1
0 20 40 60 80 100
Rateof"heavy"response
Weight [g]
Rate of heavy Sigmoid Curve
20 60 100 140 180 220
0
0.25
0.5
0.75
1
0 20 40 60 80 100
Rateof"heavy"response
weight [g]
Rate of heavy Sigmoid curve
20 60 100 140 180 220

12. AR experiment comparing the same size ball
• Experiment outline
Left hand weight Right hand weight
Content： Virtual ball bouncing to the left and right hand
Question： Which weight: in left hand or right hand is heavier?
Answers： Left weight is heavy/ Slightly heavier / Same / Right is heavy / Right is heavy
Number of repetitions： 25 times
Simultaneous comparison
• Experimental conditions
– Haptic interfaces at wrist and elbow joints
– HMD，wearable web camera
– Earplugs（To prevent noise from actuators）
– Left and right hand are bent at 90 degree
– Weight of ball 5 stages(80~160g)，random order

13. AR: experiment comparing the same size of
ball
• Experimental results
LeftHandWeight
Right Hand Weight
80g 100g 120g 140g 160g
80g 3.2 4.8 2.2 5.0 4.6
100g 2.8 4.0 4.6 4.6 5.0
120g 1.4 3.4 3.2 4.2 3.8
140g 1.4 2.6 3.4 4.0 4.0
160g 1.0 2.2 3.0 2.2 2.8
LeftHandWeight
Right Hand Weight
80g 100g 120g 140g 160g
80g 0.95 0.80 0.55 1.00 0.90
100g 0.80 0.75 0.85 0.85 1.00
120g 0.85 0.65 0.95 0.95 0.95
140g 0.90 0.85 0.65 0.75 1.00
160g 1.00 0.70 0.75 0.95 0.95
LeftHandWeight
Right Hand Weight
80g 100g 120g 140g 160g
80g 3 4 4 5 5
100g 2 3 4 4 5
120g 2 2 3 4 4
140g 1 2 2 3 4
160g 1 1 2 2 3
Percentage of correct answers
Overall percentage of
correct answers 85％
Correct value, n
Experimental value, m
p =1-
m-n
4

14. NurseSim3D

15. Best Student Presentation Award,
Oct. 24-25, Irago 2013
Awards
Student Excellence Award, March 6, 2014
Best paper Award, April 7, 2010

16. LinkTouch3D:
a Novel Wearable Haptic Display with Parallel
Mechanism for Presentation of Three-DOF
Tactile Feedback at the Fingerpad

17. Спасибо за внимание!
Thank you for your attention!
ご清聴ありがとう
ございました