Getting both “results” such as POS data and "processes" including spatio-temporal data on human behavior and environmental stimuli and constraints in an actual service field, it makes the field virtually tangible. Such tangibility must be a key driver not only for understanding what happened there and why it happened more comprehensively, but also for predicting what will happen to facilitate service kaizen. The virtual tangibility can be realized by technologies and methodologies that support the idea of "Lab-forming Field" and "Field-forming Lab" such as IoT (Internet of Things), WoT (Web of Things), and MR (Mixed Reality) encompassing VR (Virtual Reality), AV (Augmented Virtuality), and AR (Augmented Reality). This talk will present several case studies on service kaizen assisted by this kind of framework while introducing the technologies and methodologies we have developed and applied to the actual cases.

1. Service Kaizen through
Lab-forming Field &
Field-forming Lab
Takeshi Kurata1, 2
1 Human Informatics Research Institute, AIST, Japan
2University of Tsukuba, Japan
E-mail: t.kurata@aist.go.jp

2. AR by PDR + Image-based registration
Panorama-based Annotation,
ISWC2001, ISMAR2003など
G
Environmental map
A
B C D
E
A
B
C
F
Input frames
Position at which
a panorama is taken
Position
Direction
235 [deg]
5 [deg]
From the user’s
camera
Located Orientated
2

3. Takeshi Kurata, Ph.D.
• Position:
– Research Group Leader, Service Sensing, Assimilation, and Modeling
Research Group, Human Informatics Research Institute, AIST
– Professor (Cooperative Graduate School Program), Faculty of
Engineering, Information and Systems, University of Tsukuba
• Professional Experience:
– 2011‐2014 Doctoral co‐supervisor, Joseph Fourier University, UJF‐
Grenoble 1, France
– 2012‐ ISO/IEC JTC 1/SC 24 Member
– 2003‐2005 Visiting Scholar, HIT Lab, University of Washington
• Education:
– 2007 Ph.D. (Eng.) from Doctoral Program in Graduate School of
Systems and Information Engineering, University of Tsukuba
– 1996 M.E. from Doctoral Program in Engineering, University of Tsukuba
• Research Interests:
– Service Research, Assistive technology, Wearable/Pervasive Computing,
Mixed and Augmented Reality, Computer Vision
3

4. Lab-forming Field &
Field-forming Lab
• Borrowing from “Terraforming”
• Lab-forming Field: Transforming a real field
into a lab-like place. (IoT)
• Field-forming Lab: Transforming a laboratory
into a field-like place. (VR)
4

5. 5
測って図る
Measure
Weigh
Survey
Hakaru Hakaru
Plan
Design
Attempt

6. Constructing big data structure
with spatial/behavioral information
6

7. ASPR Technologies for Multi-Stakeholders
7

8. CSQCC
(Computer-supported QC Circle)
8 Staying-time rate at each dinning area per personSales at each dinning area per employee
Visualization tool combining human-behavioral and accounting history
Employee taking order
while cleaning up the
guest room
Icons showing the number of
customers at each table
POS data log
Service Characteristics
1. Intangible
2. Heterogeneous
3. Inseparable
4. Perishable
Alleviate the issues due to IHIP

9. QCC in manufacturing industry
Purpose: Productivity improvement
Conventional QCC in service industry
Purpose: Productivity improvement
Subjective QCC in service industry
Purpose: Improvement of CS/ES
w/ reasonable ways to gather
objective data in plants
In 1980s, applying QCC for service industry
w/o reasonable ways to gather objective
data in service fields
In 1990s, Service industry lost interest in QCC
In 2000
QCC in the Service Industry in Japan
9
Computer-supported QCC (CSQCC)
Purpose: Productivity improvement
In 2010
CSQCC in the future
Productivity improvement
Improvement of CS/ES
w/ reasonable ways to
gather subjective data
continuouslyw/ reasonable ways to
gather objective data in
service fields
1950~ Deming Award

10. 3rd CSQCC for newly open (Movie)
10
新宿・山野愛子邸
2014.12.23

11. Case study
in Japanese Restaurant “Ganko”
• Objectives
1. (for AIST) to test the CSQCC
(Computer-Supported QCC)
suites in a real service field.
2. (for the restaurant) to observe
effects of process
improvement planned by
CSQCC.
• Place
– Japanese cuisine restaurant
GANKO Ginza 4-chome
(Tokyo)
• Term
– 1st term
• January 12 to 18, 2011
– 2nd term
• February 3 to 9, 2011
11
Dining area Course dishes
1st term
(Jan. 12-18, 2011)
for observing
ordinary
operations
QC circle
for making
improvement
plans
2nd term
(Feb. 3-9, 2011)
for observing
improved
operations

12. 12 B2
B1
Dinning Area
Kitchen
Office room
Pantry

13. During Discussion in CSQCC
13
Trajectory of a wait staff in lunch time: 12:00-14:00
Fact: Going in and out of the kitchen/office to no small extent.
Possible result: Difficulty in concentrating on guest service.
Cause: Cell phone everywhere, but reservation book only in the office room.
Possible improvement: e-reservation book
Dinning Area
Kitchen
Office room

14. Summary of 1st CSQCC for Wait Staff
14
Grasp of actual condition Shorter stay in dinning area than the manager assumed
Kaizen plan development
(1) Re-composition of service processes (SP)
(2) Thoroughly obeying each division’s roll, (3) Guts
Direct effect Stay ratio in dinning area at dinner time: UP ↑
Spillover effect Number of additional orders at dinner time: UP ↑
Side effect
(Trade-off)
(1) Work load (walking distance): No difference →
(2) Number of additional orders at 3pm: No difference →
Stay ratio in dinning areas
30%
35%
40%
45%
50%
55%
11 12 13 14 15 16 17 18 19 20 21 22
Walking Distance [m]
1,000
1,500
2,000
2,500
11 12 13 14 15 16 17 18 19 20 21 22
Num. of additional orders per customer
0.0
0.4
0.8
1.2
11 12 13 14 15 16 17 18 19 20 21 22Hour Hour Hour
Before
After
Down: Due to SP re-
comp. for preparation
of dinner/party
UP: Much more than time
decreased in Tea hour
No diff.: Due to
no SP re-comp.
No diff.: Despite SP re-comp.
for preparation of dinner/party
UP: due to reduction
of opportunity loss
No diff. on workload
Lunch Tea Dinner Lunch Tea Dinner Lunch Tea Dinner

15. 2nd ＣＳＱＣＣ: Keep your zone!
15
Jan-Feb in 2012
Actions Description
1
Stay longer
in the dining area
Waiting staff should stay longer in the dining
area to serve their customers.
2
Reduce
the movement
Waiting staff should reduce their movement.
3
Keep
your positions
Waiting staff should keep their positions
(Zones). They should not undertake jobs of
other zones and should do their jobs in their
zones.

16. Walk distance of waiting staff per
customer (meters / hour / person)
16
***
* p < .05, ** p < .01, *** p < .001
******
They were able to reduce walking distance while not
reducing staying time in the dining area!

17. Indicators for position keeping
17
B2
B1
Zone Dedication Rate＝Orange／Red
Zone Order Defense Rate ＝Orange／Blue
All of orders in
the staffʼs zone
# of accepted orders by
a staff in the staffʼs zone
The total # of accepted
orders by the staff

18. Relation between skill level and
Zone Defense/Dedication
18
IV. Expert
They take all of orders in
their zone while taking
orders in other zone for
helping others.
II. Fully occupied
They take orders in his/her
own zone but it is not
enough for covering the
zone. Support by other
staffs is needed.
III. Well organized
They take all of orders in
his/her zone, but they don’t
help other zones.
I. Purposeless
They fail to take orders in
his/her zone and take
orders in other zones.
Training is required.
Zoneorderdefenseratio(ZOD):
Theratioof#ofacceptedordersbyastaffin
his/herownzoneoutofallofordersinthezone
Zone dedication ratio (ZD):
The ratio of # of accepted orders by a staff in his/her own zone
out of the total # of accepted orders by the staff
Precision
individual skill Teamwork
performance

19. Before
19
Precision

20. After
20
Improved coverage of
each zone by each staff
Less need for helping
other staffs (zones)
Precision

21. So many kinds of positioning methods
21

22. In the year of 2010
• iPhone 4: the first popular consumer mobile device
equipped with 9-axis sensors including accelerometers,
magnetic sensors, and gyro sensors
22
G-spatial EXPO 2010:
Handheld PDR (Pedestrian
Dead Reckoning) on iPhone 4
(Maybe world’s first-ever live
demo)

23. PDR（Pedestrian Dead-Reckoning）
Estimates velocity vector, relative altitude, and actions
by measurements from waist-mounted sensor module.
 Wearing sensor module on waist
 Easy to wear and maintain
 Easy to measure data for action recognition
 Relatively easily apply for handheld setting
compared to shoe-mounted PDR based on
Zero Velocity Updates (ZUPTs)
23 Handheld PDR
From PDR to PDRplus
10-axis sensors
• Accelerometers
• Magnetic sensors
• Gyro sensors
• Barometer

24. Frontier of PDR:
Walking direction estimation
24 • Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.

25. Frontier of PDR:
Walking direction estimation
25
• Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.
• Long Paper: Christophe Combettes, Valerie Renaudin, Comparison of Misalignment
Estimation Techniques Between Handheld Device and Walking Directions, IPIN 2015.
• FIS was proposed by Kourogi and Kurata in PLANS 2014.
“Globally, the FIS method provides better results
than the other two methods.”
Frequency analysis of Inertial Signals
Forward and Lateral Acc. Modeling
Principal Component Analysis

26. Power-Aware PDR + Bluetooth LE
• Sensor module with PDRLE (power-aware PDR chip) + BLE
• Towards total management system for attendance record,
work/collaboration support, work analysis, and human-resource
developments based on name-card-like devices
27

27. Indoor Pedestrian Positioning Using
SDF (Sensor Data Fusion)
Pedestrian Dead-Reckoning (PDR)
ID
reader
ID
RSSI
Acceleration /
angular velocity
Building Structure/Layout
Magnetic
vector
Magnetometer
Output of
position/orientation
Positioning based on
stationary and mobile nodes
Atmospheric
pressure
Barometer
Trajectory
Sensor/Data Fusion (SDF)
(Particle filter)
Accelerometers
/ gyro-sensors
Walking velocity
Position /
Orientation
Trajectory matching/
Velocity estimation
Absolute
position
3D environment
model
Velocity vector /
Relative altitude /
Action type
Sensor module
Active
RFID tagID
Surveillance
camera/
RGB-D sensor
ID-LED
ID
Video/
Depth

28. Behavior Measurement of workers
at Nursing facility (Supercourt Hirano)
33

29. •Helper
•Night shift
Time flow: RYGSB
0~1hr 1~2hr
4~5hr 5~6hr
2~3hr
6~7hr
3~4hr
34

30. Time flow: RYGSB
•Helper Leader
•Night shift
0~1hr 1~2hr
4~5hr 5~6hr
2~3hr
6~7hr
3~4hr
35

31. 40
60
80
100
40
60
80
100
40
60
80
100
• Nurse R: Role as a leader. Mainly desk work and sometimes vital check of residents.
• Nurse S: Taking care of each resident while relatively flexibly circulating.
Care worker E, I, K Care worker D, H, MCare worker A, G
• Flexibly changing the role?
• Or low skill?
• High skill?
• Or assigned at specific floor?
• Mainly desk work?
# of steps
# of
utterance
(VAD)
# of floor
change
Time spent in
residents’ rooms
Nurse R
Nurse S
# of steps
# of
utterance
# of floor
change
Time spent in
residents’ rooms
# of steps
# of
utterance
# of floor
change
Time spent in
residents’ rooms
Voice Activity Detection (VAD) FrequencyLow High
RestroomBath/Dressing roomResidents’ rooms Corridor Nurse Station Stairs/EV Dining room
Work Analysis in Nursing Home
Validation of the hypotheses on what is related to high skills:
e.g. ‘Workers who are skillful at comprehensive awareness is to talk to
residents frequently everywhere, but each conversation is basically short.’
36

32. Interview with FPV
Passage of Time
+ Over 50% cost reduction on labor cost and preparation
time compared with existing time studies
+ Consideration of customer privacy by not using cameras
+ FPV with less motion sickness
+ Effective in episodic memory retrieval for retrospective
interviews considering bounded rationality
Worker’s trajectory
3D model built
from a set of photos
First-person view (FPV)
CCE (Cognitive Chrono-Ethnography) Lite
Japanese-style hotel at
Kinosaki Onsen (hot spring)
37

33. Pre-evaluation of Kaizen Plan Considering Efficiency and
Employee Satisfaction by Simulation Using Data Assimilation
－Toward Constructing Kaizen Support Framework －
40

34. Results of comparison between the actual
plan and Kaizen plans by simulation
41
We can find Kaizen plans which achieve both Efficiency (Ef) and
Employee Satisfaction (ES) by behavior measurement, modeling, and
simulation.

35. Open Data Contest in Logistics &
PDR Challenge in Warehouse
42

36. Service Field Simulator
•Supporting service design using VR technology
– Evaluating service environment and its process in advance by sensing
and analyzing human behavior in virtual environment
Risk reduction by evaluation of the
new service in advance
comparison between
• current layout and new layout plan
• current process and new process
Acquiring more detail and reliable data
• Various sensors are available because of
limited sensing area
• Easy to control the condition
As is New plan
With EEG
With Eye-Tracker
43

37. Simulators for layout and service process evaluation in advance
• Retail store simulator for marketing
 evaluation of package design in-store situation
 some benefit on cost and flexibility
 prevent to leak new package designs
VR Drugstore for marketing, Kimberly-Clark Inc.
× Insufficient scientific basis for reproducibility
compared with real environment
44

38. Simulators for layout and service process evaluation in advance
•ServLab:
– Simulator as service theatre where professional actors play some roles of
customer and employee to review possible situation
45

39. Design concept of SFS
•Keep sense of direction as well as the real
 small and easy to provide immersiveness
HMD
Full solid angle display
 Ideal display condition
× very complex and need big space
△ Keep sense of horizontal direction
 Simple structure (easy to construct)
 wide field of view
 natural to see holding real objects
Fully omni-directional display
× narrow field of view, low resolution
× eye fatigue
× unnatural to see holding real objects
× latency from head motion to CG rendering
46

40. Design concept of SFS
•imitate the way to move in real fields:
– control virtual viewpoint by walking motion
•hands free: test service process with real tools
•evaluation of physical load to move around
Omni-directional treadmill
 very similar to real motion
× required to get used to control
× initial cost
 Easy and Intuitive action for users
 Lower initial cost
△ have to develop robust detection method
Walking-in-place motion detection
47

41. Continued improvement
SFS Ver. 1.0
• low resolving power: 0.2
• short of vertical FOV
SFS Ver. 2.0
24 Full-HD(1920x1080) 27-inch LCD :
Resolving power is improved to 0.7
SFS Ver. 2.1
40 Full-HD 24-inch LCD :
Vertical FOV is improved
(Upper 35°, Lower
58.5°)
48

42. Case studies for verifying efficiency
•Gaze point analysis using combination of eye-
tracking device and SFS
– Hypothesis
•we can do the same investigation using an eye-tracker and the SFS
as real in-store marketing
in-store marketing experienced person（subjective opinion）：
"the motion of the gazed point in the virtual environment is similar to that in the real
store especially from the entrance to in front of the shelf where target products are
layout"
49

43. Case studies for verifying efficiency
•Investigation for a method for measuring human interest
using EEG and the SFS
50

44. Example of Analysis and Future Work
51
To compare the shopping behavior in detail, we made heat-map visualization of the
stay time for each 50 cm grid in the real and virtual store. The read area indicates
subjects spent longer time than other area. Because position data of the real store
situation is recorded by hand, we only have the discrete position and timestamp
data. Therefore, we could not compare both of them strictly, but we found out we
could get the similar results (Figure 9).
comparison of heat-map visualization of stay time between in the real store (left)
and in the virtual store (right)

45. Abstract
• Getting both “results” such as POS data and "processes" including
spatio-temporal data on human behavior and environmental
stimuli and constraints in an actual service field, it makes the field
virtually tangible. Such tangibility must be a key driver not only for
understanding what happened there and why it happened more
comprehensively, but also for predicting what will happen to
facilitate service kaizen.
• The virtual tangibility can be realized by technologies and
methodologies that support the idea of "Lab-forming Field" and
"Field-forming Lab" such as IoT (Internet of Things), WoT (Web of
Things), and MR (Mixed Reality) encompassing VR (Virtual
Reality), AV (Augmented Virtuality), and AR (Augmented Reality).
• This talk will present several case studies on service kaizen
assisted by this kind of framework while introducing the
technologies and methodologies we have developed and applied
to the actual cases.52