This slide was used at CHI2012 Conference (http://dl.acm.org/citation.cfm?id=2207695). Paper "panavi: recipe medium with a sensors-embedded pan for domestic users to master professional culinary arts"is here http://panavi.jp/panavi_CHI2012.pdf. http://panavi.jp

1. Recipe Medium with a Sensors-Embedded Pan for
Domestic Users to Master Professional Culinary Arts
Daisuke Uriu
Mizuki Namai
Satoru Tokuhisa
Ryo Kashiwagi
Masahiko Inami
Naohito Okude

2. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artscooking/recipe medium
Julia Child “The French Chef” (1963-1973)
- the transition of recipe media from “text” to “video” -
Her cooking show, translating traditional print or verbal
recipe, was a great hit with domestic audiences in the US.

3. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artscooking/recipe medium
Cooking or recipe medium has entered a
transitional period with regards to HCI.
Silver Spoon
text based recipe book
The French Chef
TV show
Vita Craft IHIQ
automatically cooking system product
Digital Thermometer Frying Pan
product
Personal Trainer: Cooking
by NINTENDO

4. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsconcept: title of this paper
panavi: Recipe Medium with a Sensors-
Embedded Pan for Domestic Users to Master
Professional Culinary Arts

5. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsconcept: design overview
panavi: Recipe Medium with a Sensors-
Embedded Pan for Domestic Users to Master
Professional Culinary Arts
display:
digital recipe
application
sensors-
embedded
frying pan
wirelessly
connected
with
the display
programmed
the ways of
professional
chef’s cooking

6. video

8. research contribution

9. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsuncertain environment
This research attempts to obtain ways of
designing computer mediated artifacts that
support uncertain domestic environments: kitchens.
firewater
uncertain domestic
situated actions;
all activities
in kitchens
based on
activities in kitchens cannot be
planned as a static model.
users must manage
uncertain troubles

10. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Arts
Introduction and Motivation
Cooking advanced recipes can be complicated, and
requires following detailed instructions to achieve the
desired result. Textual instructions like paper recipes
are often difficult to comprehend; they use technical
terms or instructions that require prior experience to be
understood (“How much is a handful of chopped basil?”
or “How brown are caramelized onions?”). These types of
information can be transferred more easily using images
or videos, just as cooking shows do. However, cooking
shows are not designed to be followed in real-time.
Although the increasing audience of cooking shows in
European television [4] indicates that there is a
growing interest in this format, no participant of our
preliminary studies followed a recipe while watching a
cooking show. They rather liked the inspiring aspect of
the visual aesthetics. Typically, they would follow
textual instructions when cooking, or not use any kind
of instructions at all.
The final outcome of a recipe depends on many factors
and can fail at any point during the entire process. This
can be demanding and stressful for beginners, perhaps
discouraging them. We are developing PersonalChef, an
interactive kitchen counter (see Fig.1), to avoid
disappointing hobby chefs when preparing unfamiliar
recipes and to increase their confidence.
Related Work
Several systems have been developed to bring technical
innovations into the domestic kitchen environment.
Cooking Navi [7] considers cooking as a time
optimization problem when preparing a menu
consisting of multiple recipes, and tries to optimize
single cooking processes in order to have all dishes
finished at the right time. Semantic Cookbook [10] and
Kitchen of the Future [11] installed various electronic
devices into the kitchen to record and share cooking
sessions. Living Cookbook [12] offers these possibilities
as well, but it further focuses on the social experience
of cooking and collaboration. eyeCOOK [3] and Smart
Kitchen [8] focus on the actual user input. They try to
facilitate active user input or reduce it by tracking
user’s actions in the kitchen using eye-gaze, speech
commands, and foot switches. CounterActive [9]
integrates step-by-step cooking instructions using
multimedia invisibly into the kitchen counter. Besides
research prototypes, there are also commercial
systems. Nintendo’s personal cooking instructor
(nintendo.com) on the small-screen NintendoDS device
is an interactive cookbook and provides live cooking
demonstrations.
While other work often focused on efficiency [1,5], our
system aims to support the entertaining and social
gress (Spotlight on Posters Days 1 & 2) April 12–13, 2010, Atlanta, GA, USA
3404
Figure 1. HeatSink illuminates the stream
of water according to its temperature,
becoming red when hot and blue when cold.
SeeSink
One impediment to automatic faucets in bathroom and
kitchen sinks is the lack of control over temperature and
flow. Nevertheless, their simple application of automation
Figure 2. SeeSink can interpret a variety of tasks being
performed by the user to provide useful hands-free control of
water temperature and flow.
Unfortunately these systems do not directly prevent non-
compliance (which is estimated at 50% in hospitals, for
example [30]). Dirty hands are the primary cause for
infection, and certainly very easy to prevent [3].
CleanSink seeks to motivate critical behavior change by
augmenting the role of the sink as part of the larger context
CHI 2005 PAPERS: Technology in the Home April 2–7 Portland, Oregon, USA
research through design
This paper follows “Research through Design.”
How to evaluates our work
could contribute to the HCI community.
inventionprocess
relevance extensibility

11. related studies

12. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artstraditional text based recipes
Traditional media for recipes describe steps
toward a perfect completion of cooking.
Silver Spoon
very famous recipe book
SPAGHETTI ALLA CARBONARA
SPAGHETTI WITH CARBONARA SAUCE
Ingredient for 1 person
Preparation time: 15' (preparation: 10' cooking: 5')
14 oz (100 g) spaghetti
1.5 oz (45 g) pancetta
1 egg
1 egg yolk
Method
Prepare egg sauce, beating the egg and the egg yolk with Pecorino
cheese and a little pepper in a bowl. Cut the pancetta into small strips.
Place a large skillet onto a medium heat and slowly saute the pancetta.
Cook the spaghetti in a pot of lightly salted boiling water and drain
when al dente. Turn the pasta into the skillet with the pancetta, toss
and turn off the heat. Add the egg sauce and a little of the cooking
water. Stir for 30 seconds or so with a little heat. Turn off the heat, stir
again, and serve immediately.
1 oz (100 g) Pecorino cheese
1 tsp (5 ml) extra virgin olive oil
Salt
Pepper
But it is usually difficult for non-professional users
to reproduce the taste of meals.

13. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Arts“planned” recipe medium
The problem of “planned” traditional recipe is
close to Lucy Suchman’s “situated actions.”
SPAGHETTI ALLA CARBONARA
SPAGHETTI WITH CARBONARA SAUCE
Ingredient for 1 person
Preparation time: 15' (preparation: 10' cooking: 5')
14 oz (100 g) spaghetti
1.5 oz (45 g) pancetta
1 egg
1 egg yolk
Method
Prepare egg sauce, beating the egg and the egg yolk with Pecorino
cheese and a little pepper in a bowl. Cut the pancetta into small strips.
Place a large skillet onto a medium heat and slowly saute the pancetta.
Cook the spaghetti in a pot of lightly salted boiling water and drain
when al dente. Turn the pasta into the skillet with the pancetta, toss
and turn off the heat. Add the egg sauce and a little of the cooking
water. Stir for 30 seconds or so with a little heat. Turn off the heat, stir
again, and serve immediately.
1 oz (100 g) Pecorino cheese
1 tsp (5 ml) extra virgin olive oil
Salt
Pepper
She revealed the
programmed instructions
do not support situated
actions: the errors and
problems users actually
face are not expected by
the system.
the “big green button”
came from her research.
ethnography research
about copy machine
“planned” text recipe

14. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Arts“situated” cooking support
Cooking support systems corresponding
to users’ situated actions
have been already considered.
Bradbury et al.
an interactive cookbook
supporting situated actions
using eye-tracking and
speech recognition
NATURAL INPUTS
eyeCOOK is designed specifically to use natural input
modalities: those that humans use in human to human, non
mediated communication [5]. To reduce the need for users
to provide explicit input, or change their behavior to
accommodate interface constraints, implicitly provided
attentional cues are observed and interpreted. We believe
that this approach improves the learnability and
intuitiveness of interfaces designed for novice users.
Voice Commands
eyeCOOK uses context-sensitive, localized grammars. This
allows more synonyms for a given speech recognition
command, reducing the chance of misinterpreting a word.
Eye Gaze Commands
When the user is in range of the eye tracker, eyeCOOK
substitutes the object of the user’s gaze for the word ‘this’
in a speech command. For example, ‘Define this’ will
trigger the define operation on the current eye gaze target.
Since current eye trackers are spatially fixed and offer
limited mobility to users, the user will not always be in a
location where eye tracker input is available. Our speech
grammar is designed so that system functionality is not
her/his task. To achieve this, we must
increase sensing capability [3], improve
coordination among appliances [5], and give
appliances the ability to affect the
environment [3,5].
Environmental Sensors
Temperature sensors used to keep track of the
status of the oven and the elements of the
stove can increase the system’s ability to
guide the user’s cooking experience and
could be synchronized with electronic timers.
Appliance Information Integration
Integrating knowledge of the environment
can result in improved functionality, taking
up less of the user’s time and effort. For
example, user recipe preferences, timing
constraints, as determined by the user’s
electronic schedule, and currently available
ingredients, communicated by food storage
areas, can be combined to suggest recipes.
As well, selecting a recipe can result in the
addition of necessary ingredients to an
electronic shopping list stored on the user’s
Personal Data Assistant (PDA).
Active Environmental Actions
The kitchen should not only be aware of its environment,
but it should also be able to affect it. Thus, it should be able
to take actions which increase efficiency, and reduce the
user’s action load, like automatically preheating an oven.
CONCLUSIONS
We have presented eyeCOOK, a gaze and speech enabled
multimodal Attentive User Interface. We have also
presented our vision of an Attentive Kitchen in which
appliances, informed by sensors, coordinate their behavior,
and have the capability to affect the environment. This can
reduce the user’s workload, and permit rationalizing
requests for user attention.
REFERENCES
1. Ju, W. et al. (2001). CounterActive: An Interactive
Cookbook for the Kitchen Counter. Extended Abstracts
of CHI 2001 (Seattle, April 2001) pp. 269-270
2. Norman, D. A. The Invisible Computer, MIT press,
1999
3. Schmidt, A. et al. How to Build Smart Appliances.
IEEE Personal Communications 8(4), August 2001.
pp. 66-71.
Figure 1. eyeCOOK in Page Display Mode
Interactive & Student Posters: Computers Everywhere CHI 2003: NEW HORIZONSPosters: Computers Everywhere CHI 2003: NEW HORIZONS
Olivier et al.
situated coaching system
by integrating projection
systems, RFID,
accelerometers, and
under-floor pressure
sensing technologies.
explore different configurations of sensor-dependent display
behavior significantly helps in the exploration and crafting of
design ideas. Figure 4 shows a prototype under development in
which the ambient displays (on the wall above the main worktop)
respond to the turning of the pages of a specially designed
cookbook. The cookbook has an RFID tag embedded in each page
allowing the kitchen to detect the page that the book is currently
open at. Responding to this, the ambient displays present relevant
food information and even personal media related to the recipe
and past times in the cook’s life when the corresponding meal was
prepared.
4.2 A design tool for users
Another significant challenge in designing pervasive computing
applications is the involvement with users in the design process.
We have conducted a number participatory design exercises
involving older users and found that a significant barrier to
exploring design concepts is adequately explaining the scope of
the technologies involved. By demonstrating simple mappings
between sensors and display in demonstration applications in the
Ambient Kitchen we have found that we can greatly improve lay
users' understanding of the potential functionality. For example,
figure 5 shows one such commonly used application in which
sample recipes are projected in response to the ingredients placed
on the bench. Traffic light indicators on the display also show
which of the recipes ingredients are in the kitchen’s cupboards.
Though a simple demonstration of how sensor data can be
mapped to information sources (and then to information displays)
our experience is that such illustrations can help users think about
both more mundane and adventurous (and useful) applications of
such “invisible” technologies.
Figure 5. The Ambient Kitchen has been used to facilitate
your
different people. For an
discussions and focus groups on the topic of pervasive
computing as part of a wider participatory design process
looking at ICT and nutrition for older people.
4.3 An observatory to collect sensor data
Realizing situated services that are responsive to both
actions and intentions requires significant development of activity
recognition algorithms themselves. As such, multi-sensor
benchmarks of everyday activities are not widely available, and as
part of our own research, and to support the research of
collaborators and the wider research community, we are
developing a number of such benchmarks by capturing data for
multiple subjects and activities, and hand annotating these
datasets. Activities range from gaze data for head pose tracking
algorithm development, primarily using video streams alone (see
figure 6), to naturalistic data sets relating to multi-step food
preparation for which RFID, accelerometer, pressure and video
data is collected and hand annotated.
4.4 An evaluation test bed
Evaluation means different things to
engineer the question is "does it work?" That is, are the functional
requirements met, does it complete certain tests accurately and
without failing. For the human factors engineer the question is
"does it perform a useful function in the context that it is intended
to be used". The latter question can only really be answered by
installing the technology in a range of real contexts. In the case of
kitchen technologies, this means real lived-in homes. Constraints
such as household routines and the different uses different
members of a family use different rooms for at different times can
be critical in the success or failure of home technologies. These
constraints are only really apparent in the context of real home
use. Laboratory-based facilities such as the ambient kitchen thus
are of limited use in this respect. However it is possible to use
them to do more limited evaluations of functional requirements
that still have face validity.
Figure 6. Simultaneously captured data from the embedde
y evaluations of this kind but we are
taken. Thus the teabags might be in a container on work surface,
d
cameras allows us to develop a benchmark for attention
detection algorithms based on tracking head pose and position
from multiple viewpoints.
We yet have to complete an
planning to do so. For example, we will use actors trained using
video recordings of people with dementia carrying out simple
kitchen tasks such as making a hot drink. These recordings were
made of people doing these tasks, which were of their own
choosing, in their own kitchens. The intention is to make initial
tests of algorithms to detect when these people need prompting
because they have made an error or stopped in the middle of the
task. It would be disorienting and therefore unrealistic to bring
people with dementia into the lab but these existing videos can be
used to configure the Ambient Kitchen to match the kitchen in the
video and then to get an actor to perform the sequence of actions
Figure 1. The Ambient Kitchen is a lab-based high fidelity
pervasive computing prototyping environment. The kitchen is
situated in the main research space in Culture Lab, Newcastle
University and uses modified IKEA units and standard
laminate flooring installed within a wooden structure (see top
figures). Significant care was taken that the underlying
technology is not apparent to the people using the kitchen –
even the wall projection is achieved using “up-lighter” style
projection onto mirrors below the overhead cabinets.
Figure 2. Sample off-the-shelf and custom technologies
integrated in the Ambient Kitchen: (a) 4 DLP projectors for
situated displays; (b) 4 Micaz zigbee motes and sensor
boards for object motion sensing; (c) 200 x custom capacitive
sensors for floor pressure measurement; (d) 8 Feig long
range RFID readers (and sample tag).
Figure 3. Using an RFID tagged control object the state of all
the Ambient Kitchen sensors can be examined on the main
display: floor pressure map (left); accelerometers (center-
top); RFID (center-bottom); and video feeds (right).
4. CASE STUDIES
The Ambient Kitchen has been developed to support a range of
research activities around the problem of providing situated
support for people with dementia, and situated services associated
with food planning, preparation and cooking. As a high fidelity
prototyping environment it allows us to support these activities in
a number of different ways, as an experimental space for
designers, for explaining new technologies to users, for collecting
sensor data in benchmark development for activity recognition
algorithms, and for the evaluation of complete solutions in a
naturalistic setting.
Figure 4. A current design scenario in which the pages of a
cookbook have integrated RFID tags. The workbench can
detect the current page and adapts the ambient display
according the page’s contents.
4.1 A design tool for designers
Developing design ideas for pervasive computing applications
usually requires a significant effort on the part of designers to
imagine what interacting in a fully instrumented environment
might be like. In our kitchen scenario, there are no keyboards,
mice or conventional input devices, and the ability to physically
Figure 1. The Ambient Kitchen is a lab-based high fidelity
pervasive computing prototyping environment. The kitchen is
situated in the main research space in Culture Lab, Newcastle
University and uses modified IKEA units and standard
laminate flooring installed within a wooden structure (see top
figures). Significant care was taken that the underlying
technology is not apparent to the people using the kitchen –
even the wall projection is achieved using “up-lighter” style
projection onto mirrors below the overhead cabinets.
Figure 2. Sample off-the-shelf and custom technologies
integrated in the Ambient Kitchen: (a) 4 DLP projectors for
situated displays; (b) 4 Micaz zigbee motes and sensor
boards for object motion sensing; (c) 200 x custom capacitive
sensors for floor pressure measurement; (d) 8 Feig long
range RFID readers (and sample tag).
Figure 3. Using an RFID tagged control object the state of all
the Ambient Kitchen sensors can be examined on the main
display: floor pressure map (left); accelerometers (center-
top); RFID (center-bottom); and video feeds (right).
4. CASE STUDIES
The Ambient Kitchen has been developed to support a range of
research activities around the problem of providing situated
support for people with dementia, and situated services associated
with food planning, preparation and cooking. As a high fidelity
prototyping environment it allows us to support these activities in
a number of different ways, as an experimental space for
designers, for explaining new technologies to users, for collecting
sensor data in benchmark development for activity recognition
algorithms, and for the evaluation of complete solutions in a
naturalistic setting.
Figure 4. A current design scenario in which the pages of a
cookbook have integrated RFID tags. The workbench can
detect the current page and adapts the ambient display
according the page’s contents.
4.1 A design tool for designers
Developing design ideas for pervasive computing applications
usually requires a significant effort on the part of designers to
imagine what interacting in a fully instrumented environment
might be like. In our kitchen scenario, there are no keyboards,
mice or conventional input devices, and the ability to physically
explore different configurations of sensor-dependent display
behavior significantly helps in the exploration and crafting of
design ideas. Figure 4 shows a prototype under development in
which the ambient displays (on the wall above the main worktop)
respond to the turning of the pages of a specially designed
cookbook. The cookbook has an RFID tag embedded in each page
allowing the kitchen to detect the page that the book is currently
open at. Responding to this, the ambient displays present relevant
food information and even personal media related to the recipe
and past times in the cook’s life when the corresponding meal was
prepared.
4.2 A design tool for users
Another significant challenge in designing pervasive computing
applications is the involvement with users in the design process.
We have conducted a number participatory design exercises
involving older users and found that a significant barrier to
exploring design concepts is adequately explaining the scope of
the technologies involved. By demonstrating simple mappings
between sensors and display in demonstration applications in the
Ambient Kitchen we have found that we can greatly improve lay
users' understanding of the potential functionality. For example,
figure 5 shows one such commonly used application in which
sample recipes are projected in response to the ingredients placed
on the bench. Traffic light indicators on the display also show
which of the recipes ingredients are in the kitchen’s cupboards.
Though a simple demonstration of how sensor data can be
mapped to information sources (and then to information displays)
our experience is that such illustrations can help users think about
both more mundane and adventurous (and useful) applications of
such “invisible” technologies.
Figure 5. The Ambient Kitchen has been used to facilitate
your
different people. For an
discussions and focus groups on the topic of pervasive
computing as part of a wider participatory design process
looking at ICT and nutrition for older people.
4.3 An observatory to collect sensor data
Realizing situated services that are responsive to both
actions and intentions requires significant development of activity
recognition algorithms themselves. As such, multi-sensor
benchmarks of everyday activities are not widely available, and as
part of our own research, and to support the research of
collaborators and the wider research community, we are
developing a number of such benchmarks by capturing data for
multiple subjects and activities, and hand annotating these
datasets. Activities range from gaze data for head pose tracking
algorithm development, primarily using video streams alone (see
figure 6), to naturalistic data sets relating to multi-step food
preparation for which RFID, accelerometer, pressure and video
data is collected and hand annotated.
4.4 An evaluation test bed
Evaluation means different things to
engineer the question is "does it work?" That is, are the functional
requirements met, does it complete certain tests accurately and
without failing. For the human factors engineer the question is
"does it perform a useful function in the context that it is intended
to be used". The latter question can only really be answered by
installing the technology in a range of real contexts. In the case of
kitchen technologies, this means real lived-in homes. Constraints
such as household routines and the different uses different
members of a family use different rooms for at different times can
be critical in the success or failure of home technologies. These
constraints are only really apparent in the context of real home
use. Laboratory-based facilities such as the ambient kitchen thus
are of limited use in this respect. However it is possible to use
them to do more limited evaluations of functional requirements
that still have face validity.
Figure 6. Simultaneously captured data from the embedde
y evaluations of this kind but we are
taken. Thus the teabags might be in a container on work surface,
d
cameras allows us to develop a benchmark for attention
detection algorithms based on tracking head pose and position
from multiple viewpoints.
We yet have to complete an
planning to do so. For example, we will use actors trained using
video recordings of people with dementia carrying out simple
kitchen tasks such as making a hot drink. These recordings were
made of people doing these tasks, which were of their own
choosing, in their own kitchens. The intention is to make initial
tests of algorithms to detect when these people need prompting
because they have made an error or stopped in the middle of the
task. It would be disorienting and therefore unrealistic to bring
people with dementia into the lab but these existing videos can be
used to configure the Ambient Kitchen to match the kitchen in the
video and then to get an actor to perform the sequence of actions
Chi et al.
a kitchen providing real-
time feedback, tracking
the number of calories in
food ingredients.
Enabling
Smart Kit
Copyright is held by the author/owner(s).
CHI 2007, April 28–May 3, 2007, San Jose, California, US
ACM 978-1-59593-642-4/07/0004.
Pei-yu (Peggy) Chi
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
peggychi@csie.org
Jen-hao Chen
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
r95922023@ntu.edu.tw
Hao-hua Chu
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
hchu@csie.ntu.edu.tw
Bing-Yu Chen
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
robin@ntu.edu.tw
LCD
display
smart
cabinet
figure 1. Smart Kitchen promotes
healthy cooking awareness and
cooking interaction to the cook.
smart
stove
smart
counter
CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress
To simu
on a hu
the food
time. W
the coun
an over
observe
testing o
food ing
RFID tag
After th
element
rule eng
activitie
Food ing
This inv
from a s
placed o
holding
up of a
figure 5
nutrition
overall in
For every container in the system,
the interface shows weight
information about food ingredients
the container has.
figure 4. Dialog window for asking
input the name of new food item in
the system.
CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress

15. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Arts“situated” cooking support
panavi system provides a new user
experience design by situated suggestions;
real-time sensing and giving feedback
the combination of the
pan and the interactive
recipe application
enables users to
challenge professional
culinary arts by
enjoying daily cooking
in domestic kitchens
NATURAL INPUTS
eyeCOOK is designed specifically to use natural input
modalities: those that humans use in human to human, non
mediated communication [5]. To reduce the need for users
to provide explicit input, or change their behavior to
accommodate interface constraints, implicitly provided
attentional cues are observed and interpreted. We believe
that this approach improves the learnability and
intuitiveness of interfaces designed for novice users.
Voice Commands
eyeCOOK uses context-sensitive, localized grammars. This
allows more synonyms for a given speech recognition
command, reducing the chance of misinterpreting a word.
Eye Gaze Commands
her/his task. To achieve this, we must
increase sensing capability [3], improve
coordination among appliances [5], and give
appliances the ability to affect the
environment [3,5].
Environmental Sensors
Temperature sensors used to keep track of the
status of the oven and the elements of the
stove can increase the system’s ability to
guide the user’s cooking experience and
could be synchronized with electronic timers.
Appliance Information Integration
Integrating knowledge of the environment
can result in improved functionality, taking
up less of the user’s time and effort. For
example, user recipe preferences, timing
constraints, as determined by the user’s
electronic schedule, and currently available
ingredients, communicated by food storage
areas, can be combined to suggest recipes.
As well, selecting a recipe can result in the
addition of necessary ingredients to an
electronic shopping list stored on the user’s
Personal Data Assistant (PDA).
Active Environmental Actions
The kitchen should not only be aware of its environment,
but it should also be able to affect it. Thus, it should be able
to take actions which increase efficiency, and reduce the
user’s action load, like automatically preheating an oven.
CONCLUSIONS
We have presented eyeCOOK, a gaze and speech enabled
multimodal Attentive User Interface. We have also
presented our vision of an Attentive Kitchen in which
appliances, informed by sensors, coordinate their behavior,
and have the capability to affect the environment. This can
reduce the user’s workload, and permit rationalizing
requests for user attention.
Figure 1. eyeCOOK in Page Display Mode
Interactive & Student Posters: Computers Everywhere CHI 2003: NEW HORIZONSPosters: Computers Everywhere CHI 2003: NEW HORIZONS
explore different configurations of sensor-dependent display
behavior significantly helps in the exploration and crafting of
design ideas. Figure 4 shows a prototype under development in
which the ambient displays (on the wall above the main worktop)
respond to the turning of the pages of a specially designed
cookbook. The cookbook has an RFID tag embedded in each page
allowing the kitchen to detect the page that the book is currently
open at. Responding to this, the ambient displays present relevant
food information and even personal media related to the recipe
and past times in the cook’s life when the corresponding meal was
prepared.
4.2 A design tool for users
Another significant challenge in designing pervasive computing
applications is the involvement with users in the design process.
We have conducted a number participatory design exercises
involving older users and found that a significant barrier to
exploring design concepts is adequately explaining the scope of
the technologies involved. By demonstrating simple mappings
between sensors and display in demonstration applications in the
Ambient Kitchen we have found that we can greatly improve lay
users' understanding of the potential functionality. For example,
figure 5 shows one such commonly used application in which
sample recipes are projected in response to the ingredients placed
on the bench. Traffic light indicators on the display also show
which of the recipes ingredients are in the kitchen’s cupboards.
Though a simple demonstration of how sensor data can be
mapped to information sources (and then to information displays)
our experience is that such illustrations can help users think about
both more mundane and adventurous (and useful) applications of
such “invisible” technologies.
Figure 5. The Ambient Kitchen has been used to facilitate
your
different people. For an
discussions and focus groups on the topic of pervasive
computing as part of a wider participatory design process
looking at ICT and nutrition for older people.
4.3 An observatory to collect sensor data
Realizing situated services that are responsive to both
actions and intentions requires significant development of activity
recognition algorithms themselves. As such, multi-sensor
benchmarks of everyday activities are not widely available, and as
part of our own research, and to support the research of
collaborators and the wider research community, we are
developing a number of such benchmarks by capturing data for
multiple subjects and activities, and hand annotating these
datasets. Activities range from gaze data for head pose tracking
algorithm development, primarily using video streams alone (see
figure 6), to naturalistic data sets relating to multi-step food
preparation for which RFID, accelerometer, pressure and video
data is collected and hand annotated.
4.4 An evaluation test bed
Evaluation means different things to
engineer the question is "does it work?" That is, are the functional
requirements met, does it complete certain tests accurately and
without failing. For the human factors engineer the question is
"does it perform a useful function in the context that it is intended
to be used". The latter question can only really be answered by
installing the technology in a range of real contexts. In the case of
kitchen technologies, this means real lived-in homes. Constraints
such as household routines and the different uses different
members of a family use different rooms for at different times can
be critical in the success or failure of home technologies. These
constraints are only really apparent in the context of real home
use. Laboratory-based facilities such as the ambient kitchen thus
are of limited use in this respect. However it is possible to use
them to do more limited evaluations of functional requirements
that still have face validity.
Figure 6. Simultaneously captured data from the embedde
y evaluations of this kind but we are
taken. Thus the teabags might be in a container on work surface,
d
cameras allows us to develop a benchmark for attention
detection algorithms based on tracking head pose and position
from multiple viewpoints.
We yet have to complete an
planning to do so. For example, we will use actors trained using
video recordings of people with dementia carrying out simple
kitchen tasks such as making a hot drink. These recordings were
made of people doing these tasks, which were of their own
choosing, in their own kitchens. The intention is to make initial
tests of algorithms to detect when these people need prompting
because they have made an error or stopped in the middle of the
task. It would be disorienting and therefore unrealistic to bring
people with dementia into the lab but these existing videos can be
used to configure the Ambient Kitchen to match the kitchen in the
video and then to get an actor to perform the sequence of actions
Figure 1. The Ambient Kitchen is a lab-based high fidelity
pervasive computing prototyping environment. The kitchen is
situated in the main research space in Culture Lab, Newcastle
University and uses modified IKEA units and standard
laminate flooring installed within a wooden structure (see top
figures). Significant care was taken that the underlying
technology is not apparent to the people using the kitchen –
even the wall projection is achieved using “up-lighter” style
projection onto mirrors below the overhead cabinets.
Figure 2. Sample off-the-shelf and custom technologies
integrated in the Ambient Kitchen: (a) 4 DLP projectors for
situated displays; (b) 4 Micaz zigbee motes and sensor
boards for object motion sensing; (c) 200 x custom capacitive
sensors for floor pressure measurement; (d) 8 Feig long
range RFID readers (and sample tag).
Figure 3. Using an RFID tagged control object the state of all
the Ambient Kitchen sensors can be examined on the main
display: floor pressure map (left); accelerometers (center-
top); RFID (center-bottom); and video feeds (right).
4. CASE STUDIES
The Ambient Kitchen has been developed to support a range of
research activities around the problem of providing situated
support for people with dementia, and situated services associated
with food planning, preparation and cooking. As a high fidelity
prototyping environment it allows us to support these activities in
a number of different ways, as an experimental space for
designers, for explaining new technologies to users, for collecting
sensor data in benchmark development for activity recognition
algorithms, and for the evaluation of complete solutions in a
naturalistic setting.
Figure 4. A current design scenario in which the pages of a
cookbook have integrated RFID tags. The workbench can
detect the current page and adapts the ambient display
according the page’s contents.
4.1 A design tool for designers
Developing design ideas for pervasive computing applications
usually requires a significant effort on the part of designers to
imagine what interacting in a fully instrumented environment
might be like. In our kitchen scenario, there are no keyboards,
mice or conventional input devices, and the ability to physically
Figure 1. The Ambient Kitchen is a lab-based high fidelity
pervasive computing prototyping environment. The kitchen is
situated in the main research space in Culture Lab, Newcastle
University and uses modified IKEA units and standard
laminate flooring installed within a wooden structure (see top
figures). Significant care was taken that the underlying
technology is not apparent to the people using the kitchen –
even the wall projection is achieved using “up-lighter” style
projection onto mirrors below the overhead cabinets.
Figure 2. Sample off-the-shelf and custom technologies
integrated in the Ambient Kitchen: (a) 4 DLP projectors for
situated displays; (b) 4 Micaz zigbee motes and sensor
boards for object motion sensing; (c) 200 x custom capacitive
sensors for floor pressure measurement; (d) 8 Feig long
range RFID readers (and sample tag).
Figure 3. Using an RFID tagged control object the state of all
the Ambient Kitchen sensors can be examined on the main
display: floor pressure map (left); accelerometers (center-
top); RFID (center-bottom); and video feeds (right).
4. CASE STUDIES
The Ambient Kitchen has been developed to support a range of
research activities around the problem of providing situated
support for people with dementia, and situated services associated
with food planning, preparation and cooking. As a high fidelity
prototyping environment it allows us to support these activities in
a number of different ways, as an experimental space for
designers, for explaining new technologies to users, for collecting
sensor data in benchmark development for activity recognition
algorithms, and for the evaluation of complete solutions in a
naturalistic setting.
Figure 4. A current design scenario in which the pages of a
cookbook have integrated RFID tags. The workbench can
detect the current page and adapts the ambient display
according the page’s contents.
4.1 A design tool for designers
Developing design ideas for pervasive computing applications
usually requires a significant effort on the part of designers to
imagine what interacting in a fully instrumented environment
might be like. In our kitchen scenario, there are no keyboards,
mice or conventional input devices, and the ability to physically
explore different configurations of sensor-dependent display
behavior significantly helps in the exploration and crafting of
design ideas. Figure 4 shows a prototype under development in
which the ambient displays (on the wall above the main worktop)
respond to the turning of the pages of a specially designed
cookbook. The cookbook has an RFID tag embedded in each page
allowing the kitchen to detect the page that the book is currently
open at. Responding to this, the ambient displays present relevant
food information and even personal media related to the recipe
and past times in the cook’s life when the corresponding meal was
prepared.
4.2 A design tool for users
Another significant challenge in designing pervasive computing
applications is the involvement with users in the design process.
We have conducted a number participatory design exercises
involving older users and found that a significant barrier to
exploring design concepts is adequately explaining the scope of
the technologies involved. By demonstrating simple mappings
between sensors and display in demonstration applications in the
Ambient Kitchen we have found that we can greatly improve lay
users' understanding of the potential functionality. For example,
figure 5 shows one such commonly used application in which
sample recipes are projected in response to the ingredients placed
on the bench. Traffic light indicators on the display also show
which of the recipes ingredients are in the kitchen’s cupboards.
Though a simple demonstration of how sensor data can be
mapped to information sources (and then to information displays)
our experience is that such illustrations can help users think about
both more mundane and adventurous (and useful) applications of
such “invisible” technologies.
Figure 5. The Ambient Kitchen has been used to facilitate
your
different people. For an
discussions and focus groups on the topic of pervasive
computing as part of a wider participatory design process
looking at ICT and nutrition for older people.
4.3 An observatory to collect sensor data
Realizing situated services that are responsive to both
actions and intentions requires significant development of activity
recognition algorithms themselves. As such, multi-sensor
benchmarks of everyday activities are not widely available, and as
part of our own research, and to support the research of
collaborators and the wider research community, we are
developing a number of such benchmarks by capturing data for
multiple subjects and activities, and hand annotating these
datasets. Activities range from gaze data for head pose tracking
algorithm development, primarily using video streams alone (see
figure 6), to naturalistic data sets relating to multi-step food
preparation for which RFID, accelerometer, pressure and video
data is collected and hand annotated.
4.4 An evaluation test bed
Evaluation means different things to
engineer the question is "does it work?" That is, are the functional
requirements met, does it complete certain tests accurately and
without failing. For the human factors engineer the question is
"does it perform a useful function in the context that it is intended
to be used". The latter question can only really be answered by
installing the technology in a range of real contexts. In the case of
kitchen technologies, this means real lived-in homes. Constraints
such as household routines and the different uses different
members of a family use different rooms for at different times can
be critical in the success or failure of home technologies. These
constraints are only really apparent in the context of real home
use. Laboratory-based facilities such as the ambient kitchen thus
are of limited use in this respect. However it is possible to use
them to do more limited evaluations of functional requirements
that still have face validity.
Figure 6. Simultaneously captured data from the embedde
y evaluations of this kind but we are
taken. Thus the teabags might be in a container on work surface,
d
cameras allows us to develop a benchmark for attention
detection algorithms based on tracking head pose and position
from multiple viewpoints.
We yet have to complete an
planning to do so. For example, we will use actors trained using
video recordings of people with dementia carrying out simple
kitchen tasks such as making a hot drink. These recordings were
made of people doing these tasks, which were of their own
choosing, in their own kitchens. The intention is to make initial
tests of algorithms to detect when these people need prompting
because they have made an error or stopped in the middle of the
task. It would be disorienting and therefore unrealistic to bring
people with dementia into the lab but these existing videos can be
used to configure the Ambient Kitchen to match the kitchen in the
video and then to get an actor to perform the sequence of actions
Enabling N
Smart Kitch
Copyright is held by the author/owner(s).
CHI 2007, April 28–May 3, 2007, San Jose, California, USA.
ACM 978-1-59593-642-4/07/0004.
Pei-yu (Peggy) Chi
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
peggychi@csie.org
Jen-hao Chen
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
r95922023@ntu.edu.tw
Hao-hua Chu
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
hchu@csie.ntu.edu.tw
Bing-Yu Chen
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
robin@ntu.edu.tw
LCD
display
smart
cabinet
figure 1. Smart Kitchen promotes
healthy cooking awareness and
cooking interaction to the cook.
smart
stove
smart
counter
CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress
To simula
on a hum
the food
time. Wh
the count
an overhe
observer
testing ot
food ingr
RFID tags
After the
elements
rule engin
activities
Food ingr
This invo
from a st
placed on
holding th
up of a se
figure 4. Dialog window for asking
input the name of new food item in
the system.
CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress
interactive cookbook
situated coaching
system
kitchen providing the
number of calories
previous “situated” cooking support systems

16. design process

17. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary ArtsCarbonara: initial menu
We chose a recipe of Italian pasta Roman-
styled Carbonara as an initial menu.
this recipe requires several ways of cooking including
sensitive temperature control, which is difficult to master.
We quoted a
recipe of Tsutomu
Ochiai a famous
Japanese chef of
Italian cuisines
who sometimes
appears in TV.

18. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsiterative design process
sketching
experiment: cooking
sketching
preliminary user study
prototyping
user study

19. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsiterative design process

20. Tinkering: Sketching a circuit sensing temperature

21. Tinkering: attaching a circuit embedded
with functions of wireless communication and
temperature sensing, to an iron made fry-ing pan

22. Tinkering: attaching a circuit embedded with functions of wireless
communication and temperature sensing, to an iron made fry-ing pan

23. Tinkering: A thermocouple sensor attached
with the pan using Aluminum tape

24. Tinkering: Cooking with monitoring the temperature

25. Tinkering: Cooking with monitoring the temperature

26. Tinkering: Cooking with monitoring the temperature

27. Tinkering: Trials and errors where to attach the sensor
（But it burns toride TOXI）

28. Tinkering: Repeating a cycle of making a prototype and
cooking with it, once a week iteratively during 90days.

29. Tinkering: testing designs of user interface,
while hardware sketching

30. Tinkering: Repeating a cycle of making a prototype and
cooking with it, once a week iteratively during 90days.

31. Tinkering: Repeating a cycle of making a prototype and
cooking with it, once a week iteratively during 90days.

32. Tinkering: Sketching a case for a circuit
using the laser cutter after the board design is fixed

33. Tinkering: Sketching a case for a circuit
using the laser cutter after the board design is fixed

34. Tinkering: Considering a system of projecting the temperature
to the fry-ing pan for vision problems

35. Tinkering: Considering a system of projecting the temperature
to the fry-ing pan for vision problems

36. User Study: Actually having 2 persons use the prototype

37. User Study: Actually having 2 persons use the prototype

38. User Study: Understanding the need of navigation

39. User Study: Need to understand the skill and cognition
difference between individuals

40. Prototyping: Preparing to create Firm Prototype

41. Prototyping: Using Aluminum Molding

42. Prototyping: Using Aluminum Molding

43. Prototyping: Casting a mold to seal sensor

44. Prototyping: Creating Circuit using Manufacturer

45. Prototyping: Creating Circuit using Manufacturer

46. Prototyping: Cutting Fry-ing pan Handle using MODELA

47. Prototyping: Complete Prototype
(when separated into pieces)

48. Prototyping: Completed Prototype

49. Exhibition: JST/CREST Ubiquitous Contents Showcase

50. Exhibition: JST/CREST Ubiquitous Contents Showcase

51. Exhibition: JST/CREST Ubiquitous Contents Showcase

52. design

53. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsarchitecture overview
the touch monitor and the projection system, and the software
system showing cooking sequence information (Figure 6).
panavi
display
Computer
(panavi OS + Original
Cooking Sequence)
Electronic
Circuit
(MOXA-B)
Electronic
Circuit
(MOXA-A)
Actuators
Xbee Wireless
Communication
Sensors
Thermocouple Sensor
Acceleration Sensor
Vibration Motor
LEDs
USB Cable
(Serial Communication)VGA
Cable
Mirror
Projector
Speaker
Projection
Touch
Monitor
Special Pan
Figure 6. System Architecture

54. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsdisplay
Figure 5. Three Kinds of Setting Displayed in the Monitor; (a) Summary, (b) Detail, (c) Condition / Name of each Area [See, (b)]; (b-1) Main Panel,
(b-2) Current Step Panel and Checkbox, (b-3) Comment Panel, (b-4) Temperature Panel, (b-5) Timer Panel
elapsed time, and the timer of boiling the pasta, which is also
a supplemental function to Condition mode.
IMPLEMENTATION & ARCHITECTURE
panavi system mainly consists of three parts; the special
the projector. Actuators, LEDs and vibration motors are
embedded in the handle, controlled by the signals from the
computer system via MOXA-B.
Display and Computer
There are three kinds of setting
displayed in the monitor
(1) Main Panel
(2) Current Step Panel and Checkbox
(3) Comment Panel
(4) Temperature Panel
(5) Timer Panel
Summary Detail Condition
Users can use any setting anytime.

55. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsoriginal cooking sequence
Step 1. Pasta (1) - Boil Water
Step 2. Making Egg Sauce
Step 3. Pasta (2) - Start Cooking Pasta
Step 4. Pasta (3) - Start Timer
Step 5. Pancetta (1) - Fry Pancetta [A]
Step 6. Pancetta (2) Add Wine
Step 7. Pancetta (3) - Seasoning
Step 8. Pancetta (4) - Cooling [B]
Step 9. Pasta (4) - Drain Water
Step 10. Finish (1) - Add Pasta to Pan
Step 11. Finish (2) - Dress with Egg Sauce
Step 12. Finish (3) - Heat Egg Sauce [C]
Step 13. Serve
The original cooking sequence
models the recipe of Carbonara.
Each step is programmed with
settings; temperature, sounds,
and vibrations etc.
LED lamps indicate temperature conditions and a vibration
motor moves when the users’ temperature control is good,
which are embedded within the handle of the pan. The
instructional graphics by the projection can also be checked
on the display monitor to support the recognition of the
projection when the pan is removed from the stove or when
the projection cannot be recognized since there is much
ingredient on the pan. If part of the system fails, other parts
could still help the user (i.e. if the projector fails, the display
and the LED indication work). See, Table 1. The temperature
color changes from white to blue, green, yellow, and red,
depending on the value, which also synchronizes with the
LEDs. White is the default color or indicates when the
condition temperature is too low, blue indicates 5-10 degrees
lower than the proper temperature setting, green indicates
when it is at the proper temperature within 5 degrees, yellow
indicates 5-10 degrees higher, and red indicates over 15
- 0 54 - 64 65 - 74 75 - 84 85 -
ON—
the second
hand of a clock
ON
warning
tone
—
—
Color
TEMP
（°C）
Vibration
Sound
- 69 70 - 79 80 - 89 90 - 99 100 -
B
- 149 150 - 159 160 - 169 170 - 179 180 -A
C
Blue Green Yellow RedWhite
ON
the second
hand of a clock
the second
hand of a clock
Table 1. Parameters to Cook the Carvonara Programed in the System.
Its procedure is divided into 13 steps.

56. user study

57. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsuser study
focused on not only general or common findings
but also each user’s originality; their respective
backgrounds, and experiences about cooking.
We observed how the system effects the
users experience while cooking.
User A User B User C & D

58. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsmethod
1. prior interview 2. cooking with panavi 3. posterior interview
The authors interviewed
each user about his/her
cooking experience.
Firstly, we introduced
how to use this system.
Next, we showed what
is the perfect Carbonara
by the photo.
During the cooking, we
did not help the users
except troubles (e.g.
system errors).
We interviewed the
users with watching a
video of his/her cooking.
They were required to
explain intentions and
impressions about each
action, activity, and
process.

59. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artssetting
selected, because experts or professional chefs are able to
cook without the system.
Utensils
rubber spatula, ladle,
tongs, tablespoon,
folk
Ingredients
olive oil, white wine,
pancetta, eggs, pasta,
salt, black pepper
Wet
washcloth
Wet
washcloth
Bowls
DisplayDisplay
PotPot PanPan
Table
Camera 1
Camera 2
User
Camera 3
Plate
Figure 7. Layout of the User Study
T
(F
e
e
is
q
n
h
r
o
c
th
M
T
fi
e
c
s

60. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Arts
User A
User A (Study 1)
UserA: 23 years old male
He has an experience of living
on his own for a brief time,
but lives with his parents now.
Although he has tried to do
basic cookings, he could not
make well.
As a result, he does not do
any cooking recently.
not enough
experience
in cooking
does not do
any cooking
recently

61. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Arts
User B
User B (Study 2)
User B: 24 years old female
She lives with her parents and
gives some help to their
cooking. They have interests in
food and cooking, and hold
their own home garden.
Raised in such a family
environment, she has basic
knowledge of cooking, but has
not had a chance to challenge
authentic menus.
has basic
knowledge
of cooking
does simple
cooking
once or twice
a week.

62. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary ArtsUser C&D (Study 3)
User C: 24 years old male
User C lives with his parents and
brothers.He has various cooking
experiences; as a Boy Scout member
and in part-time jobs at a restaurant.
He does cooking (mainly breakfast) 3
or 4 times a week.
User D: 22 years old female
User D lives on her own for 5 years
and has a habit of cooking.
She mastered the recipe book for
beginners her mother gave her, and
see web sites to expand her cooking
repertoire.
basic
knowledge
of cooking
User D
UserC
various
knowledge
of cooking
cooking
3-4 times
a week.
prefers
to expand
her cooking
repertoire

63. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary ArtsResult (Study 1: User A)
He sometimes could not understand
texts of instructions.
As a result, he spent a lot of time to
finish cooking. His pasta was too
boiled and soft...
At the final stage, he could not stop heating even after
the sauce was thick and creamy. His sauce was
slightly baked compared to the perfect example.
“I could not image the completion, because I had
never eaten this menu.”
Even he faced many difficulties, finally got a fine dish!

64. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary ArtsResult (Study 2: User B)
She enjoyed the process like playing
a game, and completed a delicious
dish except it was slightly scorched.
“I am very satisfied to make a
delicious dish. I have felt the process
was easier than I had imagined.”
She easily finished the first 5 steps, just following the
instructions without any difficulties.
After that, she precisely checked instruction texts and
carefully processed the further steps.
“At first, I could not understand the meaning of the
task on this step. It was difficult so I tried carefully,
different from the previous steps.” - Step 6
She enjoyed cooking with panavi. “What an easy cooking!”

65. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary ArtsResult (Study 3: User C&D)
User D mainly cooked with reading
the recipe texts many times.
While, User C frequently checked the
proper & current temperature and
supported her activities.
They did not check recipe texts or videos so much,
since they relied on their experience. But, in fact, they
said “We’d forgotten about watching the video.”
Therefore, their cooking was very quickly finished, but
their Carbonara was slightly sloppy (not heated well).
Both User B&C were disappointed with the result, and
said “I want to cook it again using the system.”
They quickly finished cooking on their collaborative works.

66. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary ArtsReflections
This study revealed the difference between
the understandings of instructions for each.
User A User B User C & D
finished in 32min. finished in 18min. finished in 15.5min.
The users had distinctive prior
knowledge and skills about cooking.
- normal cooking time: 20 min.-

67. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary ArtsReflections
This system could take care
each user’s situated actions,
even for the beginner user: User A.
User A sometimes
could not understand
texts of instructions.
He could keep the proper
temperature and did not
failed at the last stage.
He did not know
how to prepare
pasta...

68. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary ArtsReflections
But, at some points, the system could not
respond to the users’ situated actions,
because the instructions by texts and vide tutorials
are fixed and not changeable.
User C&D sometimes
skipped instructions.
User A: “I could not judge between the important
steps and the omissible ones, because it was too
much information for me to understand and follow.”

69. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary ArtsReflections
The current system cannot navigate
the timing of judging
when the heating should be stopped.
All users could keep good temperature,
but nobody made the perfect one.
One of User A was
slightly baked.
One of User B was
was slightly scorched
One of User C&D was
was slightly sloppy

70. conclusion

71. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artssmart daily commodities
This research provided a model for designing
daily commodities (e.g. kitchen utensils)
as smart objects enriching our everyday life.
Figure 5. Three Kinds of Setting Displayed in the Monitor; (a) Summary, (b) Detail, (c) Condition / Name of each Area [See, (b)]; (b-1) Main Panel,
(b-2) Current Step Panel and Checkbox, (b-3) Comment Panel, (b-4) Temperature Panel, (b-5) Timer Panel
elapsed time, and the timer of boiling the pasta, which is also
a supplemental function to Condition mode.
IMPLEMENTATION & ARCHITECTURE
panavi system mainly consists of three parts; the special
kitchen utensil (frying-pan) with embedded sensors and
actuators, the display system connecting the computer with
the touch monitor and the projection system, and the software
system showing cooking sequence information (Figure 6).
panavi
display
Computer
(panavi OS + Original
Cooking Sequence)
Electronic
Circuit
(MOXA-B)
Electronic
Circuit
(MOXA-A)
Actuators
Xbee Wireless
Communication
Sensors
Thermocouple Sensor
Acceleration Sensor
Vibration Motor
LEDs
USB Cable
(Serial Communication)VGA
Cable
Mirror
Projector
Speaker
Projection
Touch
Monitor
Special Pan
Figure 6. System Architecture
the projector. Actuators, LEDs and vibration motors are
embedded in the handle, controlled by the signals from the
computer system via MOXA-B.
Display and Computer
The computer is connected with ‘panavi display’ packaging
touch panel monitor and projector, and ‘panavi OS’ with
the ‘Original Cooking Sequence’ works as an Adobe Flash
application on the computer (as shown in Figure 6). The
panavi OS displays the instructions by analyzing the sensors’
degrees against parameters programed in the system.
Original Cooking Sequence
The original cooking sequence models the recipe of
Carbonara, consisting of videos and photos in addition to
the general text recipes. For the panavi OS, this cooking
sequence was reconstructed by the development team and its
procedure is divided into 13 steps (as shown in Table 2). Each
step is programmed with settings; temperature, sounds, and
vibrations settings etc. (as shown in Table 1).
Preparing Pasta (Table 2, Step 1-4) When the checkbox
of the Step 1 is touched, the elapse time counter from the
beginning (See, Figure 5-5) starts. The normal cooking time
is set to 20 minutes. When Step 4 is checked, the pasta timer
technical framework methodology
this framework senses and analyze users’
state, and generating feedback suitable for
each user’s situations.
this integrated technical elements
should be designed through
the prototyping process
in real domestic environments.
×

72. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artssmart daily commodities
The design of panavi
enables the users to
concentrate in cooking
without noticing if it is a
computing device.
It seems the same as normal
consumer products.
This design model is needless of
any massive equipment to
construct the system.

73. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsfuture visions
We would like to measure and model chefs’
cooking ways and apply these data to
navigation, and produce various kinds of
menu working on our future prototype.

74. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsfuture visions
We will proceed to develop & improve this
system to be more suitable for real domestic
contexts, which encourage people’s daily
cooking to become more enjoyable.

75. “Interactivity”
Please come to our demo booth!
We will introduce panavi
by real cooking show.
MON 18:00-20:00
TUE 16:00-19:00
WED 13:00-14:30
Today’s Reception
Highlight on Interactivity
Interactivity Encore
Maybe and so on...

76. thanks!

77. http://panavi.jp
panavi@ok.kmd.keio.ac.jp