The team explored different directions for their camera project, focusing on domains like music, cooking, physical therapy, and storytelling. They decided to focus on using the camera's voice recognition and sound output capabilities to enhance storytelling. The camera would produce sounds related to actions or environments described in stories in response to trigger words, images, gestures, or sounds. At a subsequent make session, the team further refined their idea to focus on adding ambiance to an environment using sound and movement in response to music.

1. Story Time
Gene Dreyband | Dennis Ellis
Manali Gortekar | Jonathan Partlow

2. Project Overview
Our first path was to learn about
our technology with which we have
been given a task to design. We
first learned through the exemplars
mentioned in class during the
October 21st Make Session. We
then read, watched, and listened to
examples from the Internet to gain
more insights on how the camera
functioned. We also asked more
knowledge colleagues about certain
functions and then getting into
specifics of coding.
After getting a handle on the
specifics of the camera and trying to
focus on the unique capabilities, we
started to think of different design
domains. We first thought of music
and using sound as starting off
point. It was a struggle to learn how
to problem frame and problem set,
but not problem solve. We thought
it would be best to keep music
or sound as part of our concept
but to move on to new areas for
exploration.
We considered domains like the
kitchen and cooking as well as
physical therapy. Here we focused
on discrete actions and how the
camera could be utilized to improve
the experience. This was a step
closer, but we were still trying to
problem solve rather than explore
a new area. With the knowledge
we had gained over the past weeks
about the camera itself as well as
different domains, we decided to
choose some of the capabilities of
the camera to explore and then
shape a context around the design.
Voice recognition and sound output
were selected as a focus of our
design. We decided to reintroduce
an earlier domain, that of turning a
child’s room into a jungle, or some
other environment. Immediately
the idea of storytelling came up
in a conversation and we began to
iterate on the idea of trigger words
and images, sounds, and gestures
the could enhance the storytelling
experience. We also incorporated
gestures into the storytelling.
When teachers, parents, or other
children tell stories, they act out
the adventures of the story. We
utilized the gesture imputs of the
camera to produce sounds that
reflected actions or environments in
the story.

3. Make Session 10.28
Our concept from the original Intel make session needed reworking. We were able to accomplish a
proof of concept, but we wanted to
change our direction. Jeff Bardzell
gave us some advice about trying to
truly capture the unique capabilities
of this technology with our design.
We started to consider other areas
for which we could design. It was
a lesson to learn how to navigate
the line between problem framing,
problem setting, and problem solving. For this project we needed to
avoid problem solving, concentrating more on the other two aspects
of design.
We explored different areas for a
setting. We proposed working with
sound and music, facial recognition
for the seeing-impaired, and physical rehabilitation. Physical rehabilitation sounded the most intriguing
space at the time, so we started
iterating on what type of feedback
could be utilized by the camera. We
(A)
(B)
learned that we were too focused on
a solution, and not focused enough
on exploring the capabilities of the
camera to the greatest extent.
(C)
A - Team members discuss directions reflecting ideas generated
during the make session.
B - Other teams critique and consider our directions and give
feedback as how to continue.
C - Other teams critique and consider our directions and give
feedback as how to continue.

4. Affinity Diagram Insights
Several areas arose from this that
intrigued us most.
Empowerment and removing
barriers came up regularly from the
discussions that we had had about
rehabilitation, in the sense that
we wondered how we could use
the camera to augment someone’s
senses. Could the functions of the
camera be helpful for those with
mobility issues, visual or hearing
impairment? Could it be useful for
the colorblind?
Creating wonder was a space we
thought was also exciting, and we
came up with ideas such as turning
a child’s bedroom into a fantasy
world, using the camera to perform
magic tricks, or just using the
device for undirected play. Rather
than trying to solve a problem, play
could be used as an interesting way
to explore what the device could do,
rather than creating a solution and
expecting the camera to conform.
-Empowerment/Removing Barriers
-Ease of Access to Information
-Sensory
-Creating Wonder
-Connecting People
-Bridging Digital/Physical
-Distributed Cognition
-Sensing Emotion

5. Why can’t I just...
-Grab something from
one screen and put it
onto another
them in existing art
programs instead of
using a tablet or a stylus
-Tell my computer to
turn off the lights
-Detect user facial
expressions and cheer
them up if they’re sad
-Skip through images
and presentations using
a finger swipe gesture in
the air
-Use simple thumbs up/
down gestures to vote
for songs on Pandora/
Spotify
-Use voice commands to
look for a specific file
-Delete things with
a unique gesture like
squishing my hands
together
-Record and “play back”
gestures that I just
performed
-Make drawable shapes
with my hands and use
-Tell the computer to
start recording video
and audio on command
and do simple edits on
the recorded footage
-Push and pull objects
in real space and reflect
their real depth on
camera
-Look out the window
and use my computer
-Workout while using an
interface
-Draw on my computer
like I can in my
sketchbook.
-Use public media
players rather than
having to buy my own.
-Control the lights in my
house from anywhere
-Do more in my car than
I currently can
-Change the pitch of my
drums digitally without
losing any of the
acoustic properties
-Listen to music in my
own space without
earphones
-More passively digest
news
Conduct a digital
orchestra
-Be more grateful
-Make people think
differently
-Let people find the
“new” by exploration
-See the world from a
different perspective
-Help people understand
that we are all part of
one system
-Motivate others to take
action rather than just
talk about it
-Remove barriers to
access

6. Why can’t I just...
-Re-ignite / ignite a new
spark
-Make people thirsty for
knowledge
-Give people a tool to
impact lives
-Create new awareness
-feel data
-get sensory feedback in
the form of touch
-smell data
-get sensory feedback in
the form of taste
-not have people getting
stuck in their selfish little bubble of augmented
reality
-come across meaningful
information that makes
sense
-invent something that
helps me document my
thinking process
-physically move things
as I am thinking
-have a system that
could tell me where I
could have been productive
-travel places around
the world virtually if I
can’t go there physically
-edits on the recorded
footage

7. Wouldn’t it be cool if I could...
-Do magic tricks without
any props
-Make hand shadow
puppets that “talk” to
each other
-Put together blocks and
record the instructions
-Recreate the lightning
scene with Elmer Fudd
from the “What’s Opera
Doc” cartoon
-Have a series of movable blocks that correspond to things in the
real world like lights,
TV, or appliances
-Change the camera
color scheme with my
fingers (b/w, color,
depth, etc)
-Play catch with a
digital ball with my
computer
-Just utter the title of a
movie and it shows up
on your TV screen.
-Connect multiple
cameras and take 3D
photos of objects and
convert them to models
-Never have to adjust
the thermostat again.
-Make a rainbow in
between my hands
-Detect different objects
like using a blue vs. a
green pen or detect all
of the different sides of
an object
-Video chat with
someone at the front
door and unlock the
door remotely so they
can come in if you’re
not able to do so (like
working in the back
yard)
-Never have to use a
cookbook again, just
had the recipes in your
kitchen
hallway a different way
each time
-Have friends
experience your travels
in more immersive
way than through
photographs
-Talk to anyone,
regardless of language
-Read a book on any
surface
-Smile at a blind person
and have them know it
-Have more private
interactions in a public
space
-Have more public
interactions I a private
space
-I could if the ability
to give was always
available
-Turn your child’s room
into a jungle
-Walk down the same
-I knew someone was
hurting inside and could
reach out to them
-I could pay it forward
from anywhere at
anytime

8. Wouldn’t it be cool if I could...
-I could practice random
acts of kindness in the
world from my living
room
and into the world
facts
-consume data in literal
sense
-I could be in touch but
out of reach
-give me new/ fresh
data that smells nice
and raw/outdated data
that smells like garbage
-get a readymade document/video/image for
reflection
-I had time to do what I
wanted to do as well as
needed to do
-I could rearrange my
house without actually
touching anything
-I could hug my kids
while I am away
-give me feedback by
hugging/ shaking hands/
patting my back
-touch it and play with it
as I am getting information
-I could design in 3d
without manipulating
a ton of buttons and
controls
-socialize with people in
real
balance physical and
mental activity together
while working
-I could get these
thoughts out of my head
-get information I need
instead of displaying
-simulate what would be
like if I had been to places like Paris, Slovenia
-make sure if I am living
a good life.

9. Brainstorming body mechanics
-Eating a chicken wing
-Picking up a box
-Rolling a barrel
-Putting a beverage up to
your mouth
-Smoking a rolled cigarette
-Eating a brownie
-Shopping for twinkies
bending and putting in cart
and then removing and
eating...
-Stretching your arms
-Praying
-Lifting weights
-Tuning the strings on a
guitar
-Sanding a piece of wood
-Stroking a piece of wood
-Displaying your o-face
-Turning off an alarm clock
-Looking at your phone/
watch
-Holding onto the headboard
-Playing a drumset
-Putting on a new drumhead
-Eating corn on the cob
-Putting a vinyl record on
the turntable
-Rolling a cigarette
-Getting freaky
-Hugging
-Playing Guitar
-Paddling (Both versions - in
water and out of… ;-)
-Opening a bottle of wine
-Washing dishes
-Moving Furniture
-Serving in Tennis
-SkyDiving
-Swimming - Freestyle,
-Backstroke
-Replacing a flat tire
-Eating a chicken wrap
-Pushing the walk
-Crying
-Rolling out in a laughter
-Rolling tongue upside down
-Striking a match
-Sharpening a pencil
-Rolling eyes
-Driving a stick shift
-Nodding
-Typing on keyboard
(two handed and one
handedBoxing)
-Carving a pumpkin
-High-fiving a bro
-Fixing hair while walking
-Sneezing
-Crowd surfing
-Skipping
-Blowing up a balloon
-Tying a balloon
-Fire by friction
-Applying ointment
-Getting a shot
-Taking a medication
-Suppository
-Bungee Jumping
Final List
1. Praying
2. Sanding a piece
of wood
3.Putting on a new
drumhead
4.Hugging
5. Opening a bottle
of wine
6. Washing dishes
7. Replacing a flat
tire
8.Fire by friction
9. Getting a shot
10. Putting a
vinyl record on the
turntable
11. Crowd surfing

10. Body-storming activities
We came up with a large number of
ideas for bodystorming activities, from
the smallest activities like picking up
a box, to extremely complex activities
like starting a fire. The activities that
we chose to actually body storm once we
were finished coming up ideas were the
following:
Praying
Sanding a piece of wood
Putting on a new drumhead
Hugging
Opening a bottle of wine
Washing dishes
Replacing a flat tire
Fire by friction
Getting a shot
Praying - This was an interesting
activity, because while praying often
consists of slow, repetitive gestures
which the camera is good at identifying,
there were so many different types of
stances and positions and activities that
even something so “simple” had a lot of
very complex gestures embedded in them.
Sanding a piece of wood
- While sanding a piece of wood has
relatively few activities like the sanding
process itself and examining the wood,
the camera was something that could
be useful to identify the smoothness/
roundedness of the wood, however it
would require the camera to be positioned
in such a way to pick it up. Certain angles
would make this difficult.
Putting on a new drumhead
- The activity of tuning a drumhead was
helpful because it contained several
things the camera could identify: key
locations to hit the drum using the camera
and using the microphone to pick up
timbre/intonation.
Hugging - While hugging is a common
activity, body language is a very subtle
but key consideration that brings up a lot
of questions. Is the hug between friends/
family/strangers? What aspects make a
hug appropriate or uncomfortable (length
of time, familiarity with the person,
appropriateness of the context)? Was it
more joking (like in our own activities),
or is the hug in earnest? What can we do
with this kind of information?
Opening a bottle of wine
- Opening a bottle of wine is tricky since
different openers have different optimal
positions. Some are so unwieldy that they
need to be held between the legs, others
it’s a matter of knowing how far to twist
the screw, and what about removing the
cork on champagne and worrying about
hitting someone?
Washing dishes - Washing dishes
is a repetitive activity, but it’s only so
similar in the abstract sense. Small
dishes require different motions to wash
than pots and pans, or cups or utensils.
Things with lots of food stuck on them
require vigorous scrubbing that maybe
a plate with crumbs would not. Even
the difference between whether dishes
would be washed and stuck on a drying
rack versus washed to be placed in a
dishwasher significantly changed the
specific context.

11. Body-storming activities
Replacing a flat tire - With its
many steps and trouble spots, it would
be easy to envision ways that technology
could improve the tire changing process.
From finding the right spot to place the
jack, the properly tightening the lug nuts.
We found that each activity in the process
was relatively simple, but the emotional
state of having a flat tire is probably
worse than the reality so it was definitely
an interesting activity to bodystorm.
Fire by friction - Starting a fire
was such a complex process to explain
and navigate that just the process alone
took something like 5-10 minutes to
even effectively pantomime, let alone
the description of it to people that hadn’t
done it before in the group. It was fun
and involved, but ultimately way too
sophisticated for the camera.
Getting a shot - One would think
getting a shot would be simple, but
there is the preparation of the syringe,
preparing the spot in the arm, giving the
shot and finally treating the area after the
shot. Identifying a key area here might be
how to help find a vein more effectively,
but that is just one particular area.
The key takeaway from the bodystorming
activity is just how involved almost any
activity we engage in truly is. Those
minor adjustments we do that are
contextual are so subject to change, that
it becomes difficult to make assumptions
about what the person will do. With
current technology it is difficult to create
something that accounts for those use
cases without forcing the person using
them to conform to some pattern, rather
than having the device adapt to their
existing patterns.

12. Make Session 11.11
During this Make session, we began to focus on a particular space.
We needed to scale down our ideas
so that our design had a core idea
from which we could build our new
functionality for the camera. From
the six design directions we found
for the Make session we dissected
and compared the different characteristics of the designs. The idea of
adding ambiance to an environment
stuck out to us.
We began with the idea of working
the the realm of music reappropriation. We explored use of sound,
movement, and dance. Yet, we
weren’t able to identify how we
would use the unique capabilties of
the camers so we decided to explore
other arenas. We started talking
about the space of the kitchen and
the many differen
(A)
(B)

13. Make Session 11.18
During this make session, we began
to explore the different capabilities
of the camera. We still needed to
find a way to problem set, and we
thought that focusing on the unique
characteristics of the camera was
the way to do it. The past 3 weeks
of research gave us knowledge that
we want to use, primarily that we
wanted to use sound in our design.
We were intrigued by the use of
voice recognition and music output,
as well as possible facial recognition.

14. 21 Balançoires
Exemplar description:
The musical swing interactive installation called ‘21 Balançoires,’ consists
of a number of swings that both lit up and play a tune.Each swing plays a
different pre-recorded sound, some being piano notes and sounds from a
xylophone. When all 21 swings are in use and even greater song is created
encouraging cooperation among people. This installation was put in
Montreal, Quebec in the Quartier des Spectacles.
Design Direction:
“Adult Playground Installation” - An installation of playful objects
like swings, spun chair , see-saws, etc. for adults that will lit up and
play a tune. Adults will be given a task of playing together a melody or
completing a picture together by playing with installation objects. Like
the exemplar, all the objects in installation will be associated with a
particular sound note and a light color in order to incorporate audio and
visual feedback. The installation intends to offer a fresh look on the idea
of cooperation - the notion that together we can achieve more than alone
and it is specifically directed to adults to encourage playful learning among
them.
Inspiration:
The idea of “playfulness” and “ co-operation” are
inspirational elements of the exemplar implied
to the “Adult playground installation” design.
Playing is a psychological necessity and so it is
not restricted only to children. We wanted to give
an experience to adults where they could re-live
their childhood by playing objects like swings and
spun chair in public installations. We intend the
individuals to un-mask their identity and escape
from reality for a while in the world of playfulness
and fun.
Context of Use:
We see this design being used as public installation
like Amusement park in cities. Adults could
purchase tickets and enter the installation to
join the fun activity. Through visual and audio
feedback we expect adults becoming more
engaged with the exhibit objects and also realizing
the notion of “oneness” among people.
The Novel Experience:
One of the important feature of this design is that
it involves an entire body experience. All part of
body are engaged while playing spun chairs . Also
everyone’s minds get in sync to create a melody
or picture. Thus, we intend to provide the “Whole
body experience” in this design that mainly
includes audio and visual feedback. Through
perceptual computing technology, the playful
experience could be enhanced and made more
seamless.

15. Talk to Dead Artists
Exemplar Description :
This interactive art exhibit allows visitors to have interactive
chats with the works of art. Its is a novel way to give
information about a classic painting to visitors where the
painting could talk.For example a visitor can ask Mona Lisa why
doesn’t she has eyebrows and have a conversation with her or
listen to Jesus and his disciples in The Last Supper.
Design Direction:
“Talking objects” - The everyday object like cell phones, books
and computer talking for themselves. The basic idea is that of
an ensoulment of a non-animated object. To de-familiarize a
commonly used object and make people “think” about them.
This critical design aims to encourage people to imagine a life
without these objects and appreciate their presence. It could
facilitate a conversation between an object and its owner so
that the owner knows more about it.
Inspiration:
The notion of “de-familiarization” and
“collaborative learning” are inspirational elements
of the exemplar implied to the “Talking objects”
design. In the exemplar, people have conversation
with an exhibit object and know more about its
rich history. Through perceptual computing, we
intend to bring similar learning experience by
letting people know more about their everyday
objects. The conversational way of learning
can be very effective as it gives people right
information at the time as and when needed.
The everyday object could talk about its history
and designer and make people more aware of
its rich background. One of our goals is to give
information to the people in a way that makes
sense to them and the “conversational” element of
the exemplar seems to be a perfect design fit.
Context of Use:
The “Talking object” is more of a critical design.
Its only purpose is to make people think and
appreciate the presence of everyday objects.
Thus, we don’t have a specific usage of the design.
We wanted it to be open ended, exploratory and
playful to people.
The Novel Experience:
“Talking objects” - Design that incorporates the
notion of ensoulment as the objects speak for
themselves in their usual environment. It is a
critical design in daily lives of people. A normal
object like computer could just start a friendly
conversation with its user and ask him about his
day. It could introduce itself by telling about its
background in brief. The object could also respond
depending on how its user is handling it by
enhancing the tactual experience of the user.

16. Nervous Structure
Exemplar description:
Nervous structure is an exciting design because it uses the body
to create an experience that is many things: playful, delicate,
collaborative, and easy to understand. The wave-like motion
of the threads creates an easy sense of motion that translates
nicely into full body motions.
Design Direction:
“full body motion” - Using the full body as an interface as a
way of getting people to be active. The goal of such an exhibit
is to create an easy to understand interface that invites play
and interaction. Creating an experience that is inviting and
soothing. Semi-directed activity works well when cameras are
not totally precise, as fluid-like motion can mask some of the
imperfections of a time delay as well as issues with detection.
Inspiration:
In the artists own words, it is a synthesis between
the physical, digital, and viewer’s perception.
The art resembles physical fabric (through using
a moire pattern of the projection on the fixed
grid), but the perception is that of a real flowing
material. By creating a relatively simple design,
people are able to understand almost immediately
in an embodied way how to use the design, even if
they don’t understand exactly how it works.
Context of use:
Using our camera, people can interact with the
threads in 3 dimensions, pushing them out and
side to side. The soothing motion of the design
is something that could be seen in a home as a
way to encourage someone to get out of their
seat for a few minutes just to move around, even
aimlessly. In close quarters, it could also be used
like a conventional blind to change colors or adjust
lights in a room.
The Novel Experience:
The novelty lies in the simplicity of the actions.
While the “wow” factor of this type of design may
be lower than others, it makes up for it in being an
interaction that can remain fun and novel for long
periods of time. Being able to express yourself
freely in an undirected way could lead to some
interesting interactions and gestures that people
do regularly that make sense as a standard for
perceptual computing.

17. Concept Map

18. Final Concept
Design Rationale
Our design concept is based on
creating a space for possibility.
The overall concept is to aid in
augmenting, expanding the sensory
input / output and increasing
engagement using the specificity of
the Intel Technology.
For the proof of concept of this
application of the design we
placed it in a classroom setting
with a storyteller / teacher as this
happens millions of times a day in
classrooms all over the world.
Specificity of the device
Voice and Language Recognition
We utilized the ability of the voice
and word recognition of the device
as a tool to aid the storyteller while
telling a simple childrens story.
Typically, this would be done by
holding a book and reading to the
children, flipping the book around
or passing the book around to
show the images to the children
to assist in meaning making and
understanding the relationships of
Size, Color, Location, Proximity,
etc.
By utilizing the capabilities of the
device the speaker reads the story
and the images are immediately
triggered along with the additional
sound appropriate sound.
The rationale for this was the
increased engagement and
understanding (mapping) provided
by engaging additional senses.
We chose this structured
storytelling approach as a starting
point as it allows the audience to
‘discover’ what is happening on
their own creating an additional
level of learning :“the cause and
effect relationship”.
Although this highly valued
learning concept may be valuable
be incorporated into the story being
told it is not typically incorporated
into the traditional storytelling
process itself.
Using a “build upon prior learning”
system we introduced the gestures
and their functionality into the
storytelling process.
This introduction served multiple
purposes.
The audience had already figured
out that the images and sounds
were reacting to the audio
storytelling and recognized
the characters and concepts
individually. (ie: Blue, Dog, In)
At this point the children were
already starting to tell the story out
loud and anticipating what the next
slide would say. They were saying it
out loud before it appeared.
By the storyteller not speaking
the words aloud, but instead
advancing to the next logical step
of combining elements in the story
by gesture the audience was ‘invited

19. Final Concept
to participate’ and also quickly
associated and mapped the cause
and effect of the gesture: moving
the hand in for ‘in’ and out for ‘out.
The other purpose of using
this structure was introduce
another way of storytelling and
communication available - using
body language and gestures rather
than just words.
unique features and separation of
controls by input type of the Intel
camera. The storyteller in front of
the camera has the ultimate control
of the images, pace and additional
sounds but the audience, by virtue
of the word recognition, is able to
also have limited control simply by
speaking the words.
With the concepts of the design
introduced and discovered the
dynamic of the storytelling
experience itself changed from a
one way exchange, where the the
storyteller simply reads and the
audience listens; or not as typically
happens with young children in this
situation.
The experience has now become
a participatory session due the
audiences ability to advance the
story by using the words recognized
by the Intel SDK, patterns, process
and story logic introduced earlier.
This participation is enabled by the
We found that once the concepts
were introduced the audience
wanted to engage and play with
the system. Without technical
knowledge or special skills the
children can (and did) create their
own stories either by triggering
an image and / or sound and then
building off of what they saw /
heard, or by intentionally using
words that they knew in advance
would display / play what they
wanted.
Target Audience
Storytelling is a formative part of
the human experience, and one that
colors each and every transition
we go through. Particularly, we
focus on primary school children
aged 6-11. During this period,
storytelling permeates education.
Not only do children learn through
stories, they learn how to tell
stories and how to communicate
their emotions, thoughts, and
desires through storytelling. We
hope to enhance this experience and
thus inspire future generations to
think, dream, make, and believe.
Expected Outcomes
Bringing the story out of the book
is a primary goal for our design.
We hope to engender a spirit of
exploration and wonder with the
effects our design creates. What
will occur in the mind of a child
when she hears a lion roar when
she says ‘Lion!’? We expect for
children to use our design in an

20. Final Concept
improvisational way, crafting their
own stories and adventures as they
see fit.
assesment.
- Increased understanding and
engagement of the storyteller and
the audience.
- Additional sensory stimulation
providing expanded association of
concepts from different domains.
- Introduction to other
communication tools and methods
beyond voice.
- Convergence of meaning making
and making meaning.
- Changing the dynamic from
a one way communication to a
participatory event.
- Children using the design as a tool
to express and create their own
stories.
- The design speaks to the same
issues that we had to deal with
in this project: Understanding
cause and effect but we want also
want the participants to be able to
think of “the possibilities” without
Our design engenders imagination,
storytelling, adventuring, and
wonder. Of course, each of these
is notoriously difficult to evaluate.
After some time with our design,
we could examine how children take
to storytelling. Is their imagination
different? We could observe
how children create new stories
our observe how they physically
interact with the device. We
could record the number and time
value of each visit, and how that
changes over time. We could make
more subjection judgements about
emotions and such, but measuring
change is a primary objective.
Evaluation Approaches
- Direct Evaluation in Situ:
Observation of participation and
engagement in storytelling
- Direct Evaluation Individual:
Testing of progress in concept
understanding
- Indirect effects: Tracking
progress in areas outside of
storytelling sessions that involve
communication, creativity,
imagination, cause and effect and
logic.
- Direct Evaluation: Interviews with
participants and those within their
ecosystem - Parents, Teachers.
- Video and Photo Ethnographical
Studies
- Do the participants relate the
cause and effect and still create
their own original stories?

21. Final Concept
Video: