Slides used in the Digital Arts Atelier class of the Sound and Image course (School of Arts, Porto). Oct 3, 2012

1. Interaction modalities,
technologies, and tools for
interactive art
Jorge C. S. Cardoso
http://jorgecardoso.eu

2. Objectives
• Overview of most common non-desktop
interaction and computing paradigms
• Examples of technologies and tools that
can be used to implement those
interactions
o Available tools at EArtes/UCP
• Most prominent people/references in
these areas
• No code (for now)

3. Topics
Gestural interaction Tangible interaction Brain interaction
Wearable computing Physical computing Digital public displays Mobile computing

4. Gestural interaction
Photo credits: Flickr user artmakesmesmile: http://www.flickr.com/photos/artmakesmesmile/169193698/

5. Gestural interaction
"A gesture is a motion of the body that contains
information. Waving goodbye is a gesture.
Pressing a key on a keyboard is not a
gesture because the motion of a finger on its
way to hitting a key is neither observed nor
significant. All that matters is which key was
pressed".
-- Kurtenbach and Hulteen (1990), cited by Billinghurst &
Buxton (2011)

6. Gestural interaction
Technologies/Tools
• Touch surfaces
(ordered by the complexity of gestures: simplest to more complex)
• Computer vision techniques
• Wiimote
• Kinect
• MoCap

7. Gestural interaction: Touch surfaces
• Mostly planar, rectangular, glass(y) surfaces
• Various technologies for detecting touch
o which have different limits for the number of simultaneous
touches it can detect
• Gestures
o Fingers only (small devices)
o Hands/forearms (large(r) devices, depending on technology
used)
o 2d movements only
• Number of users
o one - personal devices (phones, tablets)
o few - large surfaces (tables, walls)

8. Gestural interaction: Touch surfaces
Tablets/phones
• Singing fingers
o Jay Silver
o https://vimeo.com/12010952
Single-person interaction,
fingers

9. Gestural interaction: Touch surfaces
Medium/Large surfaces
• Touch-Wall @ EArtes
o Laser light plane (LLP) technique
(see Sandler 2012)
o Multi-touch surface (in
development)
o Vertical orientation
o One-two person interaction
o Fingers, hands

10. Gestural interaction: Touch surfaces
Medium/Large surfaces
• Bookstore @ EArtes

11. Gestural interaction: Touch surfaces
Medium/Large surfaces
• Interactive tables (Reactables)
o Diffused illumination technique
o Horizontal orientation
o Up to four people interacting
o Fingers, hands shapes
o Three available @ EArtes

12. Gestural interaction: Touch surfaces
Very large surfaces
• Massive Curved Multi-Touch Screen
o Reality touchscreen University of Groningen
o http://www.youtube.com/watch?v=AlWFtF06RFo
Several-person interaction,
fingers, hands

13. Gestural interaction: computer vision
Computer vision (CV) techniques (non depth-
cameras)
• Various kinds of surfaces/scenes
• Gestures
o Varied, depends on positioning of camera
§ fingers
§ hands
§ whole body
o mostly 2d, but can also infer distance
o not very precise
§ Can be very precise with controlled conditions

14. Gestural interaction: computer vision
Video Place
• Myron Krueger
• http://www.youtube.com/watch?v=dmmxVA5xhuo

15. Gestural interaction: computer vision
Edge/Swap
• Tiago Dionisio, Rudolfo Quintas
• http://www.youtube.com/watch?v=OSKpXZjECtY
• http://www.youtube.com/watch?v=ryX4Vq2-IKE

16. Gestural interaction: computer vision
BES Interactive Video Wall
• YDreams
• http://www.youtube.com/watch?v=ijrZQbSOONQ

17. Gestural interaction: computer vision
Balance
• Jorge Coutinho
• https://vimeo.com/40506946

18. Gestural interaction: computer vision
Sketchsynth
• Billy Keyes
• http://vimeo.com/42053193
• http://golancourses.net/2012spring/05/13/billy-keyes-
final-project-sketchsynth/

19. Gestural interaction: computer vision
All eyes on you
• Britzpetermann
• https://vimeo.com/33186969

20. Gestural interaction: computer vision
Knee deep
• Emily Gobeille & Theodore Watson
• https://vimeo.com/8805152

21. Gestural interaction: computer vision
Night lights
• YesYesNo
• http://vimeo.com/8525186

22. Gestural interaction: computer vision
aB
• Jorge Coutinho
• http://vimeo.com/29524291

23. Gestural interaction: Wiimote
Wiimote
• Nintendo game controller
o Wireless: uses bluetooth to connect to a
computer
o Has various accelerometer sensors to
sense movement
o IR sensor for pointing
• Gestures
o 3d movement, rotation
o combined with the sensor bar, allows
pointing gestures
• Various wiimotes available @ EArtes

24. Gestural interaction: Wiimote
Wiimote
• Wiimote hacks
o Johnny Chung Lee
o http://www.ted.com/talks/
johnny_lee_demos_wii_remote_hacks.html

25. Gestural interaction: Wiimote
Pizza breakout
• Bruno Santos

26. Gestural interaction: Wiimote
Wii Drum High Tutorial
• He Zhao
• http://vimeo.com/2406224

27. Gestural interaction: Kinect
Kinect
• XBox game controller
o Depth camera: senses the distance of
each pixel to the camera (Borenstein
2012).
• Gestures
o Detects a "point cloud" - pixels and their
distances to the camera
o With extra software it's possible to
detect limbs (skeleton) positioning
o More accurate than simple cameras
• Two available @ EArtes

28. Gestural interaction: Kinect
Kinect Physics Tutorial for
Processing
• Video showing the
result of Processing
code that uses Kinect
• http://
www.creativeapplications.net/
processing/kinect-physics-
tutorial-for-processing/

29. Gestural interaction: Kinect
YScope medical interface
• YDreams
• Gestural interface for an operating room
• http://www.youtube.com/watch?v=91F6zErnCrs

30. Gestural interaction: Kinect
Control the Humanoid Robot by Kinect
• Using the skeleton detected by the
Kinect to drive a robot
• ikaziso
• http://www.youtube.com/watch?v=w8BmgtMKFbY

31. Gestural interaction: Kinect
Unnamed soundsculpture
• Daniel Franke & Cedric Kiefer
• Using various Kinect to model a dancer and then
process the data into visual effects
• https://vimeo.com/38505448
• honorary mention from the Prix Ars Electronica

32. Gestural interaction: Kinect
Make the line dance
• "1024"
• Detection of skeleton + projection into human
body
• https://vimeo.com/21308228

33. Gestural interaction: Kinect
Fragmentos de um olhar
• Pedro Serrano

34. Gestural interaction: LeapMotion
Leap Motion
• New controller for hand interactions
• https://leapmotion.com/

35. Gestural interaction: MoCap
MoCap - Motion Capture
• MoCap Room
• High-speed infrared cameras detect markers
placed on actor's body
• Gestures
o Highly accurate gestures
• (Vicon MoCap available @ EArtes)

36. Gestural interaction: MoCap
T(ether)
• Tangible Media Group
• Interaction with virtual volumetric
data
• http://vimeo.com/42173010

37. Gestural interaction: MoCap
Portrait of the ghost drummer
• Odaibe
• Visualization of the process of
playing on a drum kit
• http://vimeo.com/34682556
• Non-interactive

38. Gestural interaction: computer vision
Computer Vision Tools
• Processing
o BlobDetection
o JMyron
• Max/Msp
o Jitter
o jit.cv external
• Community Core Vision
o http://ccv.nuigroup.com/

39. Gestural interaction: Wiimote
Wiimote Tools
• Processing/Java
o Motej - http://motej.sourceforge.net/
o Wigee - http://www.wiigee.org/
• Max/Msp
o aka.wiiremote external -
http://www.iamas.ac.jp/~aka/max/

40. Gestural interaction: Kinect
Kinect Tools
• http://openkinect.org/wiki/Main_Page
• Book "Making things see" (Borenstein 2012)
• Processing/Java
o OpenKinect - http://www.shiffman.net/p5/kinect/
o SimpleOpenNI -
http://code.google.com/p/simple-openni/
o http://www.creativeapplications.net/processing/
kinect-physics-tutorial-for-processing/
• Max/Msp
o jit.freenect.grab - http://jmpelletier.com/freenect/

41. Gestural interaction: Touch surfaces
Tools
• Web apps
o jQMultiTouch library
§ http://dev.globis.ethz.ch/jqmultitouch/

42. Gestural interaction
References
• Billinghurst & Buxton (2011). Gesture Based Interaction, http://www.billbuxton.com/
input14.Gesture.pdf
• Borenstein, Greg (2012). Making Things See: 3D vision with Kinect, Processing, Arduino, and
MakerBot, O'Reilly Media / Make
• Buxton, Bill (2012). Multi-Touch Systems that I Have Known and Loved, http://
www.billbuxton.com/multitouchOverview.html
• Kurtenbach, G. & Hulteen, E. (1990). Gestures in Human-Computer Communications. In B.
Laurel (Ed.) The Art of Human Computer Interface Design. Addison-Wesley, 309-317, http://
www.billbuxton.com/inputManuscript.html
• Levin, Golan. Computer Vision for Artists and Designers: Pedagogic Tools and Techniques for
Novice Programmers, http://www.flong.com/texts/essays/essay_cvad/
• Sandler, Seth (2012). LASER LIGHT PLANE MULTITOUCH TECHNIQUE, http://
sethsandler.com/multitouch/llp/

43. Tangible interaction
Photo credits: Flickr user kedume: http://www.flickr.com/photos/kedume/746632510/sizes/l/in/photostream/

44. Tangible interaction
"[...] tangible interfaces give physical form to digital
information, employing physical artifacts both as
representations and controls for computational media. TUIs
couple physical representations (e.g., spatially manipulable
physical objects) with digital representations (e.g., graphics
and audio), yielding user interfaces that are computationally
mediated, but generally not identifiable as "computers" per
se." - Ullmer & Ishii (2000)

45. Tangible interaction
Picture from Ullmer and Ishii (2000).

46. Tangible interaction
Tangible Media Group site
• http://tangible.media.mit.edu/projects/

47. Tangible interaction
Reactable
• Tangible musical instrument
o Users manipulate physical objects to create and control the sound
o Can be used as a general tangible framework
• http://www.youtube.com/watch?v=Mgy1S8qymx0

48. Tangible interaction
Little Boxes
• ECAL/Joelle Aeschlimann
• Music boxes for iPad
• https://vimeo.com/45704273

49. Tangible interaction
Sketch-a-TUI
• Alexander Wiethoff, et al.
• Set of printable objects for tangible prototypes
• http://vimeo.com/38793875
• Paper
o http://tei-conf.org/12/Main/Sketchatui
• Printable objects
o http://project-premium.org/sketch-a-tui/index.html

50. Tangible interaction
Birds on Paper
• Chen-Wei Chiang, et al.
• Drawing music
• http://www.youtube.com/watch?v=qqkkNotSQMo
• Paper
o http://tei-conf.org/12/Main/Birdsonpaper

51. Tangible interaction
Chronotape
• Peter Bennett
• Tangible timeline with annotations
• https://vimeo.com/28191512
• Paper
o http://tei-conf.org/12/Main/Chronotape

52. Tangible interaction
Splash Controllers
• Luc Geurts and Vero Vanden Abeele
• Uncareful Manipulation of Water
• Paper
o http://tei-conf.org/12/Main/Splashcontrollers

53. Tangible interaction
Skube
• Andrew Nip, Ruben van de Vleuten, Malthe Borch, and
Andrew Spitz
• Tangible interface to Last.fm & Spotify Radio
• http://www.creativeapplications.net/maxmsp/skube-
tangible-interface-to-last-fm-spotify-radio/

54. Tangible interaction
Faustine
• Carlos Caires & Jorge Cardoso
• Interactive video installation inspired by the short story
"Morel's Invention"
• https://vimeo.com/19043202

55. Tangible interaction
The garden of time
• Carlos Caires & Jorge Cardoso
• Interactive video installation inspired by the short story
"The Garden of Forking Paths"
• https://vimeo.com/24836769

56. Tangible interaction
Tools
• Reactable
• TUIO - Fiducial tracking
o http://www.tuio.org/?software
o Processing
o Max/MSP
o CCV
• Wiimotes
• Other off-the-shelf hardware/sensors

57. Tangible interaction
References
• G. W. Fitzmaurice, H. Ishii, and W. A. S. Buxton, “Bricks: Laying the
Foundations for Graspable User Interfaces,” in Proceedings of the SIGCHI
conference on Human factors in computing systems - CHI ’95, 1995, pp.
442–449.
• H. Ishii and B. Ullmer, “Tangible bits,” in Proceedings of the SIGCHI
conference on Human factors in computing systems - CHI ’97, 1997, pp.
234–241.
• Ullmer, B. and Ishii, H. (2000). “Emerging frameworks for tangible user
interfaces,” IBM Systems Journal, vol. 39, no. 3, pp. 915–931, Jul. 2000.

58. Brain interfaces
Photo credits: Flickr user Saad Faruque: http://www.flickr.com/photos/cblue98/7254347346/sizes/l/in/photostream/

59. Brain interfaces
Brain-Computer Interfaces (BCI) or Mind-
Machine Interface (MMI)
Using brain signals to interact with computers.
• Usually based on EEG readings
• Various commercial devices available
• http://en.wikipedia.org/wiki/Brain
%E2%80%93computer_interface#Low-cost_BCI-
based_Interfaces

60. Brain interfaces
Câmara Neuronal
• Adolfo Luxúria Canibal, João Martinho
Moura, Miguel Pedro Guimarães
• http://projects.jmartinho.net/3486412/
Camara-Neuronal-Video-Teaser

61. Brain interfaces
Brainwave beats
• Collin Cunningham
• Midi notes from brain signals
• Uses the Neurosky Mindset device
• http://blog.makezine.com/2011/05/05/collins-lab-
brainwave-beats/

62. Brain interfaces
Mansion Maze 3 BCI Game Trailer
• Eric Rosendale, Ian McCabe, Veronica Cole, Aaron
DeChamplain, and Matt Stephan
• BCI Game
• http://www.youtube.com/watch?v=sdYAIrr9nDw

63. Brain interfaces
Mind Control of Lego NXT Telepresence Robot
• Mark Bruce
• Controling a lego robot using brain signals
• Uses EPOC BCI device
• https://www.youtube.com/watch?v=nX741DZw8l4

64. Brain interfaces
Tools
• Neurosky Mindset (available @ EArtes)
• EPOC neuroheadset - http://emotiv.com/emortal/cart/
• Processing library
o Neurosky Mindset Processing
§ http://jorgecardoso.eu/processing/
MindSetProcessing/
• http://en.wikipedia.org/wiki/Brain
%E2%80%93computer_interface#Low-cost_BCI-
based_Interfaces

65. Wearable computing
Geordi La Forge, Star Trek The Next Generation character wearing the VISOR

66. Wearable computing
"Wearable computing is the study or practice of
inventing, designing, building, or using
miniature body-borne computational and
sensory devices. Wearable computers may be
worn under, over, or in clothing, or may also
be themselves clothes" (Mann, 2012)

67. Wearable computing
Image from: http://www.eyetap.org/research/wearables/wearcomp/wearables.html
Image from: http://www.interaction-design.org/

68. Wearable computing
Google Project Glass
• https://plus.google.com/+projectglass/posts
• https://www.youtube.com/watch?
v=9c6W4CCU9M4
(how will our world be in 5/10 years, when we
all start wearing devices even more powerful
than Glass?)

69. Wearable computing
XS Labs
• http://xslabs.net/

70. Wearable computing
Blazer 01.2
• A light emitting bracelet
• Can be used to "draw" letters in the air
• Vincent leclerc
• http://www.uttermatter.com/blazer/012/index.htm#

71. Wearable computing
cidZiy
• Capacitive sensing textile surface
• xslabs
• http://www.youtube.com/watch?v=zH4J6Uaf88A

72. Wearable computing
Kukkia
• Kukkia flowers frame the face and slowly open and
close over time
• xslabs
• http://www.youtube.com/watch?v=R5fd6AwGMhg

73. Wearable computing
See U in my Tee
• Inês Petiz
• T-shirts that react to the presence of other t-shirts
• http://wearable-technology.blogspot.pt/

74. Wearable computing
References
• Mann, Steve (2012): Wearable Computing. In: Soegaard, Mads and Dam,
Rikke Friis (eds.). "Encyclopedia of Human-Computer Interaction". Aarhus,
Denmark: The Interaction Design Foundation. Available online at
http://www.interaction-design.org/encyclopedia/wearable_computing.html

75. Physical computing

76. Physical computing
"Physical computing, in the broadest sense, means building
interactive physical systems by the use of software and
hardware that can sense and respond to the analog world.
(...) In the broad sense, physical computing is a creative
framework for understanding human beings' relationship to
the digital world. In practical use, the term most often
describes handmade art, design or DIY hobby projects that
use sensors and microcontrollers to translate analog input
to a software system, and/or control electro-mechanical
devices such as motors, servos, lighting or other
hardware." - (Physical computing 2012)

77. Physical computing
Physical computing book
• O'Sullivan & Igoe 2004

78. Physical computing
reaDIYmate
• Olivier Mével and Marc Chareyron
• http://www.readiymate.com

79. Physical computing
An Instrument for the
Sonification of Everyday
Things
• Dennis P Paul
• http://
www.creativeapplications.net/
processing/an-instrument-for-the-
sonification-of-everyday-things/

80. Physical computing
Piano playing stairs
• http://www.youtube.com/watch?
v=DuNWkKBokts

81. Physical computing
What the phonics
• Andrew Spitz & Momo Miyazaki
• https://vimeo.com/45747333

82. Physical computing
Barcode band
• Kang woon jin, Lee ha lim, Kim yong duk
• https://vimeo.com/41132461

83. Physical computing
Syndyn
• André Rangel and Anne-Kathrin Siegel
• https://vimeo.com/20859799

84. Physical computing
Little Bits
• Hardware components that can be assembled together
• http://market.littlebits.cc/
• https://vimeo.com/45276780

85. Physical computing
MakeyMakey
• Circuit board that connects to a computer via USB and
allows everyday objects to become controllers
• http://www.makeymakey.com/

86. Physical computing
Lego Mindstorms NXT
• Available @ EArtes
• Lego kit with
o motors
o distance sensor
o light sensor
o button
o lego parts...

87. Physical computing
Tools
• Arduino - http://www.arduino.cc/
• Galago - http://outbreak.co/galago
• Digispark - http://digistump.com/
• Raspberry pi - http://www.raspberrypi.org/
• MakeyMakey -
http://www.makeymakey.com/
• LittleBits - http://littlebits.cc/
• Lego NXT Mindstorms (available @EArtes)
o http://mindstorms.lego.com/en-us/Default.aspx

88. Physical computing
References
• Physical computing. (2012, July 21). In Wikipedia, The
Free Encyclopedia. Retrieved 14:37, September 22,
2012, from
http://en.wikipedia.org/w/index.php?
title=Physical_computing&oldid=503476286
• Dan O'Sullivan, Tom Igoe. (2004). Physical Computing:
Sensing and Controlling the Physical World with
Computers, Thomson Course Technology, Boston,
Massachusetts,

89. Digital public displays
Photo credits: Flickr user bmann: http://www.flickr.com/photos/boris/5104900447/sizes/l/in/photostream/

90. Digital public displays
Digital displays located in public areas, allowing anyone to
see/interact with them
• Traditionally, public displays have been used to broad
information from one active source to millions of
passive receivers
• Digital public displays can change this
o Turning receivers into active receivers
o Supporting dynamic locally relevant information
o Supporting user-generated content (many to many
content distribution)
• Everyone can interact
o Everyone can see you interacting

91. Digital public displays
Advertising

92. Digital public displays
Honda Interactive Billboard using SMS and
Bluetooth
• Users can "start" the car via SMS
• http://www.youtube.com/watch?v=OudaJGtwdQ8

93. Digital public displays
BBC Billboard
• Voting billboards, users can SMS their vote
• http://www.bbc.co.uk/blogs/theeditors/2007/06/
live_in_new_york_city.html

94. Digital public displays
Audio Jack Billboards
• Promoting a TV show "Big Love"
• Users can listen to people's thoughts by plugin
headphones into the billboard
• http://www.trendhunter.com/trends/audio-jack-billboard-
for-big-love

95. Digital public displays
Nokia's N90 interactive billboard
• YDreams
• The billboard snaps people's pictures and displays them
on the billboard
• http://www.youtube.com/watch?v=UWgIswo3lV4

96. Digital public displays
Mini Billboard Interactive
• The billboard displays the names of the Mini car owners
passing by
• http://www.nytimes.com/2007/01/29/business/media/
29cooper.html

97. Digital public displays
Nikon D700
• The billboard plays flashing cameras as people pass by.
• http://www.thecoolhunter.net/article/detail/1570/nikon-
d700-guerrilla-style-billboard

98. Digital public displays
Ambient displays
• Information about their surroundings
• Information visualization, aesthetics, art

99. Digital public displays
Hello Wall
• (Prante et al. 2003)
• Can display various abstract information patterns
• Can display group activity

100. Digital public displays
Informative Art ambient display
• Skog et al. 2003
• Display bus information (only the bus number 16 - two in
each direction)
o Size of square: amount of time before bus leaves
o Color: Time to leave to the bus stop
o Position: right - buses to the city center; left - buses from
the city center

101. Digital public displays
Information Percolator
• Heiner et al. 1999
• A water bubble display
o Can be used to display various
types of ambient information
• http://www.cs.cmu.edu/~hudson/
video.html

102. Digital public displays
Entertainment/Waiting times

103. Digital public displays
Mobilenin
• Scheible, Jürgen and Ojala, Timo
• Paper
o http://www.mendeley.com/research/mobilenin-combining-multi-track-
music-video-personal-mobile-phones-public-display-multi-user-interac
o Allows users to vote on the next video clip

104. Digital public displays
Public Youtube Player
• Allows users to have some
control over the videos that are
played
• at the EArtes' Bar display

105. Digital public displays
Everybody Votes
• Users can express their opinions regarding various
questions and suggest their own questions
• at the EArtes' Bar display

106. Digital public displays
References
Heiner, J. M., Hudson, S. E., & Tanaka, K. (1999). The Information Percolator : Ambient Information
Display in a Decorative Object. Proceedings of the 12th annual ACM symposium on User
interface software and technology - UIST ’99 (pp. 141–148). New York, New York, USA: ACM
Press. doi:10.1145/320719.322595
Prante, T., Röcker, C., Streitz, N., Stenzel, R., Magerkurth, C., van Alphen, D., & Plewe, D. (2003).
Hello.Wall - Beyond Ambient Displays. Video Track and Adjunct Proceedings of the 5th Intern.
Conference on Ubiquitous Computing (UBICOMP’03). Seattle, Wash., USA.
Skog, T., Ljungblad, S., & Holmquist, L. E. (n.d.). Between aesthetics and utility: designing ambient
information visualizations. IEEE Symposium on Information Visualization 2003 (IEEE Cat. No.
03TH8714) (pp. 233–240). IEEE. doi:10.1109/INFVIS.2003.1249031

107. Mobile computing

108. Mobile computing
"Mobile computing is human–computer
interaction by which a computer is expected
to be transported during normal usage.
Mobile computing involves mobile
communication, mobile hardware, and mobile
software." - (Mobile computing 2012)

109. Mobile computing
X-Fetch
• Daniel Santos, João Rema
• City-wide game, peddy paper like
• Players have to discover a set of clues spread across
the city (Porto) using a mobile device that plays videos
pointing to next clues and that tracks players GPS
locations
• http://jpn.c2com.up.pt/2005/10/29/
xfetch_quando_o_real_e_o_virtual_se_confundem.html

110. Mobile computing
• GPS Drawing
Jeremy Wood
http://www.gpsdrawing.com/gallery.html

111. Mobile computing
Mobile mobile
• http://www.theophane.co.uk/mobile-mobile
• https://vimeo.com/8118881

112. Mobile computing
Pom
• Jorge Coutinho
• Jogo Pong estendido a seis jogadores
• https://vimeo.com/18579379

113. Mobile computing
References
• Mobile computing. (2012, September 8). In Wikipedia, The Free
Encyclopedia. Retrieved 17:56, September 30, 2012, from
http://en.wikipedia.org/w/index.php?
title=Mobile_computing&oldid=511364202