A one-hour lecture I prepared for fulfilment of a MSc-level Advanced Human Computer Interaction class

1. Embedded & Tangible Interaction David Shaw “We live in a complex world, filled with myriad objects, tools, toys, and people. Our lives are spent in diverse interaction with this environment. Yet, for the most part, our computing takes place sitting in front of, and staring at, a single glowing screen attached to an array of buttons and a mouse.” Wellner, Mackay and Gold (1993)

2. Introduction Talk is about Embedded & Tangible Interaction What it is Related fields History The necessities to facilitate the technology Problems and challenges

3. About Tangible and Embedded interfaces allow us to move beyond being limited to mouse and keyboard input to interact with a computer We are quickly moving towards a post-WIMP revolution Novel interaction devices are becoming commonplace Simultaneously, computers are increasing being embedded in everyday objects and environments

4. What is Embedded Technology? “Embedded means enclosed; these chips and software are not considered computers. They are unseen parts of everyday things.” Malcolm McCullough, Digital Ground, 2004

5. What is Tangible Interaction? Tangible Interaction encompasses user interfaces and interaction that emphasize Tangibility and materiality of the interface Physical embodiment of data Whole-body interaction The embedding of the interface and the users’ interaction in real spaces and contexts. Eva Hornecker

6. What is Tangible Interaction? Tangible computing covers Distributing computation over many specialised and networked devices in the environment Augmenting the everyday world computationally so that it is able to react to the user Interaction by manipulating physical objects

7. What is Tangible Interaction? Tangible computing shares these characteristics No single focus or interaction No enforced sequentially or modal interaction Interface objects make intentional use of affordances

8. What is Tangible Interaction? Classifications of Tangible User Interfaces (TUIs) Interactive Surfaces Tangible objects can be placed onto a surface and interpreted by the system Constructive Assembly Modular and connectable elements attached to each other Token & Constraint Token represents an item, can be moved Constraints provide structure to limit positioning and give tactile guidance

9. Related fields Tangible and Embedded Interaction Design is an interdisciplinary field that draws influence from: Ubiquitous Computing The Internet of Things Industrial Design Actuation and Sensor based technology Robotics and Mechanics

10. Technology That Disappears “We have been very good at putting computers into the environment, but we have been very bad at getting them out of the way.” “The most profound technologies are those that disappear” “They weave themselves into the fabric of everyday life until they are indistinguishable from it” Weiser (1991) The computer will “take on the appearance of the task; it can disappear behind a facade.” Norman (1990)

11. 3rd Phase of Computing Mainframe > PC > Ubiquitous Lähdemäki (2007)

12. History of the Technology Emerged alongside Ubiquitous Computing as a research field philosophically opposed to Virtual Reality (VR) Approach to “retain the richness and situatedness of physical interaction” whilst simultaneously “embedding computing in existing environments” “Humans are of and in the everyday world” Shaer and Hornecker (2009)

13. Example Projects

14. Example projects Marble Answering Machine (Bishop, 1992) Phonecall represented by coloured marbles Drop marble to play message or call back Graphic from Shaer and Hornecker (2009)

15. Example projects Graspable User Interface (Fitzmaurice, Ishii, Buxton, 1995) Uses wooden blocks as handles to manipulate digital objects, early form of multi-touch Blocks are placed on monitors

16. Example projects Tangible Bits (Ishii and Ulmer, 1997) The entire world as an interface Connect data between physical artifacts and surfaces Move from ‘graspable’ to ‘tangible’ Three key concepts; Interactive surfaces Coupling of bits with graspable physical objects Ambient media for background awareness Ishii identifies the abacus as the ultimate tangible interaction metaphor

17. Example projects LiveWire (Jeremijenko) Piece of string dangling from the ceiling Visualisation of network traffic Pioneer project was influence for ambient display

18. Example projects Intelligent Physical Modeling Systems (Frazer) Intelligent cubes that know the position of its surrounding neighbours Siftables (Merrill & Kalanithi) 1.5” cubes that sense motion & each other http://www.youtube.com/watch?v=ZgF2rRzTg8Q

19. Example projects URP (Underkoffler and Ishii. 1999) A TUI for urban planning Combines physical models with interactive simulation Can project / model wind flow, sunlight simulation, building materials Graspable tokens Collaborative

20. Example projects Tern Tern is a tangible programming language for education Program actions for robots Uses interlocking wooden block which represent actions Shape of blocks creates a physical syntax

21. Example projects reacTable Tangible music interface Each token has has a function Dynamically attract using proximity “The foremost goal was to design an attractive, intuitive and non-intimidating musical instrument for multi-user electronic music performance.” http://www.youtube.com/watch?v=Ni_x_74VKU0

22. Surface Technology Microsoft Surface (2007) Multi-touch is arguably the most commercially successful application of horizontal surfaces Implicit capability of table interfaces is to support physical items on them The Surface adds digital information to everyday physical objects, allowing digital entities to coexist as fully digital non-physical form and as shared digital-physical form

23. Arduino Open source physical computing platform Simple I/O board that can be used as a stand-alone device or connecting to software on a computer Add-on modules, shields, that provide additional functionality

24. Benefits of Tangible UI Tangible User Interfaces (TUI) have many benefits Facilitating the kinds of collaborative activities that are not possible or poorly supported by single user technologies Appropriate for those who have lost their sight or have difficulty with motor control Andrew Cyrus Smith, Interactions 09/10 – 2010 Enhance learning - physical learning environments engage all sense and thereby support the child development. Lego Mindstorms and Topobo Support ambient awareness Can use tags to trigger digital information

25. Necessities to facilitate this technology McCullough (2004) suggests 10 essential building blocks to computing beyond the desktop

26. 1. Sites and devices are embedded with microprocessors “Less than a quarter of the chips produced by Intel, the largest manufacturer, are put into desktop or laptop computer motherboards” “The rest are embedded into things you carry about, drive, or wear; or are embedded into physical locations.” More than 95% of devices containing microchips do not present themselves to their users as computers.

27. 1. Sites and devices are embedded with microprocessors Practical economies of engineering do not always warrant providing a full service network operating system; devices can communicate at lower levels without that kind overhead. With connectivity, embedded systems can communicate their status and receive ongoing instruction to and from their surroundings.

28. 2. Sensors detect action “If technologies are to keep out of the way, they need to see us coming.” “If computationally embedded environments are to be useful yet unobtrusive, they have to recognise what is happening in them.” Examples of sensors Accelerometer Tilt sensor Pressure sensor Light sensor Microphone

29. 2. Sensors detect action Sensors have become the ‘key enabling technology’ for computing A sensor responds to a change in state Continuous sensor field Wirelessly interlinked sensors Passing or ‘hopping’ message directly amongst themselves Compare to a typical setup – LAN -> Dedicated Network -> Hardwired

30. 3. Communication links form ad hoc networks of devices Pervasive computing depends on unplanned communication Not all linked objects will benefit from a full-featured web browser. More will run slimmer set of communications Decentralised networking

31. 4. Tags identify actors Contextual awareness begins from an ability to recognise who or what is present Recognition is easy with the use of tags Smart badges RFID Tags Proximity detection Passive, Active, or Battery Assisted Passive (BAP)

32. 5. Actuators close the loop A device to automatically control a system via motion Open / close windows & doors, turn lights on / off, produce sound, motion or haptic feedback Bridges and dams can detect and identify deterioration, and signal for upkeep before failure occurs

33. 6. Controls make it participatory Smart systems need to be operable where it is appropriate This means providing an override facility

34. An example of a smart system Outdoor wind sensor detects wind speed and direction Indoor temperature sensor monitors room temperature Building Management System calculates that it can save energy by shutting off air conditioning system and opening windows Actuators physically open windows to allow air to flow into the building Staff may override system and close windows or turn AC system on, if they wish

35. 7. Display spreads out Before Gutenberg, text was reproduced using woodblock printing technique The Gutenberg press revolutionised the type industry and his printing methods spread rapidly across the world Today, the world thinks nothing of text. It is practically everywhere we look. On every conceivable surface Embedded interaction will do the same for computing

36. 8. Fixed locations track mobile positions “Let’s put GPS in necklaces and dog collars. Everything that moves should have GPS.” KanwarChadha, CEO at SiRF “This kind of stuff has enormous potential for abuse by the authorities, or by anyone who can break into the information.” Emily Whitfield, spokesperson for the American Civil Liberties Union Practical applications – GPS, Google Maps, Augmented Reality, Social Networks

37. Proximity “When you walk up to your computer, does the screensaver stop and the working windows reveal themselves?” Bill Buxton Important for context-aware properties of embedded interaction

38. Proximity There are four proxemic zones (Hall, 1966) Intimate (< 1.5 feet) Personal (1.5 to 4 feet) Social (4 to 12 feet) Public (12 to 25 feet) Each have expectations of engagement and behaviour

39. 9. Software models situations System may begin to model a physically proximate area by polling local ad hoc links between known tags and devices As hardware becomes less expensive, more diverse, and more plentiful, software becomes more challenging “Who is here, and what are they doing?”

40. 10. Tuning overcomes rigidity Much of the place-centred character of situated interaction design comes from the fact that any fixed collection of devices has to be integrated Question arise pertaining to protocols, distributed object programming systems The challenge of embedded interaction design is how can we make these interactions meaningful.

42. Security

43. What if it breaks?

44. Social acceptance

46. How Important is Privacy? Sorry, Slide removed for privacy issues Ironic, I know…

47. How Important is Privacy? Sorry, Slide removed for privacy issues Ironic, I know…

48. Security Privacy and Security are two different concepts Implementation of security does not ensure privacy Data collection and processing are core components of ubiquitous computing, and therefore embedded interactions

49. Privacy & Security Scenario Intelligent fridge scenario Knows what products you regularly buy and sources offers and coupons Re-orders food when your stock levels are low What if it gets hacked? ‘Hacker’can capture usage data Can infer information

50. What if it breaks? What if

51. What if it breaks? Critical systems Health systems Flight systems

54. References Buxton, W. (1997) Living in augmented reality: Ubiquitous media and reactive environments. Video Mediated Communication. K. Finn, A. Sellen, and S. Wilber (eds.). Erlbaum, Hillsdale, N.J, 1997. Greenfield, A. (2006). Everywhere: The Dawning Age of Ubiquitous Computing, New Riders, Berkeley, CA, USA Hall, E.T. (1966) The Hidden Dimension. Doubleday, New York,1966. Ishii, H. & Ullmer, B. (1997). Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms. Proc. CHI 1997, ACM Press (1997), p. 234-241. McCullough (2004). Digital Ground, MIT Press, London, England Norman, D. (1990). The Design of Everyday Things. Doubleday/Currency, New York. Shaer, O., and Hornecker, E., (2009) “Tangible User Interfaces: Past, Present, and Future Directions” Foundations and Trends in Human-Computer Interaction, Vol. 3 Nos 1-2 Underkoffler, J. and Ishii, H. (1999) , “Urp: A luminous-tangible workbench for urban planning and design,” in Proceedings of CHI ’99, pp. 386–393, NY: ACM,1999. Vogel, D. and Balakrishnan, R. (2004) Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. Proc. of the 17th Annual ACM Symposium on User Interface Software and Technology. (Santa Fe, NM, Oct. 24-27). ACM, New York, 2004,137-146. Weiser, M. (1991). The Computer for the 21st Century. Scientific American. Sept, 94-104. Wellner, P., Mackay, W., and Gold, R., (1993) “Computer-augmented environments.Back to the real world,” Communications of the ACM, vol. 36, no. 7, pp. 24–26,1993.