Brett Renfer, creative director at the LAB at Rockwell Group, gave a presentation at NYCCHI Presents: Connected Spaces - the Internet of Things held in September 2013 at Parsons School of Design. Brett is a creative director at the LAB at Rockwell Group, where he leads interaction design efforts across a wide range of projects. Since joining the LAB in 2008, Brett has worked on a diverse range of projects that includes Plug-In-Play in San Jose, California, Build Up! in Washington, DC, and the Cosmopolitan Hotel West Lobby in Las Vegas, Nevada. Brett also manages and develops TSPS and other open-source LAB projects. ROCKWELL GROUP LAB: As an interactive design firm embedded within a larger architecture firm, the LAB seeks to blur the line between the physical and virtual in all of our projects. Our process relies heavily on iterative prototypes. Through making and experimenting we gain a deeper understanding of the user experience and uncover new opportunities and insights. In our presentation we will explore two projects that showcase different aspects of our work process. Plug-in-Play is a performative celebration of city life and activity, where both physical and virtual activity is filtered and translated into an abstracted cityscape that is projected on the façade of the San Jose City Hall building. The West Lobby of the Cosmopolitan is a site-specific, permanent new media platform that creates a larger-than-life inhabitable environment. See Eventbrite for more details: http://www.eventbrite.com/e/nycchi-presents-connected-spaces-the-internet-of-things-tickets-7898234833?aff=eorg

1. P R O T O T Y P I N G I N T E R A C T I V E S P A C E S
B R E T T R E N F E R

2. R O C K W E L L G R O U P

3. A C A D E M Y A W A R D S
2 0 0 9 & 2 0 1 0
M A Z E , L O N D O N
W P A R I S
I M A G I N A T I O N
P L A Y G R O U N D
architecture firm of 25+ years
* we like to tell stories in spaces, whether it be through art directing the academy awards,
working in hospitality and retail, or rethinking playgrounds from the ground up

4. T H E L A B A T R O C K W E L L G R O U P

5. W E ’ R E A M U L T I - D I S C I P L I N A R Y G R O U P O F
G R A P H I C D E S I G N E R S , S O F T W A R E D E V E L O P E R S ,
H A R D W A R E H A C K E R S , A R C H I T E C T S ,
A N D P R O D U C T D E S I G N E R S
as a part of the larger architectural firm, we focus on ways to integrate technology into designed spaces

6. W E T H I N K A B O U T C H O R E O G R A P H I N G E N V I R O N M E N T S
T H E I N T E R A C T I O N
O F
E N V I R O N M E N T S/ P E O P L E S T O R I E SO B J E C T S / /
B E C A U S E W E L O V E T H E P H Y S I C A L W O R L D A N D T H E
V I R T U A L , W E U S E T H E M B O T H I N O U R T O O L K I T F O R
C R E A T I N G S P A C E S T H A T T E L L S T O R I E S .
our sweet spot is the intersection of the physical and the digital

7. I N T E L C E S 2 0 1 3
we design and build temporary pieces, such as the Ultrabook Tree at Intel’s CES booth in 2013
* temporary spectacle, short term interaction

8. T H E C O S M O P O L I T A N O F L A S V E G A S
...and permanent pieces, such as the cosmopolitan west lobby
* intersection between art, architecture, and technology
* permanent == infinitely updateable, simple technology, graceful failure

9. W E ’ R E P R O P O N E N T S O F L E A R N I N G T H R O U G H M A K I N G
T H E B E S T W A Y T O D E S I G N U S I N G N E W T E C H N O L O G Y
I S T O A C T U A L L Y T R Y T O M A K E S O M E T H I N G W I T H I T

10. We like to test new materials and techniques through designed scenarios, some more fleshed out than others
* Trying out emerging tech: face tracking signage, distributed displays
* Testing new material techniques: material/projection studies, display + material studies
* We create prototypes to vet scenarios for ourselves
* those that are successful, we develop just to the point that others can try them

11. In addition to isolated prototypes, we like to attack specific problems
* Set of tests for a transparent LCD: simplest viable prototypes
* combination of custom interactives, existing sites, video prototypes
* as much for us as it is for our clients!
* Goal: know the “gotchas” of new tech so that we may better design around them
* want work to transcend the technology, so we need to know the limits and pain points

12. W E ’ R E P R O P O N E N T S O F L E A R N I N G T H R O U G H M A K I N G
W E B U I L D P R O T O T Y P E S T H A T T E S T + C H A L L E N G E
/ E X P E R I E N C E SA E S T H E T I C S /T E C H N I C A L C H A L L E N G E S
With real-world projects, we take the same approach: design, build, test, iterate!
* three main categories of prototypes, though they tend to combine and separate over the process
* more on what they mean shortly

13. P L U G - I N - P L A Y
S A N J O S E , 2 0 1 0
San Jose Biennale, 2010

14. W E W E R E A S K E D T O D E S I G N A P R O J E C T I O N F O R T H E
R I C H A R D M E I E R D E S I G N E D S A N J O S E C I T Y H A L L
T H I N G S G O T W I L D P R E T T Y Q U I C K L Y

15. A big reason I want to talk about this project is that it was very formative for our prototyping process
* This image, crazy as it is, was a great jumping off point:
* if we can do anything, and it has to be projection, what do we want to make?
* with such a blank canvas, where do we even begin?

16. P R O J E C T I O N I S G R E A T ,
B U T W H Y D O S O M E T H I N G H E R E I N T H E F I R S T P L A C E ?
we realized quickly we needed a stronger narrative to tie together the stories we wanted to tell:
* interaction in public places
* power of architectural scale interaction
* emerging engagement of technology in civil discourse

17. T H E R E ’ S A W E A L T H O F I N T E R A C T I O N S + E N G A G E M E N T
I N P U B L I C S P A C E S , A N D S A N J O S E W A S / I S N O D I F F E R E N T

18. there’s a rich history in concrete touch points to civic engagement, including
* protest / public speaking
* play
* community
* wildlife
* traffic
* technology

19. W E D E C I D E D T O R I F F O F F F A M I L I A R M O T I F S
A N D C O M B I N E T H E M I N T O A M A S S I V E , C O L L A B O R A T I V E P R O J E C T I O N

20. so, we went back and made another crazy rendering!
* actually we made quite a few, but this is the one that stuck:
* concrete references to the moments we loved in our research
* concrete connection to the building ourself
* this left us with an exciting concept, now all we had to do was make it!

21. O U R I N T E R F A C E M O M E N T S B E C A M E C L E A R ,
B U T W E S T I L L H A D Q U I T E T H E C O M P L E X S Y S T E M O N O U R H A N D S
E N T E R : P R O T O T Y P E S

22. R E T U R N I N G T O O U R T H R E E T Y P E S :
W E B U I L D P R O T O T Y P E S T H A T T E S T + C H A L L E N G E
/ E X P E R I E N C E SA E S T H E T I C S /T E C H N I C A L C H A L L E N G E S
i owe you some definitions! i’d like to give a quick overview, then illustrate with San Jose

23. T E C H N I C A L C H A L L E N G E S
H O W I S I T G O I N G T O W O R K ? W H A T A R E T H E T E C H N I C A L
C H A L L E N G E S ? I S T H I S E V E N G O I N G T O W O R K ? !
T H I S I N C L U D E S :
• D E V E L O P I N G F U N D A M E N T A L P R O O F S - O F - C O N C E P T
• S U S S I N G O U T E M E R G I N G / E X P E R I M E N T A L T E C H
• D E T E R M I N I N G T H E C R I T I C A L P A T H F R O M P R O T O T Y P E
T O F I N A L E X E C U T I O N
in some ways the easiest prototype to understand, though not the one we always start with

24. A E S T H E T I C S
H O W D O O U R F O R M A L C H O I C E S H O L D U P W I T H I N T H E
R E A L D E S I G N S P A C E ? W H A T A R E T H E A E S T H E T I C
U N K N O W N S ?
T H I S I N C L U D E S :
• M O T I O N A N D A N I M A T I O N T E S T S
• P H Y S I C A L M O C K U P S
• C L I C K - T H R O U G H S A N D O T H E R M I X E D P R O T O T Y P E S
these are hugely important in our type of work
* example: determining scale of content on projection, video walls, ?
* disadvantage of building physical work vs screen-based work is you do have to leave your desk
and physically try stuff!

25. E X P E R I E N T I A L
W H A T I S I T G O I N G T O F E E L L I K E ? I S S O M E O N E G O I N G
T O U N D E R S T A N D H O W T O U S E I T ? H O W D O E S T H I S
F E E L P H Y S I C A L L Y ?
T H I S I N C L U D E S :
• B O D Y S T O R M I N G A N D R O L E P L A Y I N G
• S C A L E M O C K U P S
• P A P E R P R O T O T Y P I N G
this is a crucial type of prototyping, and often the most challenging to make
* at its simplest, often involves just standing up and acting something out
* once an interaction involves more than one person, this is crucial!
* these are often the prototypes we pull external people in on earliest

26. returning to our rendering: we have a lot going on. this was actually the product of rounds of
prototyping, but does best illustrate our challenges:
* how were we going to get people to know they were affecting the building?
* what is the right scale for each piece? is it for one person or more?
* what is it going to look like?
* how were we going to get these pieces to work together?

27. that rendering actually began with another sketch, which was our kickoff into technical prototypes

28. W E N E W W E H A D T O S T A R T S O M E W H E R E , A N D S O W E
S T A R T E D T O B R E A K T H E P R O B L E M I N T O S M A L L E R P I E C E S
W E S T A R T E D W I T H W H A T B E C A M E A N E L A B O R A T E
E X P E R I E N T I A L + T E C H N I C A L P R O T O T Y P E

29. so we began with the one we new how to interact with, and new how to scale: hopscotch
* great demonstration of our prototyping process: visuals were from an old project,
quickest way to make a valuable prototype!
* third iteration, after trying pressure sensors, huge boards
* not durable enough!
* camera proved most flexible
* showed us a crucial aspect: a physical connection to the screen
* was confusing even with it next to you!

30. W E H A D A G R E A T S O L U T I O N F O R O N E I N P U T ,
B U T W H A T A B O U T T H E O N E S T H A T R E A L L Y N E E D E D S E N S O R S ?
we new we had to face the inevitable problem: we had committed to putting pieces in the city plaza
we couldn’t properly capture with a camera:
* megaphone, bird feeder, (to some extent) picnic table
we needed a solution to capture input and send it back to a computer somewhere

31. the solution seemed obvious: iPods! this was a perfect technical prototype:
* an untested piece of hardware
* a proof of concept for a proper wireless sensor
* a complete failure: horrible range, lots of setup (jailbreaking, etc), too many unknowns

32. along came a real contender: the arduino Fio
* terrible photo is of a timely coincidence: LAB-er Josh discovering this at an Arduino conference
* arduino: simple microcontroller, enabler of easy interface of hardware sensors + software
* arduino fio: built in radio frequency communication

33. A Q U I C K P R O O F O F C O N C E P T S H O W E D T H E A R D U I N O F I O T O B E P E R F E C T
S O W E F I N A L L Y W E R E A B L E T O M O V E I N T O A C R U C I A L P I E C E
O F B O T H T H E T E C H N I C A L A N D E X P E R I E N T I A L :
H O W W O U L D T H I S A L L F E E L T O G E T H E R ?
the prototype: two arduino fios as far apart as we could get them in union square
* worked fantastic

34. picnic table
LAB Physical Sensor: Arduino Fio, XBee radio, Adafruit Waveshield,
LAB-developed board + sensor(s)
LAB Digital Sensor: openFrameworks application connected to online
services (twitter, flickr, foursquare)
LAB Sensor Router: XBee and LAB-developed Processing application
TSPS application: computer vision software toolkit developed
by the LAB (www.openTSPS.com)
Plug-In-Play app: LAB-developed application (openFrameworks)
P
L
U
G
-
IN
-
P
L
A
Y
P
L
U
G
-
IN
-
P
L
A
YP
L
U
G
-
IN
-
P
L
A
Y
P
L
U
G
-
IN
-
P
L
A
Y
P
L U G - I N - P L A Y
megaphone birdfeeder open plug traffic light hopscotch C.D.M.| | | | | | twitter| flickr| foursquare|
P
L
U
G
-
IN
-
P
L
A
Y
P
L
U
G
-
IN
-
P
L
A
YP
L
U
G
-
IN
-
P
L
A
Y
P
L
U
G
-
IN
-
P
L
A
Y
P L U G - I N - P L A Y
from here, we were able to make a proper system diagram and begin testing the individual interactions further

35. finally, our first real prototype
* megaphone + hopscotch + picnic table + twitter
* as ugly as it was, it proved a mostly a success (minus the barely-functional “picnic table”)

36. W I T H O U R M A I N T E C H N I C A L C H A L L E N G E U N D E R W A Y ,
W E K I C K E D I N T O H I G H G E A R O N A E S T H E T I C S
as you can see from the prior prototype, it was time for us to get serious about the aesthetics

37. picnic
megaphone
birdfeeder
hopscotch
traffic signal
playground
twitter
flickr
foursquare
open
TAPECOLOR--------->
x7
x 6
each side diff
I have made 3 blocks for each square in the hopscotch board (there are 7 squares).
There is a full size, medium & small box for each square. Can the 3 sizes be released randomly when a user hops on a square? (Sides are aprox. height, will need adjustment.)
topBLOCK S.side
OR can volume level = saturation? ---->
E.side btm N.side W.side
each side diff
topBLOCK S.side E.side btm N.side W.side
|-----------=======each one is all sides=======-----------|
|-----------===each one is all sides===-----------|
|-----------===each one is all sides===-----------|
|-----------======-----------|
|-----------======-----------|
|-----------======-----------|
|-----------======-----------|
|-----------======-----------|
===================--------------------------------|
|-----------------=======--------------5 unique blocks --- each one is all sides---------------========----------------|
|-----------------=======--------------each one is all sides---------------========----------------|
|-----------------=======--------------tops---------------========----------------|
BLOCK top S.side
top sidetopsidetopsidetopsidetopsidetopsidetopside
E.side
top sidetop sidetop sidetop sidetop sidetop sidetop side
BLOCK
top S.side E.side
btm N.side W.side btm N.side W.side btm N.side W.sidebtm N.side W.side
BLOCK
top S.side E.side
BLOCK
top S.side E.side
BLOCK top S.side E.side btm N.side W.side
BLOCK top S.side E.side btm N.side W.side
BLOCK
top S.side E.side
|-----------------=======--------------tops---------------========----------------|
luckily, we were working with an amazing designer of beautiful graphic systems
* broke each piece into its own family as the technical system gained resolution

38. with the system underway, we began to create real aesthetic prototypes
* motion studies for when pieces began to react
* form studies for how objects could combine

39. snapshot of a high-res technical + aesthetic prototype
* same hardware as boxes test, higher res output
* first prototype where interactions tied together
* this is a screen cap, we actually tested these projected onto our office walls

40. B E F O R E W E N E W I T , I T W A S R E A L

42. * tape: product of the hopscotch prototype!
* needed a real connection to the projection, so created a circuit board!

46. L O N G S T O R Y S H O R T :
W E D I D M O R E P R O T O T Y P E S T H A N I H A V E T I M E T O S H O W ,
B U T C O U L D N ’ T H A V E D O N E T H E P R O J E C T W I T H O U T T H E M
we ended up with an incredibly complex system that completely hid the technology
* where do we go from here?
* we had made a lot of things talk to each other:
* C++ apps for the visuals
* processing apps listening to the sensors
* python scripts scraping the web
* more arduinos than we care to mention
* we were suddenly experts in coordinating a lot of different types of tech, and wanted to do something about it

47. A F T E R O U R P R O C E S S O F M A K I N G S O M A N Y T H I N G S
T A L K T O E A C H O T H E R , W E K N E W W E N E E D E D T O
S H A R E O U R W O R K
I N T R O D U C I N G :

49. W H A T I S S P A C E B R E W ?
A N O P E N - S O U R C E S O F T W A R E T O O L K I T F O R G E T T I N G
T H I N G S T O T A L K T O E A C H O T H E R , O R I N F A N C Y L A B
S P E A K F O R C H O R E O G R A P H I N G I N T E R A C T I V E S P A C E S .
I T M A K E S I T E A S I E R T O P R O T O T Y P E A N D P R O D U C E
P R O J E C T S T H A T I N V O L V E M U L T I P L E T E C H N O L O G I E S
T A L K I N G T O E A C H O T H E R .

50. W H Y S P A C E B R E W ?
P L U G - I N - P L A Y W A S C O M P L I C A T E D ,
A N D W E K N E W I T C O U L D B E E A S I E R .
P L U S , W E W A N T P E O P L E T O M A K E M O R E P R O T O T Y P E S !
F O C U S Y O U R T E C H N I C A L K N O W - H O W O N T H E P A R T S
T H A T R E A L L Y M A T T E R
E A S I L Y S W A P T H I N G S I N A N D O U T T O T E S T N E W
E X P E R I E N T I A L I D E A S
P R O T O T Y P E Y O U R A E S T H E T I C S W I T H A C T U A L T E C H

51. T H A N K Y O U .