Acting Naturally with Information - IA Summit 2017

1. Marsha Haverty Acting naturally with information @mjane_h 2017 Information Architecture Summit

2. The Tree of LifeArtistic illustration of the tree of life by Evogeneao I started out in biology; the tree of life was my first taxonomy. (NOTE: this image is an artistic illustration of evolutionary complexity. For the latest structure with scholarly rigor, see Banfield and Hug: http:// www.nature.com/articles/nmicrobiol201648 )

3. Then I learned that not just biology had a classification system…

4. ...Any subject could.

5. Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance InstanceInstance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance And you could structure the information.

6. Instance Instance Instance InstanceInstance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance InstanceInstance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance And the whole thing could move with an actor's meaningful action.

7. Instance Instance Instance InstanceInstance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance InstanceInstance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Structure could even facilitate it.

8. Instance Instance Instance InstanceInstance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance InstanceInstance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Information Architecture I'll always love biology. But I'd found information architecture. That's home.

9. Earth image: NASA Apollo 17 crew https://upload.wikimedia.org/wikipedia/commons/thumb/7/78/The_Blue_Marble.jpg /1023px-The_Blue_Marble.jpg Sagan (1994) And, that’s home. That’s us. (Carl Sagan, Pale Blue Dot, 1994). A bunch of tribal hunter-gatherer poets. As designers, we’ve opened a lot of wormholes to new information around here, designed information. As humans, we’re really good at assembling our actions with designed information. Some designs are complicated, hard to understand, and we cope, and some are clear, aligned with out values, and we thrive. But whether we are engaging something difficult or clear, there’s something deeper about us as humans that makes the way we behave feel natural, or distinctly off.

10. Earth image: NASA Apollo 17 crew https://upload.wikimedia.org/wikipedia/commons/thumb/7/78/The_Blue_Marble.jpg /1023px-The_Blue_Marble.jpg Girl with Amazon Echo: Android Central, Infraworks AR Sandbox: Autodesk, Tilt Brush VR: Google, Woman hailing drone: Mercedez-Benz, self-assembling material: Adventurine, generative design for a chair: David Benjamin, Autodesk Variety in designed information Manifestation Agency Action to engage The information we’re experimenting with around here has more variety in how it manifests to actors, the kinds of actions actors must use to engage it, and in its agency (autonomy, shades of intelligence).

11. Earth image: NASA Apollo 17 crew https://upload.wikimedia.org/wikipedia/commons/thumb/7/78/The_Blue_Marble.jpg /1023px-The_Blue_Marble.jpg Natural vs. Unnatural Information physically present vs. simulated Actions to engage new vs. familiar Does not depend on: Some of these things feel pretty natural, and some of them feel like novelties that we don’t really want to be doing for very long. But the distinction doesn’t depend on whether the information is physically present or simulated; it doesn’t depend on whether the kinds of actions we use to engage is are new or familiar. Throwing a hologram across the room or talking at an appliance or hailing a drone can all come to feel perfectly natural. The distinction between natural and unnatural comes in something deeper still about human behavior.

12. The Observer from Fringe: Michael Cerveres The nature of human behavior? We don’t have a handy gadget like the one the Observer on the science fiction show Fringe had through which to peer at the nature of human behavior.

13. The Observer from Fringe: Michael Cerveres ECOLOGICAL PSYCHOLOGY The nature of human behavior? But we do have a very powerful lens called Ecological psychology. This field was founded in the 1970s and is just now coming into its own.

14. Golanka (2015) ECOLOGICAL PSYCHOLOGY Events of our actions Events of information Human behavior is about aligning events (NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition is not what you think it is,” available in full from Frontiers in Psychology.)

15. Golanka (2015) ECOLOGICAL PSYCHOLOGY Events of our actions Events of information Human behavior is about aligning events

16. Golanka (2015) ECOLOGICAL PSYCHOLOGY Human behavior is about aligning events Events of our actions Events of information

17. Golanka (2015) Human behavior is about aligning events ECOLOGICAL PSYCHOLOGY Events of our actions aligned with events of information assemble a system with dynamics that complete a task Behavior is a system. Behavior includes us, but isn’t only us. It’s the events of information too.

18. We’re aligning our actions with information, but it feels to us as if we’re wielding the dynamics of the world to act in task-fulfilling ways Barrett (2011)

19. We’re aligning our actions with information, but it feels to us as if we’re wielding the dynamics of the world to act in task-fulfilling ways Barrett (2011) We’re not acting to impose our will on the world, we’re acting to receive the world in a useful way

20. Image: Barcroft Media via Daily Mail, 3D image of human brain connections That changes the role of these beautiful brains we walk around with. Our brains don’t have to do all this processing, predicting exactly how we’re going to behave before we do it. Our brains act as coordinators: they coordinate the way we assemble our actions with information in the world around us.

21. Natural human behavior? But, what makes behavior feel natural?

22. Natural human behavior Selecting action Golanka (2015) Controlling action Natural human behavior is about two things: selecting action and controlling action. (NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition is not what you think it is,” available in full from Frontiers in Psychology.)

23. Selecting action Switching action Styling action Selecting Golanka (2015) Gif of Alondra de la Narra: http://indieclassical.tumblr.com/post/138392136778/exercisesinhumiliation-alondra-de- la-parra-is Selecting is about deciding for a given task which of the possible actions will we engage (if we’re thirsty and we see a glass of water we decide to drink from the glass), we decide when to switch actions (as we are reaching for the red glass, someone says, “take the blue glass” so we switch to reach for the blue glass), and we style the way the action unfolds (if we notice that the glass is fragile, we slow our actions and pick it up carefully). Animated gif: http://indieclassical.tumblr.com/post/138392136778/exercisesinhumiliation-alondra-de-la-parra-is

24. Selecting action Switching action Styling action Selecting Golanka (2015) ACTOR AS CONDUCTOR Gif of Alondra de la Narra: http://indieclassical.tumblr.com/post/138392136778/exercisesinhumiliation-alondra-de- la-parra-is When an actor does this with designed information, it feels to her like she’s conducting with information. Browsing a website, the actor anticipates the performance of the categories as she selects one; then aligns with the meaning of the results to anticipate and selecting more, sometimes switching to go in another direction. It’s this cascade of selecting and switching that feels like she’s conducting with information.

25. Selecting action Switching action Styling action ACTOR AS CONDUCTOR Berrypicking model for search: Marcia Bates (1989) Golanka (2015) Cascade of selecting, switching, styling Anticipating-aligning with the performance of information We’re also conducting when we perform digital search. In Marcia Bate’s berrypicking model for search, the actor anticipates the performance of the domain indexing in selecting and styling her initial wordings of her query. She then aligns with the meaning of the results, grabbing the good stuff, and then selecting and styling and switching her query to anticipate further performance, aligning with the meaning of the results, further refining her anticipation of the performance of the system. Actor-as-conductor is a cascade of selecting, switching, styling in which the actor is anticipating and aligning with the performance of the system of information.

26. Selecting action Controlling action Natural human behavior Golanka (2015) Selecting is behavior: the action of selecting action Selecting is an event: the action of selecting action. Now, we’ll look at controlling action. (NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition is not what you think it is,” available in full from Frontiers in Psychology.)

27. Image by permission Cherrico Pottery LLC http://www.cherricopottery.com/ Controlling Controlling action unfolding continuously over time Golanka (2015) ACTOR AS SCULPTOR We are always orienting ourselves physically in place and time with our surroundings. When we act in placetime, we control the way our physical actions unfold continuously over time with information in our surroundings. When an actor engages designed information that does this, it feels to the actor like she’s sculpting with information.

28. ACTOR AS SCULPTOR Golanka (2015) Controlling action unfolding continuously over time The industrial design of a car supports actor as sculptor. The steering wheel, the gas and brake pedals, and the transparent windshield are all streaming information that the actor relies on to control action unfolding over time; the actor feels as if she is wielding the dynamics of the surroundings so that she and the car are receiving the right road, and the lane stays centered around her and the car, and the other cars flow off to her sides instead of into her.

29. Golanka (2015) Controlling action unfolding continuously over time ACTOR AS SCULPTORACTOR AS CONDUCTOR The industrial design also supports actor-as-conductor of information. A glance at a map to select the right road. A glance at the speedometer to decide to switch her actions to slow down: she engages actor-as-sculptor behavior, relying on the shifting position of her foot on the brake pedal, and the way the visual information is streaming through the windshield to control how she slows down continuously over time. (NOTE: Experienced drivers are really bad at mimicking the act of driving. That’s because drivers don’t decide in their heads how to handle the steering wheel or adjust the gas and break pedal positions. They rely on the continuous flow of information to help them choreograph the behaviors of slowing down and choosing the right road and staying in the lane and moving around a car. Without this continuous flow of information to rely on, the meaning of the behavior is broken.)

30. ACTOR AS SCULPTORACTOR AS CONDUCTOR Historically, our field lived here Historically, our field lived here– actor as conductor– and actor as sculptor was the stuff of industrial design. For actor-as-conductor designs, there is of course some physical action that has to happen to get the cursor on the pixels, or get the fingertip on the target, or say the words to trigger the response of the system, but there were lots of degrees of freedom for how the actor controlled physical actions over time. Whether the actor was sitting down, or walking, or reclining, and the arc and speed of the motions to work a mouse or tap a screen didn’t participate directly in the meaning of the activity.

31. ACTOR AS SCULPTORACTOR AS CONDUCTOR Historically, our field lived here The thread of meaning is wandering here… But, now the thread of meaning is wandering over here, to actor as sculptor.

32. ACTOR AS SCULPTORACTOR AS CONDUCTOR Autodesk Alias sample car concept Same with physical product design at smaller scales. Designers of physical things like cars and products are always looking for ways to experience their designs before they are brought into the world in physical materials.

33. Feel human proportionality, how designed objects will occupy surroundings ACTOR AS SCULPTORACTOR AS CONDUCTOR Autodesk Alias sample car concept; woman with Tilt Brush by Google; Tilt Brush car doodle by wordpresshelp1234 They are experimenting with sketching in 3D with simulated light trails. They walk around in 3D drawing the geometry exactly as it will take up space. They feel the curves, they can walk around the designs, feeling their proportionality to humans, how the designed objects will physically occupy their surroundings. This is actor-as-sculptor of information: the way they control action continuously over time to draw the light trail is the meaning. There are no degrees of freedom in their physical actions because the physical actions participate in the meaning directly.

34. ACTOR AS SCULPTORACTOR AS CONDUCTOR Designs in Autodesk Revit and VR by Samuel Arsenault-Brassard Building architects are experimenting with the handoffs across conducting and sculpting with information. This architect has detailed a building in architectural software (conducting with information). He brings the building into 3D space in order to sketch some new concepts. He scales the building down so he can walk around it and look down on it. He relies on information about the way the building occupies space to guide his hand motions as he sketches new towers into the existing geometry. He can feel how the curves will occupy space with respect to the rest of the building.

35. ACTOR AS SCULPTORACTOR AS CONDUCTOR Designs in Autodesk Revit and VR by Samuel Arsenault-Brassard Then, he can bring his rough sketch back into the software and do all the conducting (selecting, switching, styling) to detail the sketch and rationalize it with the building.

36. ACTOR AS SCULPTORACTOR AS CONDUCTOR Work with holograms as physical objects MIT is experimenting with focused sonar that creates mechanical force against an actor’s hand. Now actors can do things like touch holograms, and explore their surfaces and textures. An actor could work with holograms as physical objects, relying on haptic information to help control physical actions continuously over time. (Image: MIT)

37. ACTOR AS SCULPTORACTOR AS CONDUCTOR High-precision motion tracking collapses degrees of freedom of spacetime action into meaning Image: Google Soli project Google is using radar to very precisely track the position and motion of human fingers. The actor can mimic things like turning a dial, moving a slider, pushing a button. These are still things for conducting (selecting, switching, styling), but there is a tiny bit of actor-as-sculptor: the actor’s fingers become tactile information for each other to rely on to control action continuously over time. It’s just fingers now, but soon it will be hand, arm, the actor’s entire body position at high precision that could participate in sculpting with information.

38. Actor as Conductor Selecting action, switching action, styling action Dynamics of surroundings Concepts INFORMATION Physical alignment Semantic alignment MEANING Golanka (2015) Golanka (2015) Actor as Sculptor Controlling action unfolding over time Structure of natural human behavior Those are just a few examples of how we’re stating to need to worry about both human behaviors. Now I want to build out the structure of natural human behavior for both of these. For actor as conductor, the meaning is semantic alignment: why we’re selecting an action, why we’re switching action, why we’re styling the action. For Actor as sculptor, the meaning is physical alignment. For actor as conductor, the information is about concepts; for actor as sculptor, the information is about the dynamics of the surroundings. Let’s look at what that means. (NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition is not what you think it is,” available in full from Frontiers in Psychology.)

39. This is not information for surfing Underlying dynamics This is not information for surfing. This is how wind transfers friction to water molecules, pushing and pulling them to form a wave. This is the underlying dynamics that create a wave, but this is not information for surfing.

40. This is information for surfing Invariant structure–the shape of the opening–creates affordances for a surfer to stay inside this place made of liquid information This is information for surfing. This expert surfer doesn’t see these dynamics, he sees the enclosure of a wave as a place to inhabit. And he aligns his actions with information about the wave continuously over time to keep the enclosure around him. The shape and slope of the opening of the wave is information that he can use to position the tip of his surfboard. He is calibrated to the way he must align his actions to this information; the information becomes affordances for surfing for him. And when the shape of the opening changes, when he feels water hitting his back, he knows that his place is collapsing and he needs to switch action, engaging other affordances, to quickly get out.

41. Skeletal muscle animation: Eleanor Lutz; This is not information for goal keeping (soccer) Underlying dynamics This is not information for goal keeping for soccer. This is the underlying dynamics for how our muscles contract to move our skeletons.

42. This is information for goal keeping (soccer) Point light image: https://quote.ucsd.edu/cogs91/ This is information for goal keeping for soccer. Twelve points of light are enough to give information to a goal keep about a human kicking a soccer ball. The goal keep can align her actions to this information to prepare to stop the ball. She doesn’t have to have seen this particular human kicking a ball, she doesn’t have to have seen ball kicking from this particular perspective in this particular light. She sees the relationships among how one foot plants while another moves with respect to the knee. This becomes invariant structure, information about ball kicking, that work as affordances to align with for a goal keeper. (NOTE: visit the URL above to see the animated gif to get the idea of this information.)

43. Information about concepts is different Macro photo of Kindle Reader e-ink; vocal tract anatomy Bruno Dubuc Point to meaning by convention FREEDOM SPOON (Not about underlying dynamics) When we hear or read words like Spoon and Freedom, those things are not about the underlying dynamics of the chemicals or electrical process creating the text, or the function of the larynx creating the sounds. These words point to meaning by convention. We have decided what these mean in agreement spanning time and culture.

44. For actor as conductor, we assemble semantic alignment (why are we selecting, switching, styling an action?) with information about concepts by way of convention. For actor as sculptor, we assemble physical alignment with information about the dynamics of our surroundings by way of affordances. We see that conducting with information is not dependent on time. We can conduct in quick succession, or over weeks or lifetimes. Time does not factor into whether conducting with information feels natural or not. For actor-as-sculptor, natural behavior is directly dependent on being continuous over time. And that places some very important demands on the kind of information that can support actor as sculptor.

45. For actor as sculptor, information must be dense, persistent, and lawful. And it must be made of perceptual information. Language is not dense enough, persistent enough, (though it acts as if it’s lawful across an actor’s lifespan) to rely on to control action continuously over time. The two key things actors use perceptual information for to sculpt with information are: orienting & wayfinding, and prospecting with objects. We’ll look at both of these.

46. Orienting & Wayfinding Dense, persistent, lawful Visual Information Surfaces Edges Textures Proprioception, Kinesthesia We are always orienting ourselves physically in our surroundings. We humans are wired for proprioception and kinesthesia: we know where our bodies are in space, and we know when we’re moving. We also rely on visual information for information about the relationships among surfaces, edges, and textures in our surroundings that we can use as affordances for us to move: this path is walkable to me, these stairs are climbable to me, this gravelly hill is not passable to me. And this information is dense, persistent, and lawful. We can rely on it to control our actions continuously over time.

47. Wayfinding Haptic Information Dense (enough) persistent (enough) lawful Simulates surfaces, edges This woman has low vision and can’t rely on visual information for wayfinding. She’s wearing a device that simulates information about obstacles as haptic information. The haptic information is much less dense than visual information, and is not persistent (it’s not a constant flow, but kicks in with proximity to obstacles). But it’s dense enough, persistent enough, and lawfully specifies obstacles in her surroundings that she can use as information for orienting and wayfinding.

48. Wayfinding Sonic Information Dense (enough) persistent (enough) lawful Incoming cars Train/no train Crossing light timers Conversations Physical obstacles Incoherent with “everyday sounds” for wayfinding Gaver (1994) UNNATURAL BEHAVIOR This system can use sound instead of haptic information about obstacles. Sound is dense enough, persistent enough, and lawful, but the problem with sounds is that it’s directional, but not spatial. It piles up. That’s great for music, but not for wayfinding. Haptic information can use spatial relationships to represent multiple obstacles and their relative positions. Sounds just pile up. The other problem with sound fir wayfinding is that humans rely on many other very important sounds: incoming cars, train/no train, crossing light timers, conversations. And the sounds about physical obstacles steps on those so they are not dense, persistent, and lawful. This is unnatural behavior. (NOTE: see Gaver 1994 in references for an excellent discussion of ecological taxonomy of “everyday sounds” humans rely on for information.)

49. So we see that we rely on many kinds of perceptual information for orienting and wayfinding, and we see that, in our designs to support this, we must ensure the information we use is not only dense enough, persistent enough, and lawful, but coherent with the other perceptual information actors are relying on for the activity. So, we’ll add coherence as a design consideration.

50. VR has a coherence problem "I started forgetting which walls were virtual and which ones were real, and I ended up walking into a few very solid walls" “A player attempted to push his hand through the floor to sculpt something’” Disrupts critical information for placetime orienting and wayfinding Dense (enough) persistent (enough) lawful UNNATURAL BEHAVIOR Virtual Reality also has a coherence problem. VR disrupts critical information for placetime orienting and wayfinding. Players of VR games report things like, “A player attempted to push his hand through the floor to sculpt something.” Another: “I started forgetting which walls were virtual and which ones were real, and I ended up walking into a few very solid walls.” These designs are mitigating this by adding in wall alerts and visual grids to show were physical walls are. These things are dense enough, persistent enough, and lawful to represent the walls. The problem is that they are like aroras from another reality slicing through the information from the present reality. They disrupt the dense, persistent, lawful information in the present reality for orienting and wayfinding and ask the actor to align with two sets of conflicting information. That’s unnatural behavior. Quotes from: http://www.businessinsider.com/virtual-reality-gamers-are-literally-colliding-with-the-physical-world-2016-4

51. Image credit: Designboom (Alison Brooks Architects ‘The Smile’) “One sees the environment not just with the eyes but with the eyes in the head on the shoulders of a body that gets about” J. J. Gibson, founder Ecological Psychology Gibson (1979) The other problem with virtual reality is that we humans naturally need to explore our environment. JJ Gibson, the founder of Ecological Psychology says…

52. Image credit: Designboom (Alison Brooks Architects ‘The Smile’) Principle of action for VR: “A participant must have affordances for moving about in the scene…take action in the world and perceive the effects. This is part of the larger sense of personal agency.” Brenda Laurel Laurel (2016) Brenda Laurel has a principle of action for VR...

53. So, we’ll add explorability as a design concern.

54. Prospecting with objects Image: https://quote.ucsd.edu/cogs91/ Color relationships become affordances for banana edibility Surfaces green : yellow : brown We don’t just start using objects, we prospect with them using information about them. If we have experience eating bananas, the relative amounts of green:yellow:brown color on the surface of the peel becomes information to us. We become calibrated to the relative colors to know that a banana like this will have a peel that droops, we anticipate seeing brown spots on the fruit, we’d expect the fruit to be a little mushy and sugary. This information becomes affordances that we can rely on about whether this banana is edible to us. object affordances: Gibson (1979)

55. Prospecting with objects Surfaces Edges Textures “smooth” “glassy” Relationships of surfaces, edges, textures become affordances for finding my phone by dynamic touch Curve of edges to face End edges : side edges Edge thickness We can also prospect with objects using dynamic touch. I know my phone visually, but I also know my phone by touch. If I’m reaching into my bag to retrieve my phone, I don’t just touch it once and know that it’s my phone, I must explore the relationships in the haptic information about the surfaces, edges, textures of the phone. I understand that the surface is glassy and smooth. I understand the relationships among the end edges to the sides; the edge thickness; the curve of the edges to the face. This haptic information becomes affordances to me to retrieve my phone. dynamic touch: Gibson (1979)

56. Are our simulated objects explorable? Do they know what kinds of objects they are? Prospecting simulated objects What kinds of information should they simulate to reveal new underlying dynamics? New kinds of transformations Permutational Computational Smart Materials Same concern as simulated objects Tactile/HapticVisual Sonic Same information as physical objects What about prospecting simulated objects? We can simulate the same kinds of infomraitno that physical objects have: visual, tactile, sonic. But we can also infuse simulated objects with new kinds of underlying dynamics that physical objects don’t have: new kinds of transformations. But, we also need to infuse them with information about these new kinds of dynamics that humans can rely on. We need to start thinking about what a permutational affordance is like; what a computational affordance is like. Smart materials have the same concern as simulated objects: smart materials are physical materials that are infused with new underlying dynamics. How do we coax smart materials to radiate information about their underlying dynamics for humans to calibrate with and use as affordances? (NOTE: We also need to ask, do we truly need a spacetime simulation? If the information is more abstract than the way the object orients and embeds in spacetime and relative to the actors proportions, are we just making a 3D hologram versio of a skeuomorphism?)

57. Simulations have no mass! Prospecting simulated objects “There's no physical furniture to break your fall when you try to sit in a virtual chair. Someone asked, Where did that bruise come from? I achieved presence.” UNNATURAL BEHAVIOR “Breaking the ecological law” Wilson (2012) Yet… they provide affordances for physicality COHERENCE Tactile/HapticVisual Sonic touch-ability, throw-ability, transformability sit-on-ability, climb-on-ability Yet… one physicality affordance is not lawful Visual Explorability leads to a coherence problem for simulated objects. Simulations have no mass! Yet, they provide affordances for physicality, we perceive them as objects that take up spacetime embedded in our surroundings. Get dense, persistent, lawful visual and haptic and tactile and sonic information, depending on how they’re wired up, about touch-ability (surfaces and textures), pickup-ability, throw-ability, transformability, and so on as far as our imaginations take us. Yet, one aspect of their affordances for physicality is not lawful. We get visual information for things like sit-on-ability, climb-on-ability. Surfaces and textures suggest the lawful presence of mass. These are affordances we’ve been attuned to our whole lives, and we will engage with. One person in VR recently said: “There’s no furniture to break your fall when you sit on a virtual chair. Someone asked, ‘Where did that bruise come from?’ I achieved presence.” Andrew Wilson, a present day Ecological Psychologist, would tell us we are breaking the ecological law by simulating affordances for properties that our objects do not lawfully provide. That’s unnatural behavior.

58. True mixed reality: 3D scanned objects + simulated objects coexisting Prospecting simulated objects Scanned objects have physical affordances simulated objects don’t How can we help distinguish them so they can coherently co-exist? When we get away from this hard divide between virtual reality and holograms in “augmented reality,” and get to true mixed reality, we will have 3D scanned physical objects co-existing with simulated objects. Scanned objects have physical affordances that simulated objects don’t. How can we help distinguish them so they can coherently co-exist? (NOTE: see Brenda Laurel’s reference for a description of mixed reality as a spectrum.)

59. So, that’s the structure of actor as sculptor behavior. Now let’s build out actor as conductor. Information to support conducting can take the form of both language and perceptual information. Language for conducting is about coordinating and understanding. We’ll start with coordinating.

60. Ecological psychology tells us that natural human conversation is about coordinating. When someone speaks and someone listens, that evokes concepts. Concepts point to meaning. Meaning points to values. And that’s enough to evoke a virtual array of cultural possibilities among the participants. Something as seemingly here-and-now as asking which movie should we see tonight locates speakers and listeners in place and time, and evokes all of their placetime histories as an array of cultural possibilities. Speaking is anticipating cultural possibilities by improvising wordings. Listening is aligning with the discovered nature of the cultural possibilities, and entraining with rhythms, timing, pronunciation, grammar of the participants in the conversation. A conversation is a creative tension: anticipating and aligning; improvising and entraining. The acts of conversation feed back into the structure of the cultural possibilities, changing the structure over time. The overall behavior of this system of conversation is about coordinating actions among the participants to realize values.

61. Natural Human Conversation Hodges (2014) A creative tension in which participants resonate to the larger story in which they are participating, orienting and directing them toward a good continuation Coordinating action to realize values

62. Giant, ongoing, distributed conversation with highly refined entrainment of grammar across its placetime histories Many overlapping conversations with participants resonating to very different structures for good continuation When someone that aligns with a particular cultural structure for good continuation encounters acts of conversation that resonate to a very different structure for good continuation, there is nothing to align with. The array of cultural possibilities is invisible to both parties. Is there any way for us to visualize these structures to help facilitate some kind of understanding (if not aligned values) across conversations?

63. Conversation If you want to visit virtual reality, just speak to someone, and listen, with intent toward a good continuation A virtual semantic array made of your collective histories of culture at every placetimelevel will emerge all around you

64. So, we see that natural human conversation is about coordinating. Now let’s look at using language for understanding. (NOTE: I am using “understanding” as the label here to mean the notion of design for understanding in Morville, Rosenfeld, Arango (2015).)

65. When an actor engages a design that supports something like shopping, banking, creating, hailing/styling a service, etc., the actor is engaging with language based on classification: the underlying information architecture of the designed system. And the purpose of the behavior is understanding/performing for a good continuation. The actor engages the design and that evokes concepts. Concepts point to meaning. Meaning points to values. And that’s enough to evoke a virtual array of classificational possibilities. This takes the form of the underlying system of information architecture. The actor anticipates the performance of the classification by improvising selecting/switching/styling. Selecting/switching/styling can take the form of browsing, searching, commanding, hailing, parameterizing, etc., whatever the design facilitates. The actor then aligns with the meaning of the performance of the system (discovered nature of the classificational possibilities). And entrains with the labeling and grammar of the system. Entraining is vital here: if the actor does not entrain to the labeling and grammar of the system, the virtual array of meaning falls apart. We know from our field of IA that a system must be clear, and resilient across channels and contexts. The behavior of the system is about understanding/performing for a good continuation.

66. Now we are playing with the parts of this system: autonomous digital agents offload from the actor the tension of anticipating/aligning with the classificational possibilities. The agent does this work. But, when the agent needs to interrupt the human with a decision point or status, the agent must bring the right kind of information to let the human anticipate/align with the digital agent for a good continuation. We are also injecting machine learning into the system. That can greatly expand what is allowable for entraining with the system. Entraining is still vital, but the human has more flexibility in what is allowed for labeling and grammar. Machine learning in which the system is expanding the nature of the underlying classificational possibilities will expand the scope of what the human needs to anticipate and align with. I think that things like machine learning and conversational UI are really about expanding the tolerance for imprecision the human must have to engage the system.

67. Human-System imprecision tolerances NARROW Tolerance Precise entrainment WIDE Tolerance Forgiving entrainment Natural Language Processing Machine Learning (entraining actor’s wordings) Conversational UI Standard searching, voice commanding Gestural UI (degrees of freedom of motion, subtle vs elaborate) For any given system, the system has a tolerance for imprecision in what is needed to engage it. A narrow tolerance for imprecision would be a system that requires very precise entrainment with labeling and grammar. A wide tolerance for imprecision would be a system that has forgiving entrainment with labeling and grammar. Something like standard searching and voice commanding have narrow tolerance for imprecision. We can do things like autosuggest, and spelling forgiveness, but these are generally very precise labeling and grammar requirements. Conversational UI expands the tolerance for imprecision by suggesting the classificational possibilities ahead; this gives the actor a hint of the performance of the system as she is selecing/switching/styling. Natural language processing expands the tolerance for imprecision. Machine learning in which the system entrains on the actor’s particular wordings greatly expands the tolerance for imprecision with a very forgining entraining requirement on the human. Something like gestural UI right now has a narrow tolerance for imprecision. I believe the tolerance for imprecision should be much more forgiving to mime concepts, both in terms of the degrees of freedom of motion to give the system the gist of the concept, and in whether the human can use subtle motions or must do elaborate gestures.

68. So, we’ll add imprecision tolerance as a design consideration. For actor as conductor, we can also use perceptual information. This takes the form of mapping: infusing concepts with perceptual qualities in the form of visualizations and notifications: maybe a sound or haptic buzz or color change to signal a state change.

69. Mapping Air flow and heat transfer simulations, Autodesk Visualization Values-realizing, good continuation, clear, resilient Curvature mapped to airflowColor mapped to temperature This is a visualization of the air flow and heat transfer in a hospital operating room on the left, and an airport walkway on the right. Color is mapped to different temperature ranges. The arc of the lines shows the direction of the airflow. We could describe something like this in words, but this visualization captures the ambient comfort of these places as a quick, visceral understanding. That’s because perceptual information and language feel different in terms of the ease of the flow of engaging them.

70. Perception flows easily Tacit Reflexive photo credit: Jason Pratt via Creative Commons Language is viscous, more work to flow Awareness Associativity Requires attention These kinds of information have different viscosities, or ease of flow.

71. Perceptual information Language Mercedes-Benz F 015 Concept Car VISCOSITY: ease of flow of conducting visceral Low viscosity, visual information about density, directionality, color Situational visualization human can manipulate? intense concentration conceptual High viscosity, full entrainment with author’s wordings Viscosity with precise grammar/label entrainment, linear over simultaneous Self-driving car handoff to human…? How will the car interrupt us? How will the car map sculpting information to concepts? intense coordination Something like the airflow visualization is all perceptual information, no language. It is a very visceral way to understand with information. Something like reading a novel is all language, high viscosity entraining with concepts and the author’s particular wordings. It is a conceptual way to engage meaning and requires active attention. Something like talking to Alexa also is all language, and requires precise entrainment with the grammar and wordings. It is also high viscosity because it requires linear order (you can only do one thing at a time). What about when we think about the interaction among a human and a self-driving car? The car is offloading all of the sculpting behavior of driving a car. What about when the car needs to interrupt the human for a critical decision? How will the car sum up all of the perceptual information it has been aligning with to control the actions of the car continuously over time into concepts to align with humans? Will it use wordings? Will it offer an ongoing visualization of the situation that the human can manipulate for a decision point? Like a flight tracker the human can tweak? We need to worry about the form that information will take in this kind of important handoff between sculpting (the car) and conducting (the human). NOTE: I fear my brief examples seem to imply language viscosity is negative. On the contrary! Please see my 2015 IAS talk for a more detailed treatment of viscosity: https://www.slideshare.net/mjane_h/what-we-mean-by-meaning-new-structural-properties-of-information-architecture- ias15-47485499?qid=95845315-8b4c-4c4b-8949-48b464ba8547&v=&b=&from_search=5

72. So, we’ll add viscosity as a design concern. We see that the structure of natural human behavior for actor as sculptor is physical alignment using the dynamics of the surroundings by way of affordances. It relies on perceptual information for orienting & wayfinding, and prospecting with objects. Perceptual information for sculpting can take many forms, but must always be coherent with the way other kinds of information are being used for the activity. Information for sculpting must be dense enough, persistent enough, and always lawful. Natural human behavior for actor as conductor is semantic alignment using concepts by way of convention. Both language and perceptual information can be used to support conducting. Language for coordinating takes the form of human conversation. Language for understanding takes the form of systems of classification. Perceptual information for conducting is a mapping to concepts using visualization or notification. Information to support conducting must be values-realizing, a good continuation, clear, and resilient.

73. ACTOR AS CONDUCTOR Controlling action ACTOR AS CONDUCTOR ACTOR AS SCULPTOR Which behavior is our design supporting? Degrees of freedom in controlling action Action unfolding continuously over time participates in the meaning

74. Earth image: NASA Apollo 17 crew https://upload.wikimedia.org/wikipedia/commons/thumb/7/78/The_Blue_Marble.jpg /1023px-The_Blue_Marble.jpg Follow the thread of meaning wherever it may wander As designers and information architects, we follow the thread of meaning wherever it may wander.

75. Natural doesn’t have to be familiar Image: Hugh Hochman Natural doesn’t have to be familiar, or even physically present. Nascent technologies will always race around, and we should knock ourselves out experimenting with the possibilities.

76. Image: DMITRIS_K Big Think Disrupting the ecology of information If our designs feel deep-down unnatural, and we can’t pin it on usability or usefulness or lack of clarity, or any of our other ways to say what’s good, maybe it’s because we’re disrupting the ecology of the ways our human actors are relying on information to behave.

77. Human behavior is events aligning, Tuned to align our actions For 100s of 1000s of years, and more across the branches of the tree of life, our perceptual systems have been tuned to align our actions…

78. with information around us …with information all around us.

79. with information around us (Not the underlaying dynamics causing the information.)

80. with information around us But the information. We align our actions with information around us...

81. to realize our values …to realize our values.

82. All human behavior is information behavior From an ecological psychology POV, we see that all human behavior is information behavior.

83. Marsha Haverty Thank you @mjane_h 2017 Information Architecture Summit

84. References Bates, Marcia (1989). The Design of Browsing and Berrypicking Techniques for the Online Search Interface Sagan, Carl (1994). The Pale Blue Dot Golanka, Sabrina (2015). Laws and conventions in language- related behaviors, Ecological Psychology 27:236-250. Gibson, James J. (1979). An Ecological Approach to Visual Perception Marsha Haverty @mjane_h Hodges, Bert (2014). Righting language: the view from ecological psychology. Language Sciences, Jan Cowley, Stephen and Harvey, Matthew (2015). The illusion of common ground, New Ideas in Psychology p. 1-8. Wilson, Andrew (2012). Breaking the (ecological) law: why illusory sounds don’t make for safer cars. Online http://psychsciencenotes.blogspot.com/2012/10/breaking- ecological-law-why-illusory.html Barrett, Louise (2011). Beyond the Brain: How Body and Environment Shape Animal and Human Minds, Princeton University Press Arora, Rahul et. al (2017). Experimental evaluation of sketching on surfaces in VR. CHI 2017, May 6-11 2017. Founded Ecological Psychology Science in contemporary Ecological Psychology Laurel, Brenda (2016). What is Virtual Reality? Published online on Medium, 15 June Thibald Paul J. (2011). First-Order Languaging Dynamics and Second-Order Language: The Distributed Language View. Ecological Psychology 23(3): 210-245. Gaver, William W. (1994). What in the world do we hear? An ecological approach to auditory perception. Ecological Psychology 5(1): 1-29. Covert, Abby (2014). How to make sense of any mess. CreateSpace. Hinton, Andrew (2014). Understanding Context: Environment, Language, and Information Architecture. O’Reilly Media. Resmini, Andrea and Rosatti, Luca (2011) Pervasive Information Architecture. Morgan Kaufmann. Rosenfeld, Louis, Morville, Peter, Arango, Jorge (2015). Information Architecture for the Web and Beyond. O’Reilly Media.

Acting Naturally with Information - IA Summit 2017

Du liest eine Vorschau.

Hier ansehen

Acting Naturally with Information - IA Summit 2017

Weitere Verwandte Inhalte

Diashows für Sie (20)

Andere mochten auch (20)

Ähnlich wie Acting Naturally with Information - IA Summit 2017 (20)

Aktuellste (20)