5. We don’t know how to find out about things
we don’t know about
5
6. People don’t live in the world doing one task
with one device out of context
WebEx - meetings
Marriott.com - reserving rooms
Me, Inc. - retirement planning
eHarmony - coded messages in Libya
Facebook & Twitter - organizing
demonstrations in Egypt
6
16. The nature of
being online is
social
It always has been.
email
calendaring
sharing documents
--> Name some online
transaction you think is
not social.
http://www.flickr.com/photos/holeymoon/ 10
17. Scale is a game
changer
Caution: large sample sizes
20,000 googlers didn’t
expose real issues.
Quora moderating
overburdened
Google Buzz
http://www.flickr.com/photos/daguerreotyped/ 11
18. Tasks aren’t
what you think
Activities = goals that emerge and
change
Effectiveness
Efficiency
... aren’t meaningful
http://www.flickr.com/photos/jeremybrooks/ 12
19. Satisfaction is
correlated with
task completion
Instead: control, engagement But how do you measure
control and
engagement?
What does engagement
mean?
13
http://www.flickr.com/photos/litebriteneonstudio/
20. Users How do you design effective tasks in
this case?
/cc @
continuously product reviews that turn into literary
projects
design your UI Using comments in Facebook for a
conversation
Hey, where did everybody go? -
user-created help page to describe
in real time a hack to get around a feature
they don’t like
Coded messages in Egypt on dating sites
Workarounds = hacks
Etiquette and norms
http://www.flickr.com/photos/juleshabib/ 14
21. And we have to be
careful not to violate
either of those things:
Etsy Circles
Facebook engagement
ring story
15
22. Social is about context.
And we have to be
careful not to violate
either of those things:
Etsy Circles
Facebook engagement
ring story
15
23. Social is about context.
And relationships.
And we have to be
careful not to violate
either of those things:
Etsy Circles
Facebook engagement
ring story
15
24. Methods are not robust for understanding
context and relationships.
16
25. Cultivate polymaths
Decision sciences Communication
Anthropology
sciences
Psychologies Sociolinguistics
Behavioral economics Organizational behavior
Psychobiology Sociology
Social neuroscience Social networks
17
26. The most successful user
research methods for SxD
combine field and testing
techniques
18
31. Methods
Reviewing online profiles, connections in groups in
interviews
Stories of how you met
Multi-user sessions with people who have strong and
loose ties
19
32. Methods
Reviewing online profiles, connections in groups in
interviews
Stories of how you met
Multi-user sessions with people who have strong and
loose ties
Video diaries with retrospective review
19
33. Methods
Reviewing online profiles, connections in groups in
interviews
Stories of how you met
Multi-user sessions with people who have strong and
loose ties
Video diaries with retrospective review
Experience sampling with SMS
19
45. Rethinking user
research
We’re not getting the answers we need.
Experimenting is limited because we’re
pressured to go to market.
We’re looking for things we know about.
28
\nThe story of the session. \nEnding: she and her father would have done this together. \n\nThis tool is a piece of the social web \n\nNothing unusual about this test. \none person, one computer\npretend to do a task \n\n
In use at least 20 years \n\nMethod failed Shelly desperately \n\nStarted thinking about this with Buzz\nFTC\n\n
\nBuzz testing \n\nAfter I blogged about it... \n\n
We’re not sure how to do user research that will tell us what we need to know about relationships online\n Something’s not working, and I think it’s this: \n
People simply do operate in real life the way we ask them to behave in usability tests. \n\nWebEx - observe meetings? No \n\nMarriott.com - take into account the others who decide what hotel? No \n\nRetirement planning tool - husbands + household expenses; wives + children about healthcare decisions \n\nEgypt and Libya, using eHarmony and Facebook to pass coded messages: Who knew a social network would *ever* be used that way? \n
\n(talk) \n
\n(talk) \n
\n(talk) \n
The methods were probably *always* less than perfect. \n Considering SxD magnifies the imperfections of user research methods\n \n
The methods were probably *always* less than perfect. \n Considering SxD magnifies the imperfections of user research methods\n \n
The methods were probably *always* less than perfect. \n Considering SxD magnifies the imperfections of user research methods\n \n
\nSeems that with ambient technology, everything has changed. \n\nBut it hasn’t. \n\nThere are 5 factors to user research that I’m convinced we should rework, rethink to avoid things like the Buzz and Etsy privacy violations. \n
\nAnything one person does to change the behavior of another is social. \n\nName one thing you can do online that isn’t social. \n
\n From one person + one computer... \n Large sample sizes won’t tell you everything you need to know. The Google 20,000 didn’t reveal the privacy problem. Why not? \n Looking at scale another way, let’s think about Quora and it’s massively user-generated IA. How do you design for that? How do you test for that? How do you do user research that will tell you how to go? \n
Usability testing and user research methods use tasks - but users don’t come to technology with task goals in mind. \n ephemeral, changeable + context, timing, space, what just happened, what happens next\n \n Tasks are probably activities where goals emerge and change depending on context, timing, space, etc. \n
This is probably the wrong measure. \n We should look at control by the user and engagement. People want to be the boss of their data. Buzz and Facebook’s Beacon, and now Etsy’s changes take that control away. \n We need to take a hard look at conventional metrics. \n What is effectiveness and efficiency with Twitter and Facebook open all day? What’s the task? \n Measures of success don’t translate to SxD\n time?\n task completion? \n success? \n
People use your UI in ways you didn’t intend, but that help them do what they want to do.\n Individual workarounds now surface to everyone in a network. \n Etiquette and social norms adjust over time and scale. \n Design is effectively open sourced to the user community\n
Onlineness, ambient technology is about context and\n\nwho is zooming whom \n\nWe must not violate the context of the relationships. Otherwise, problems. \n\nClients learn embarrassing purchases - Etsy\nGirlfriend learns source of engagement ring - Facebook\n
Onlineness, ambient technology is about context and\n\nwho is zooming whom \n\nWe must not violate the context of the relationships. Otherwise, problems. \n\nClients learn embarrassing purchases - Etsy\nGirlfriend learns source of engagement ring - Facebook\n
Something built for HCI doesn’t cover the expanded view of this one person interacting online and the implications of that. \n\nLab testing & structured field visits are lacking. \n
\n
Borrow, borrow from, combine. \n\nI interviewed smart people.... \n\nThis is what they told me: The key is combiningmethods. \n
\nMet: Twitter road trip\n
\nMet: Twitter road trip\n
\nMet: Twitter road trip\n
\nMet: Twitter road trip\n
\nMet: Twitter road trip\n
\nMet: Twitter road trip\n
\n
\n
\n
\n
And so, we’re back in the lab in Waltham. \n\nShelly is crying as she looks out at her future self. \n\nWe asked a business question. \n
\nThe business would have been better served by our asking more about Shelly. \n\nWhat’s the right question? \n
\n
\n
With our generation old methods, we really only know how to find things we’re looking for. \n\n\n
With our generation old methods, we really only know how to find things we’re looking for. \n\n\n
With our generation old methods, we really only know how to find things we’re looking for. \n\n\n
We don’t have good ways of learning about things we don’t know about. \n
\nTime to rework the old standard methods into new remixes. \n