Evaluation of Prototypes and the Problem of Possible Futures
1. Evaluation of Prototypes and the
Problem of Possible Futures
Antti Salovaara, University of Helsinki
Antti Oulasvirta, Aalto University
Giulio Jacucci, University of Helsinki
UNIVERSITY
OF HELSINKI
AALTO
UNIVERSITY
3. Picture from p. 518 in Abowd (1999) Classroom 2000: An
Experiment with the Instrumentation of a Living Educational
Environment. IBM Systems Journal 38(4), 508–530.
Classroom of the future at Georgia Tech
Benefit of note-taking support
1997
Wireless handheld devices
4. 2008
“Please Keep Your Laptops in an Upright and Unlocked Position” by Alan Levine.
https://www.flickr.com/photos/cogdog/2871019373/ . CC0 1.0 Universal
Social media
Distracted attention
The future as it became
5. Study context (1997) Actual context (future)
Automated capture of
learning material
Wireless note sharingUpload after
lecture
Social
media
Overhead
projectors
Present–
future
gap
6. Study context Actual context
Design brief,
Requirement
specification
New way of
work
New
software
Functional
prototype
The old
way of
work
Usability evaluation
Present–
future
gap
7. A carelessly designed evaluation may
inadvertently evaluate wrong futures,
contexts, or user groups,
thereby leading to false conclusions and
expensive design failures.
8. Evaluation as a “time machine”
Envisioning
Concretisation
Projection
9. The non-issue of the present–future gap
Scenario-building, co-design, PD
Future orientation, artifacts, empirical approach
Experimental method
Future orientation, artifacts, empirical approach
Ethnographic methods, participant observation
Future orientation, artifacts, empirical approach
Engineering and programming
Future orientation, artifacts, empirical approach
Field trials, Usability evaluations
12. CONTROL Removal of present-day features
STAGING
Creation
of
futuristic
features
Wireless
note sharing
Overhead
projectors
Every methodological choice changes
the enacted future
14. INHIBITION in an urban navigation study
concretization
Free
walking
Embedding information in
urban environment for AR
Limit use of AR to a
small area
15. PROPPING in an in-the-wild AR study
concretization
Hi-fi prototype
Human
actors
Embedding information in
urban environment for AR
16. GAMIFICATION in an AR experiment
concretization
Excitement helps users
forget their hesitation
19. Near future Far future
Precise range of
futures
Singular
futures
Margin of tolerance
20. New mindset for evaluations
1. Evaluate with the future in mind
2. Consider the future-contingency of every
methodological choice
3. Consider how evaluation must be
controlled & staged
4. Report methodological choices
transparently
5. Define the margins for projective validity
21. Thank you!
Acknowledgments:
Academy of Finland (259281 and 298879), European Research Council (637991)
David McGookin, Barry Brown, Duncan Brumby, Kasper Hornbæk, Mika
Jokiniemi, Stuart Reeves, Baris Serim, John Zimmerman
Evaluations are about what the
prototype might become.