Oplægget blev holdt ved InfinIT-arrangementet "Temadag om Remote Usability Testing" afholdt den 17. januar 2012. Læs mere på http://www.infinit.dk/dk/hvad_kan_vi_goere_for_dig/viden/reportager/mere_tid_prioritering_og_penge_til_brugbarhedstests.htm

1. Hvad er Remote Usability Testing?
Jan Stage
Professor, PhD
Forskningsleder i Informationssystemer (IS)/Human-Computer Interaction (HCI)
Aalborg Universitet, Institut for Datalogi, HCI-Lab
jans@cs.aau.dk

2. Oversigt
• Forhindringer i praksis
• Ideen om remote usability testing
• Realisering af ideen
Institut for Datalogi 2

3. Oversigt
• Forhindringer i praksis
• Ideen om remote usability testing
• Realisering af ideen
Institut for Datalogi 3

4. To undersøgelser af forhindringer i praksis
50 Primært Nordjylland
45 43 1. Software-virksom-
40 heder, typisk svar fra
35 mellemledere
30 2. Deltagere i en-dages
25 26
25 23 konference om
20 usability
15
15 13
11 10
8 8
10
6 • Tallene angiver procent
5 5
5 af svarene i hver
1
0 0 undersøgelse
0
Resource Developer Customer Test Conducting No problems Low priority No answ er
demands mindset participation participants tests
Første undersøgelse
Anden undersøgelse
Institut for Datalogi 4

5. Oversigt
• Forhindringer i praksis
• Ideen om remote usability testing
• Definition og baggrund
• Tidligere arbejde
• Synkrone og asynkrone metoder
• Realisering af ideen
Institut for Datalogi 5

6. Definition and Motivation
• Remote usability testing is a situation where
“the evaluators are separated in space and/or time from users”
(Castillo, Hartson and Hix, 1998)
• Well-known challenges
• Building and setting up usability labs
• Getting users to the lab and making them feel comfortable
• New challenges
• Users, developers and evaluators are distributed geographically, sometimes
even in different countries and time zones
• Outsourcing
• Open Source Software development
• Global software development
• Remote usability evaluation is becoming increasingly relevant
Institut for Datalogi 6

7. Earlier Work (1)
• The idea of remote evaluation emerged between 1994 and 1998
• Literature survey on remote usability testing
• 21 references
• Early: presents a variety of methods
• Recently: practical advice and experiences combined with pros/cons
Institut for Datalogi 7

8. Earlier Work (2)
Institut for Datalogi 8

9. Earlier Work (3)
• Two types of methods: synchronous / asynchronous (various forms)
• The majority of references focus on synchronous usability evaluation:
Simulating a conventional laboratory-based think-aloud test
• Overall:
• Few systematic comparisons of remote methods
• Mostly comparisons of a single remote method and the conventional lab-based
method
Institut for Datalogi 9

10. Synchronous Methods
Laboratory testing (LAB)
• Conventional method
• Benchmark for comparison
Remote synchronous testing (RS)
• Spatial separation: User and test monitor in
separate rooms
• Remote sharing of desktop
• Webcam connection with picture in the
bottom corner of the screen
• Audio connection
Data: Video recording of the test monitor’s
screen and the audio
Institut for Datalogi 10

11. Asynchronous Methods
Remote asynchronous user testing (AU)
• Based on the idea of users reporting the critical incidents they experience
(Castillo, Hartson and Hix, 1998; Hartson and Castillo, 1998; Hartson, Castillo, Kelso
and Neale, 1996)
• The users solve the tasks with the system
• Simultaneously fill in a web-based form each time they encounter a usability problem
• The form briefly explains what a usability problem is and how it can be categorized
• In the form, they describe each problem and categorize it’s severity
• Spatial and temporal separation
Remote asynchronous expert testing (AE)
• Same method as AU but with users that have expertise in usability engineering (graduate
students in HCI)
Data: A set of described usability problems and a start and end time for each user
Institut for Datalogi 11

12. Survey of Remote Asynchronous Methods
Method Paper #
Auto logging 20, 28, 29, 30, 34, 35, 36, 37
Interview 10, 26, 29, 30, 35
Questionnaires 10, 19, 25, 28, 29, 30, 32, 34, 35,
36, 37
User-reported critical incident 1, 5, 6, 7, 13, 14, 31
Unstructured problem reporting 10, 19, 38
Forum 21
Diary 31
Institut for Datalogi 12

13. Oversigt
• Forhindringer i praksis
• Ideen om remote usability testing
• Realisering af ideen
Institut for Datalogi 13

14. Realisering af ideen
• Forhindringer og udfordringer
• Tekniske: Kan man få det til at fungere
• Metodiske: giver det brugbare resultater
• Praktiske: kan man gøre det i praksis
• Det skal vi arbejde med resten af dagen …
Institut for Datalogi 14

15. aa
Institut for Datalogi 15