This document proposes a method to evaluate accessibility-in-use by observing how users cope with barriers on the web. The method involves 3 steps: 1) Observing users and identifying coping strategies, 2) Developing algorithms to detect these strategies, and 3) Deploying the algorithms on websites to notify researchers of barriers encountered. A case study identified 17 coping strategies from observing 24 users. Algorithms were implemented for 8 strategies. The system, called WebTactics, injects detection algorithms onto pages and asynchronously notifies researchers when strategies are used. This allows capturing accessibility problems encountered by users in real world usage.

1. Evaluating Accessibility-in-Use
Markel Vigo1 & Simon Harper2 University of Manchester (UK)
1: @markelvigo
2: @sharpic
W4A 2013
markel.vigo@manchester.ac.uk
simon.harper@manchester.ac.uk
http://dx.doi.org/10.6084/m9.figshare.701482

2. Evidence
Guidelines cover around 53% of the problems
encountered by users
Power et al. 2012
Guidelines are only half of the story: accessibility problems encountered by blind users on the web
CHI 2012, 433-442
W4A 201313 May 2013 2

3. Problem
The perception of users about accessibility
barriers is difficultly measurable and
generalisable
W4A 201313 May 2013 3

4. Paradox
– Some barriers are not perceived
– Some barriers are encountered but overcome
– Barrier free pages can cause a great hindrance
W4A 201313 May 2013 4

5. Accessibility-in-use
“the effects that real accessibility problems have
on the quality of interaction as perceived by real
users when interacting with real pages for
achieving real goals”
Vigo and Brajnik, 2011
Automatic web accessibility metrics: where we are and where we can go
Interacting with Computers 23 (2), 137-155
W4A 201313 May 2013 5

6. What do we propose:
Step 1. Observation & Identification
of Coping Strategies
6
1. Observation
W4A 201313 May 2013
Cognitive markers that indicate
problematic situations

7. What do we propose:
Step 2. Development of algorithms
to detect strategies
7
1. Observation 2. Algorithms
W4A 201313 May 2013

8. What do we propose:
Step 3. Deployment in the wild
8
1. Observation 2. Algorithms 3. Deployment
W4A 201313 May 2013

9. Case study
Step 1. Observation and analysis
• 2 independent studies/datasets generated
from ethnographic studies and user tests
• 24 screen reader and screen magnifier
users
• 17 coping strategies were identified
9 W4A 201313 May 2013

10. Case study
Step 2. Implement algorithms
– Asking for assistance
– Impulsive clicking
– Exploration tactics
– Narrowing down search
– Gaining orientation
– Re-doing
– Not operating
– Giving up
10 W4A 201313 May 2013

11. 11
WebTactics
t1(){
..
}
DB
User Website Manager
/Researcher
1.1 Detection
algorithms are
injected onto web
pages.
1.2 Each algorithm keeps track of determined
sequences of events and actions.
3. This event is
asynchronously sent
to a remote location.
2. When the use of a
tactic is detected a
notification is triggered.
4. Reports from users are
viewed by interested parties.
ti(){
..
}
tn(){
..
}
id timestamp URL tactic
ccgu1331569030153 1333922552190 http://www.bbc.co.uk/sport/ t5
ccgu1331569030153 1333922556391 http://www.bbc.co.uk/sport/ t1
W4A 201313 May 2013
Case study
Step 3. Deployment: WebTactics

12. Contributions
• A method to observe accessibility-in-use
• Our approach allows to capture the
problems that emerge in the wild
• WebTactics
W4A 201313 May 2013 12

13. Follow up
13
Contact
@markelvigo | markel.vigo@manchester.ac.uk
Presentation DOI
http://dx.doi.org/10.6084/m9.figshare.701482
Source code
https://bitbucket.org/mvigo/cope
W4A 201313 May 2013