Quality Evaluations and Accessibility of Norwegian Public Web Sites 2006. Presentation at the T4P Conference in Kristiansand 26.06.2007

1. Quality Evaluations
and Accessibility
of Public Web Sites
Svein Ølnes
Vestlandsforsking
26.06.2007

2. Content
• Background
• Method
• Indicators
• Results 2006
• Future development

3. Background
• Evaluations started in 2001
• Norge.no responsible from 2004
• Annual evaluation of all public web sites in
Norway
• 2006: 691 web sites evaluated
• 2007: In addition to public sector web
sites, also some private web portals will be
evaluated

4. Method
• Expert evaluation
• Ca. 1 hour per. site
• Trained evaluators to reduce variations
• Indicators formulated from standards, best
practice and eGovernment objectives

5. Quality 2006
• Three main areas:
– Accessibility (10 indicators – max 21 points)
• derived from WCAG priority 1 guidelines (“must”)
• possible to transform into measurable indicators
– User orientation (16 indicators – max 38 pts)
– Usable services (9 indicators – max 28 pts)
• Relative weight
– Accessibility: 24 %
– User orientation: 44 %
– Usable services: 32 %
• Accessibility is important but must be seen in
connection with other requirements
– you can have perfect accessibility on a perfectly useless
web site

6. Accessibility is part of the picture
Peter Morville’s User Experience Honeycomb

7. 2006
Results Number of % av
Indicator Alternative sites total
1.1 ALT-text No 240 34,8 % Problem
1.1 Partly 251 36,4 %
1.1 Yes 199 28,8 %
1.2 Accessible also without colours? No 103 14,9 %
1.2 Yes 587 85,1 %
1.3 Functionable without CSS instructions? No 68 9,9 %
1.3 Yes 622 90,1 %
1.4 Free from blinking elements etc. No 3 0,4 %
1.4 Yes 687 99,6 %
1.5 Are data tables marked up correctly? No 408 59,1 % Problem
1.5 Partly 166 24,1 %
No data
1.5 table 58 8,4 %
1.5 Yes 58 8,4 %

8. 1.6 Meaningful titles on frames, if frames? No 166 24,1 %
Results 1.6 No frames 511 74,1 %
1.6 Yes 13 1,9 %
1.7 Functionable without scripts and plug-ins? No 98 14,2 %
1.7 Partly 118 17,1 %
1.7 Yes 474 68,7 %
1.8 Contrast between foreground and background
colours Poor 90 13,0 %
1.8 Good 600 87,0 %
1.9 Is there a way to jump directly to content? No 586 84,9 % Problem
1.9 Yes 104 15,1 %
1.10 When using frames, is provider clearly marked
and No 64 9,3 %
1.10 is the complete frameset loaded? No frames 564 81,7 %
1.10 Yes 62 9,0 %
2006
Total number of evaluated web sites 690

9. Quality 2006 - results
• Accessibility: 75 % of max points
• User orientation: 63 % of max points
• Usable services: 54 % of max points
• Accessibility problems:
– Alternative text (ALT text) to graphic elements
– Use of data table
– Link to content

10. Quality evaluation – lessons learned
• The proof of the pudding is in the eating ->
– the proof of the indicators is in the testing
• The difficult part is to transform guidelines
into operable indicators
– difficult to reduce the subjective factor and thus
secure minimal variations between evaluators
• What do we measure?
– do we measure what is important for the
users?

11. Possible future of quality evaluations
• Combination of
– expert testing
– automatic testing
• User tests for
evaluation of the test
framework itself

12. Thank you!
Contact information
Svein Ølnes
email: svein.olnes@vestforsk.no
Web: www.vestforsk.no