Analyzing the results of the Norwegian benchmarking system for quality of public websites in the period 2007.2011

1. www.vestforsk.no
- Size of Municipalities and Quality of Websites
Small is Beautiful or Bigger is Better?
- by Svein Ølnes, Western Norway Research Institute (Vestlandsforsking)
- ECEG 2013, Como, Italy

2. www.vestforsk.no
Bigger is Better!

3. www.vestforsk.no
Content
 Main question answered
 Benchmarking systems
 Method
 Results
 Benchmarkings: Whose quality are we measuring?

4. www.vestforsk.no
Main question answered
 Does quality of municpality websites vary with size?
 Which underlying factors can explain the observed variation?
 Whose quality are we really measuring?

5. www.vestforsk.no
Benchmarking systems
 Norwegian benchmarking system from 2001
 all public websites
 yearly evaluations (expert evaluations)
 a single set of indicators
 Denmark: Also since 2001
 tried different methods (single set of ind., multiple sets, self eval.
 gradually moved to more end user involvement
 2012 benchmarking accessibility screening + user survey

6. www.vestforsk.no
Benchmarking systems
 EU’s benchmarking system
 Conceived at the Lisbon conference i 2000
 Until 2012 based on a set of 20 services
• 12 services for citizens
• 8 services for businesses
 As the “e-readiness” approached 100 % when measuring the digital
availability of the 20 services, new approaches had to be found

7. www.vestforsk.no
Method
1. Results from the Norwegian benchmarkings of public
websites (municpality part) 2007 – 2011
2. A survey to web administrators (of the same websites as
above)
3. A broad citizen survey conducted in 2009/2010

8. www.vestforsk.no
Norwegian benchmarking system 2007-2011
2007 2011
No. of ind. Max score No. of ind. Max score
1. Accessibility 11 27 11 28
2. Usability 14 37 12 30
3. Useful services 7 28 10 35
Total 32 92 32 93

9. www.vestforsk.no
Norwegian benchmarking system (2013)
1. Accessibility
 13 indicators, max score 36 points
 Example: Ind. 1.11 – Correct use of HTML
2. Usability
 8 indicators, max score 28 points
 Ind. 2.2 – The website is functional also with mobile devices
3. Useful services
 9 indicators, max score 39 points
 Example: Ind. 3.7 – The website has published open data

10. www.vestforsk.no
Results
2007 20011 Change
Small municipalities (< 5.000) 46,6 58,4 11,8
Medium sized mun. (5.000-20.000) 50,7 66,1 15,4
Large municipalities (> 20.000) 58,4 72,0 13,6
Average, all municipalities 49,5 62,7 13,2

11. www.vestforsk.no
What do the webmasters say?
 What is your knowledge of the quality evaluation system?
Small Medium Large
No knowledge/little knowledge 36 % 15 % 3 %
Good/Very good knowledge 59 % 85 % 97 %

12. www.vestforsk.no
What do the webmasters say?
 How easy or difficult is it to understand the quality indicators?
Small Medium Large
Difficult/Very difficult 37 % 34 % 13 %
Easy/Very easy 58 % 66 % 87 %

13. www.vestforsk.no
What do the people say about the quality
of public services?
 A survey to 30.000 citizens above 18 years
 Satisfaction with public services (12.659 asked, 6.646 answers)
 Results:
 No significant difference in perceived quality of municipality services in
relation to size of municipality
 Planning and building services and elderly care got higher scores from
citizens of small municipalities
 For services like kindergarten and primary school larger municipalities
scored better than smaller
 Is it Jakob Nielsen’s first rule of usability?
 “Don’t listen to the users, whatch them work!”

14. www.vestforsk.no
Whose quality are we measuring?
 Better quality on public websites and digital public services
is the objective
 But whose quality?
 The end user (the citizen) is too far away and end user
quality is not enough taken into account
 Benchmarking systems like the Norwegian need to be more
user oriented
 Usability testing and user surveys should be key parts in
developing benchmarking systems

15. www.vestforsk.no
End of presentation
Thanks for your attention!
Svein Ølnes – sol@vestforsk.no
This presentation: http://j.mp/d6WX2G