How was it for you? Evaluating the iSkillZone. Bisset & Brain
•Download as PPTX, PDF•
1 like•293 views
Presented at LILAC 2010
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to How was it for you? Evaluating the iSkillZone. Bisset & Brain
Similar to How was it for you? Evaluating the iSkillZone. Bisset & Brain (20)
More from IL Group (CILIP Information Literacy Group)
More from IL Group (CILIP Information Literacy Group) (20)
Recently uploaded
Recently uploaded (20)
How was it for you? Evaluating the iSkillZone. Bisset & Brain
- 1. LILAC 2010 How was it for you? Evaluating the iSkillZone Presentation by James Bisset & Sarah Brain Library Services, University of the West of England
- 2. University of the West of England • Post-1992 university • Five campuses in and around Bristol • 29,300 students • Over 600 courses offered • Partnerships
- 3. Graduate Development Programme “To make academic and personal development, the learning of skills and employability explicit in order to improve the quality of the student experience through retention, progression and achievement.” University of the West of England (2008). Staff Guide to the Graduate Development Programme, p8.
- 4. Graduate Development Programme • Level 1: Sense of belonging and the development of learning skills • Level 2: Employability and personal development planning • Level 3: Academic achievement and progression to employment or further study
- 5. What is the iSkillZone? • Launched by UWE Library Services Sept 2008 • Ten Information Literacy units http://www.uwe.ac.uk/library/
- 6. Why evaluate? • Impact on Graduate Development Programme • Impact on students • Ease of use and navigation • Inform future developments and direction
- 8. Overall usage Total page views Jan-Dec 2009: 481,408
- 9. Statistics
- 11. How and why? • Usability testing of students - Is the source easy to navigate? - Do users find what they want, where they expect to find it? • Academic staff interviews - How is it being used in teaching? - What is working, what could be improved? • Questionnaires for staff and students - How is it being used and how is it being received by all users?
- 13. • 5 students participated • Set 7 tasks to find information held on site. • Morae software records entire session. • Pre- & post questionnaires to gauge experience of student and impressions afterwards. Usability testing
- 14. Usability testing 0 5 10 15 20 25 Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 NumberofMouseclicks iSkillZone Evaluation - Student Usability Testing Mouse clicks for each task Minimum clicks S1 S2 S3 S4 S5
- 15. Usability testing– preliminary findings • Liked the colours, lay-out and interactivity • Less text • More images, bullet points and interactive quizzes • Less jargon & more use of ‘student language’ • Need better search facility within the resource • ‘Basic’ & ‘Advanced’ versions • Would use in future – especially for ‘Referencing’, ‘Plagiarism’ and ‘Finding journal articles’
- 16. Questionnaires
- 17. Questionnaires Staff Questionnaire (June 2009) – 161 respondents Student Questionnaire (April 2009) – 493 respondents Library Staff Questionnaire (June 2009) – 39 respondents
- 18. Questionnaires Staff Questionnaire 59 out of 161 had used in teaching sessions (37%) Student Questionnaire: 193 out of 493 had used in teaching sessions (39%) Library Staff Questionnaire: 29 out of 39 had used in teaching / service points (74%)
- 19. Usage by section: Academic Staff 0 10 20 30 40 50 60 70 80 90 100 Percentage of staff responses
- 20. Usage by section: Staff & Students 0 10 20 30 40 50 60 70 80 90 100 Percentage of staff responses Percentage of student responses
- 21. Staff responses How has the iSkillZone been received?:- • Q14 asked respondents if they planned to use in the future? • 100% of respondents who had used the iSkillZone planned to use it again. • 60% of respondents who had not previously used it in teaching indicated that they planned to in the future.
- 22. 0 10 20 30 40 50 60 70 80 Recommend to students Link to from Blackboard Embed Quizzes in teaching materials GDP Level 1 GDP Level 2 GDP Level 3 No of responses Intended future useage
- 23. Questionnaires Conclusions from staff questionnaire Conclusions from student questionnaire Improve publicity and expand introduction to the iSkillZone’s purpose. Need better guidance on what it is and why/if it is important. Staff who use the iSkillZone generally think it is a good resource and plan to continue using it in the future. Majority of students who use it like it. Possibly offer alternative ‘context’ with no tutorials and just quick links for use to support teaching. Students introduced via structured sessions rate it more highly, get more out of it & more likely to use in future. Navigation needs to be improved. Navigation on referencing section needs to be improved.
- 24. Questionnaires Conclusions from library staff questionnaire Journal articles section possibly needs a redesign or different context options. Reduce ‘wordiness’ of sections. Consult and provide training on sections appropriate for service desk usage. Navigation needs to be improved plus the ability to search across the site.
- 25. Questionnaires “The online quizzes are very useful in GDP sessions – the Harvard referencing system one is particularly good.” “...great resource and mostly very intuitive” “I will definitely use it in next year’s teaching” “Too wordy” “I have found it so helpful – thank you!” “Its fun, user- friendly and informative!” “We need a better understanding of what it is and what it offers” “The backwards navigation needs to be sorted out”“Needs clearer and quicker links”
- 26. Questionnaires “What do students want?” “The assumption of this questionnaire is that skills are taught – this is not always the case. I would encourage students to use the iSkillZone to facilitate their own learning.” “We would like to work more with you with regards an academic skills module” “...[use a similar format] to support other generic skills development, e.g. writing skills, presentation skills, career development etc.” “Continue to respond to student needs when requested.”
- 28. • Six members of staff interviewed in June 2009 • Business, Land and property management, Film studies, Cultural studies, Humanities, Equine science •Purpose: identify impact of iSkillZone in GDP and other teaching Academic Staff Interviews
- 29. • Impact on GDP • Impact on other teaching • What works well • What doesn’t work well • Improvements Questions
- 30. Impact on GDP and other teaching • Four respondents used the iSkillZone • Key sections used in GDP: ‘Referencing’, ‘Journals’, ‘Plagiarism’ • Key sections used in other teaching: ‘Literature searching’, ‘Evaluating resources’ • Varied level of use • Meets the required needs
- 31. What works well • Content • Navigation and usability • Appearance
- 32. What doesn’t work well • Navigation • Terminology • Level
- 33. Improvements • Introduce a range of levels from basic to advanced •Incorporation of study skills •Increase interactivity
- 34. Conclusions
- 35. Some key findings •Staff who use the iSkillZone generally think it is a good resource, find it suits most of their needs for GDP and plan to continue using it in the future. • Still room for improvement and a need for promotion and publicity. • Introduction to students in structured teaching sessions seems to improve students’ usage and response to the iSkillZone. • Need to meet varying needs, ranging from quick links to more interactive quizzes and tutorials.
- 36. Some key findings •Significant demand for study skills resources to be incorporated into the iSkillZone. • Need to address appropriateness, given the current focus of the iSkillZone and study skills materials already available. • Various navigation issues need to be addressed (e.g. search function, lay-out, wordiness). • Reassess and clarify guidance and terminology used throughout the iSkillZone.
- 37. How has evaluation helped? • Identified some key navigation issues to be addressed. • Helped prioritise which areas we need to develop / redesign (what is used most, what people like the least, what people want to see added). • Additional publicity and promotion.
- 38. Changes already implemented… • Navigation on Referencing section redesigned. • ‘Evaluate’ and ‘Information on the Net’ sections redesigned as the new ‘Evaluating Information’ section.
- 39. The Future… • Plans in development to engage with ‘Study Skills’ provision at UWE and look at embedding resources already developed, in collaboration with academic staff. • Identifying and prioritising future developments from the findings of the evaluation process.
Editor's Notes
- Post-1992 univ: Bristol is in the south west of England. Based in four campuses across Bristol 29,300 students: of which 6,200 PG Over 600 courses: Largest provider of HE in the South West of England Partnerships: courses run at many local FE colleges and internationally with other universities across the world.
- GDP introduced in September 2007 as a means of enhancing the student experience whilst at UWE and also equipping students with skills that are not necessarily taught during their academic study. GDP sits outside of their normal course and is not compulsory. The over-arching aim of the programme is given above (read out).
- The GDP is different for each UG study level (read out levels from slide). At level 1, the development of learning skills includes the development of effective info literacy skills. Therefore, level 1 is where the library inputs the most, and is one of the main reaosns why we developed the iSkillZone.
- iSkillZone is a suite of 10 individual units, some focussing specifically on how to use the library at UWE, and some focussing on more general IL skills. There is a combination of instruction, self-assessment and interactive quizzes. Open access – anyone can use it. Idea – for GDP tutors/librarians to use the units as part of their GDP sessions on IL/study skills. Encourage staff to link to it from their VLE (Blackboard) modules. Also designed for the individual self-directed learner. Demo iSkillZone, following sections: 1. Getting started, test yourself quiz. There are fifty questions, divided into the ten different themes covered. Answer first five questions to show feedback given for each and then the measurement for how many were correct. 2. Finding items in the library – Understanding the numbering system. 3. Journal articles – Journal or Journal Article. 4. Referencing – go through flowchart. Goes hand in hand with plagiarism section. 5. Evaluating information – Book or ebook
- When launched in September 2008, we had already decided that we would carry out an evaluation in 6 months time. One of the main reasons was that we wanted to measure the impact of iSkillZone on the level 1 GDP teaching – were academic staff and librarians using it as part of this? We applied for some funding from the GDP project team and were successful in securing this. Second reason – to find out if students were using it and the impact it had made on their information seeking behaviour. We also wanted to find out how easy it is to actually navigate from the non-librarian’s point of view. Finally, we wanted to carry out the testing and surveys in order to identify any improvements that should be made, and to help inform the future direction of iSkillZone.
- A preliminary note – we were able to look at some usage statistics since carrying out the evaluation and thought it would be useful to share at this point. Shows the number of visits in 2009 and highlights the trends in usage. Moderate usage first part of year, tailing off in summer, peaking in November. Tied in with both the autumn term and our publicity.
- Also able to gain info on the most popular pages – Referencing was one of the most popular – we would expect this. Although other unites have been well-used as well, such as Getting started, Who owns it (now called copyright) and finding journal articles. Whilst useful to have these statistics, the results of the study were able to tell us a bit more about wht they had been used.
- Split up key aims of each evaluation technique. User testing – monetary incentives offered to students. 8 volunteered, 5 turned up. Academic staff questionnaires voluntary, but targeted via GDP coordinators in each faculty (lent some weight to importance to encourage participation). Questionnaires – emphasise 3 different questionnaires. To cover same questions as interviews, but with a broader scope. Interviews offer a more detailed insight into specific exam[ples of use across different faculties.
- Three were native English speakers (S1, S3, S5) Three had used iSkillZone previously (S1, S2, S4) Three had attended GDP sessions (S1, S3, S5) All reckoned they adequate/good IT skills. Sample size 5 – the ‘magic number’ http://www.stcsig.org/usability/newsletter/0301-number.html 80% of info’ on looking at a website you can get from only testing out 5 users. What is involved in usability testing: user performs a series of tasks on the website without interference but is asked to talk out loud as doing each task we also ask them to do a pre-task and post-task questionnaire we set SEVEN tasks, the first task for example was …….starting from the UWE homepage and using the iSkillZone find how to reference a chapter from a book using the UWE Harvard referencing system a further task was to allow them look at any part of the iSkillZone which interested them and comment £10 cash incentive up to one hour duration use of TechSmith Morae software…………………..videos the session/ records all keystrokes /all mouse clicks/ the URL of all web pages visited in order/ timings for how long on a page and between mouse clicks for example. Provides a detailed record of the usability testing session.
- Also mention it recorded time of each task, pages visited, and videoed entire session. Task 1 : Finding how to reference a chapter from a book (Harvard system) Max clicks 12 Min 4 Task 2: Database tutorial ……………. max 14 and min 3 Task 3: Find ‘Search strategy checklist’ ……………………..max 3 min 1 Task 4: Find definition of ‘collusion’ ………………………………max 13 min 8 Task 5: ‘Where do you draw the line?’ interactive activity ……………………………max 20 min 1 Task 6: Copyright activity 1 ‘Scenario 11: BBC iPlayer ’ …………………………………..max 9 min 3 Task 7: Copyright activity 2 ‘Scenario 13: Maps’ ……………………………..max 4 min 1
- Also mention it recorded time of each task, pages visited, and videoed entire session. Task 1 : Finding how to reference a chapter from a book (Harvard system) Max clicks 12 Min 4 Task 2: Database tutorial ……………. max 14 and min 3 Task 3: Find ‘Search strategy checklist’ ……………………..max 3 min 1 Task 4: Find definition of ‘collusion’ ………………………………max 13 min 8 Task 5: ‘Where do you draw the line?’ interactive activity ……………………………max 20 min 1 Task 6: Copyright activity 1 ‘Scenario 11: BBC iPlayer ’ …………………………………..max 9 min 3 Task 7: Copyright activity 2 ‘Scenario 13: Maps’ ……………………………..max 4 min 1
- Summarise Key information sought:- Level of usage across UWE Usage as part of the GDP Which sections were being used How was it being received Recommendations for improvement to guide future development.
- But of those who have HEARD OF the iSkillZone:- - Academic Staff, 59/110 (53.64%)
- Skip over – questionnaires allowed us to identify sections most heavily used by staff/students at different levels and by different faculty. PREPARE FOR QUESTIONS!
- Skip over – questionnaires allowed us to identify sections most heavily used by staff/students at different levels and by different faculty. PREPARE FOR QUESTIONS!
- Questionnaires also allowed us to identify how staff/students in tended to use in the future – very useful in how we then choose to develop the resource (e.g. Which sections focus on, how do we make it accessible, promotion etc.) Q13 also asked to rate the iSkillZone VG to VP, but results were flawed in that not all responses were received for this question. Those received showed over 90% though Good.Very Good.
- Summarise Key information sought:- Level of usage across UWE Usage as part of the GDP Which sections were being used How was it being received Recommendations for improvement to guide future development.
- Summarise Key information sought:- Level of usage across UWE Usage as part of the GDP Which sections were being used How was it being received Recommendations for improvement to guide future development.
- Summarise Key information sought:- Level of usage across UWE Usage as part of the GDP Which sections were being used How was it being received Recommendations for improvement to guide future development.
- Summarise Key information sought:- Level of usage across UWE Usage as part of the GDP Which sections were being used How was it being received Recommendations for improvement to guide future development.
- Impact on GDP: the extent to which it was used as part of GDP sessions Impact on other teaching: the extent to which it was used as part of teaching that is carried out on degree programmes and modules (i.e. subject sessions).
- Four of the six staff interviewed had use the iSkillZone as part of their GDP or other teaching. Key sections: a marked difference between GDP and other teaching in terms of which sections had been used the most. What this does show is that staff are using a wide spread of the modules, which shows that the breadth of what it offers is being utilised. Vaied level of use – ranged from merely recommending it to students, to creating links to specific units from within Blackboard, to actually using individual units with students during a session. For example, making use of the Harvard and Plagiarism quizzes in class. Of the four members of staff that use it, all of them said that it met their needs in terms of what they required it for/what they needed it to do.
- Content: all of the respondents were positive about the range of topic areas covered on the iSkillZone. In particular, they highlighted the plagiarism, referencing and getting started sections as being three of the most useful sections. Five out of the six commented that they particularly liked the interactive elements and quizzes. Navigation: In general they thought that it was easy to use, and that the navigation was fairly straight-forward. They also commented that it is easy to find from the library homepage. In terms of appearance, they commented that it was colourful and engaging, and again, that they liked the interactivity.
- Navigation: even though they had listed this as a positive, the respondents also had a few negative comments about specific aspects. Four of them had found it difficult to locate the info they were looking for and at least one felt tat the quizzes were too buried. There was also a comment about the way in which the info is presented and that this means that there is often some repetition. There was also a technical issue raised in that one person had experienced overlapping of text on their PC when using iSkillZone. Terminology: This was the second main area for concern. Feedback on this included the fact that the terminology used was sometimes difficult to understand. There was also a feeling that some of the sections were too wordy. Also – there was some feedback about the name – that it isn’t very descriptive of what it actually does. Level: Several of the respondents were not convinced that it was pitched at the right level to meet their students’ needs. For example, there was a comment that the plagiarism was too basic.
- Two themes came through strongly (which are the first two listed): Four respondents suggested that there could be a range of skill levels within the iSkillZone, from basic to advanced. This could perhaps tie in with the different UG study levels and what might be expected of them in terms of the ability to effectively search for and evaluate information. This was also linked to a further recommendation suggesting that we identify targets for students to aim towards. E.g. by the end of your first year you should be able to do X, Y and Z. The second main strand was the desire to see iSkillZone extended beyond library skills into study skills. This was mentioned by three of the respondents and aspects such as essay writing, note taking and time management were among those mentioned. The final recommendation of note was to increase the level of interactivity in terms of quizzes, etc.
- Skim over ‘key findings’. Emphasise the benefits of evaluation process:-