SlideShare ist ein Scribd-Unternehmen logo
1 von 7
Downloaden Sie, um offline zu lesen
LONG TERM EVALUATION OF INFORMATION LITERACY
PROGRAMME
Tina Hohmann
Technische Universität München, Germany, tina.hohmann@tum.de
Abstract
The Technische Universität München library was the first university library in Germany to be
certified by TÜV with respect to its ISO 9001:2001 quality management. One of the main
objectives of our quality management policy is to measure customer satisfaction not only in the
short-term, but also over longer periods of time, in order to constantly develop and improve our
services.
To this end, TUM library management have decided to conduct a long term evaluation of its
information literacy (IL) programme. We regularly ask course participants for feedback
immediately after the events and evaluate these yearly. Additionally, we have started to survey
long term changes in awareness and sustainability of course content. The library’s IL programme
has been in place for over three years, and has thus generated sufficient data for reliable
evaluation.
The main object of our investigation is the assessment of effectiveness of our IL programme and
further guidance as to how we can improve our services.
We have been evaluating about 100 IL events, which took place between 6 and 24 months ago.
Our study includes the four main types of workshops, which comprise basic and advanced
literature search, referencing, and reference management. An online survey contains questions
regarding the significance and the application of the skills taught during the courses. We also ask
about any identified lack of resources or learning goals, and our e-learning material. The first
phase of the evaluation was conducted between February and March 2014.
This paper presents our findings from the evaluation project, our conclusions, and
recommendations for the further development of the IL programme.
Keywords
Information Literacy, quality management, long-term evaluation
Background / Motivation
The Technische Universität München (TUM) library was the first university library in Germany to
be certified by TÜV with respect to its ISO 9001:2001 quality management.
One of the main objectives of our quality management policies is to measure customer
satisfaction in order to constantly develop and improve our services. We have conducted surveys
among students and academic staff in the past, but these were more generalised.
In terms of our Information Literacy (IL) programme we decided to introduce evaluations over
longer periods of time to complement the existing short term evaluation.
Immediately after IL training events we give out feedback forms to participants to measure their
attitudes to the event. Participants are asked to rate the relevance of the individual topics, to
evaluate teaching methods and leave general comments. These provide immediate feedback to
the instructor. Once a year course coordinators compile and analyse the data and disseminate
the results among the team of instructors.
Additionally, we decided to evaluate the relevance of the training content and its impact for
participants in the long-term. How sustainable are the skills acquired in the training and which
ones would participants apply in their current studies?
As library instruction at TUM is usually one off, there is limited opportunity for formative
evaluation. We agreed on a survey among participants using self-assessment. To produce
comparable results we limited the events to our standard IL programme. As this has been in place
for over three years so it has generated sufficient data.
Aims and Objectives
1. To find out if the workshops make a difference for former participants and evaluating long
term effectiveness of the training content
2. To find out if there are significant differences between short-term and long-term
evaluation
3. Suggestions (and inspirations) for improvements
We are aware of the limitations of the chosen method as explained by Schilling and Applegate
(2012). Self-report surveys do not measure skills or knowledge, but give an idea of students´
perceptions and attitudes.
Subject of investigation and timeframe
We decided to survey the four main types of workshops of our standard programme, excluding
any embedded training.
The introductory library workshop entitled “Get ready for your studies” comprises literature search
for books and journals, interlibrary loan and internet search. The advanced library workshop “Get
ready for your degree” includes developing a search strategy, finding the right database for one's
subject, searching in databases and referencing. Further we evaluated the basic and advanced
workshop on the reference management software Citavi. In total we have evaluated 96 events.
The participants were mainly students, although there are also a number of staff who attended
the reference management workshops.
The timeframe investigated covered the period between January 2012 to July 2013, which at the
time of the survey was between 6 and 24 months ago.
The survey
The survey was conducted between February and March 2014 as an online survey using the
TUM licenced software Evasys. Due to data protection1,
we were only able to contact less than
half of the participants, which came to a total of 690 emails.
It was open for 2 weeks.
The survey was prepared in consultation with a librarian adept in surveys and adhering to the
main rules for surveys established by Porst (2009).
All questions were created with the team of the course leaders and discussed several times. The
survey specialist at the university provided us with helpful advice on methodology and technical
issues to do with the software. After a pretest, we slightly changed a few questions.
The questionnaires comprised topics, which were course-specific as well as generic questions.
We had 12-14 scaled and multiple choice questions and five open-ended text questions.
The questionnaire was divided into five sections. In the first section, the main objectives and
terms of the survey were explained, and the participants had to indicate their academic status
and faculty. In the second section, we asked participants to assess the significance of the
different topics taught in the course in retrospect. These were scaled and correlated to the
questions on the feedback form. They were followed by two open-ended questions for comments
and any suggestions for additional content. The third section included a range of questions on
self-assessing skills and behavior, self-perception and confidence regarding tools and skills we
had identified as learning goals. Again, there was an opportunity to leave comments. The forth
section focused on participants´ attending other workshops and the awareness of our e-Learning-
material. In the final section we asked participants which topics they found most useful and if they
had any suggestions for improvement.
The courses were separated into three surveys containing workshops from each half-year period.
The reason for this was that we wanted to find out if the return rate would decrease over time.
The overall return rate of our survey was 23%; in sum, and 156 questionnaires to analyse.
Though these don’t represent the total of all participants, we still think the results are significant
and meet our objectives. We decided to omit the results of the advanced reference management
workshop as the total number of responses was only 12. For the remaining workshops, we
received between 41 and 63 responses each - 144 questionnaires in total.
The results - findings
The return rate was similar across all workshops. Equally, the assumption of a lower return rate
for the workshops in the first half year 2012 did not prove to be true. However, we received some
comments mentioning difficulty in remembering details as the training had taken place a long time
ago.
In the following, I will focus on the results that met (or failed to meet) our objectives…
1. indicating that the workshops do make a difference for the participants
2. showing significant difference to the feedback we get immediately after the workshops
3. Outcomes that are significant or surprising for us
1
In line with data protection regulations we were allowed to address institutional email-accounts only
4. and lead to any conclusions and/or further action on our IL programme
1. Do the workshops make a difference for the participants?
71% (introductory workshop) and 83% (advanced workshop) of the participants acknowledge that
their research skills have improved since they participated in the workshop. Over all course types
and over all topics, a clear majority of participants rated the content as very important (51%) or
important (25%).
In the comments section we received quite a few statements about the positive effect of the
workshops, e.g.
“Great presentation and helpful instructor! This event has supported me to write a great
dissertation2
“This workshop was part of my Master’s degree studies. However, the skills I have acquired in
this workshop would have been very helpful for my Bachelor’s degree studies…”3
“This was by far the best training at TUM and I received the top grade for my dissertation! The
instructor, Ms… was very good! This is a most boring topic, but the training was very
interesting! TOP!”4
To find out if specific topics were seen as more relevant than others, we asked
“Which topics have you found most useful in retrospect?”
Almost a third (29%) of the participants used the opportunity to leave their opinion at this point.
The comments included a wide range of topics that were part of the training sessions. The only
topic that stood out was bibliographic databases. Participants in the advanced library workshop
emphasised their importance for their literature search much above average.
2. Differences short-term and long-term feedback
Topic Internet Search
Contrary to the feedback we get immediately after the workshops, where participants in the
introductory library workshop are not sure about the importance of internet searches we had a
consistent high evaluation on this in the long term. On average, only 61% rated this topic as
important in the short-term. In the long-term term, 87% rated this topic as being very important
(67%) or important (20%).
On the other hand, more than 34% of participants said that they are not able to find relevant
sources on the Internet. At first, this seems rather contradictory.
This could indicate that we do not teach the right content, but we did not receive any comments
that would support this presumption. We interpret these numbers as indicating that being more
knowledgeable in this area, participants assessed their own skills more critically than before.
Topic Referencing
On the short-term feedback forms on the advanced library workshop, only 57% of participants
rated referencing as important. Additionally, we regularly received feedback about this section as
2
„Toller Vortrag und eine hilfsbereite Referentin! Diese Veranstaltung hat mir sehr geholfen, eine tolle
Abschlussarbeit zu schreiben.“ (all translations from German by Tina Hohmann)
3
„Dieser Kurs wurde als Pflichttermin im zuge meines Masterstudiums absolviert. Schon während des
Bachelorstudiums hätte mir das in diesem Kurs erlangte Wissen sehr geholfen…“
4
„Es war mit abstand der beste kurs an der tum und ich habe auf meine abschlussarbeit eine 1.0
bekommen! Die Dozentin frau … war sehr gut! Es ist ein langweiliges nerviges thema aber der kurs
war sehr interessant! TOP!“
being too extensive. From the current results, it seems this topic is valued more highly over time.
In retrospect more than 75% rated this topic as very important (57%) or important (18%). Also, it
was mentioned by a few participants as an area where they would have liked even more input.
Nevertheless, 40% of participants do not feel confident in this area. Individual comments indicate
that there is great uncertainty about non-existent regulations at the university. Equally we receive
mixed feedback from instructors that often reflects their own insecurity.
“More about correct referencing, please (esp. in connection with the recent plagiarism
scandals) – whereas collaboration with faculties is needed in order to develop a standard
referencing style across the university or at least the faculties. Currently each chair seems to
have its own referencing style…”5
3. Significant or surprising results
Reference management software Citavi
A relatively high number of participants (30%) of the Citavi workshop indicated that they are
currently not using Citavi on a regular basis. At least in some cases this was not their own
decision as explained in the comments:
“Citavi is great – unfortunately, where I did my Bachelor’s placement, everybody was using
EndNote. Inevitably I had to do the same…”6
“At present not regularly using Citavi as I had to switch to Endnote at the request of my
current supervisor”7
These comments indicate that EndNote is widely-used among TUM staff and industry, which
leads to students switching software. Therefore, we can see the usage statistics for Citavi in a
different light.
Awareness of the IL programme across university
There were a few comments mentioning the lack of awareness across the university and
suggesting embedding the workshops as compulsory components in the curriculum.
“A lot of students, even in their last year, are not aware of the range of resources on offer.
Everybody should attend this workshop.”8
“…Both of your workshops were very helpful. All students from semester 5 should attend
these.”9
“… I had no idea (about the workshops) and none of the lecturers has mentioned them.”10
“Maybe you should do more to advertise the fact that you offer workshops regularly and that
they are not time-consuming. I know quite a few other students, who would have benefitted
from the workshop, but are not aware that you run them… How about a !short! email about the
training programme to the students reps.”11
5
„Etwas mehr über korrekte Zitierweise (speziell im Zusammenhang mit den Plagiatsskandalen der
letzten Zeit) - wobei dabei eine Zusammenarbeit mit den Fakultäten nötig ist, um einheitliche Zitierstile
für die gesamte TUM oder zumindest für einzelne Fakultäten zu erarbeiten. Momentan scheint jeder
Lehrstuhl einen eigenen Zitierstil zu haben…“
6
„Citavi ist toll - leider benutzten bei meiner Bachelorarbeitsstelle alle EndNote. Ich dann zwangsläufig
auch ...“
7
„zur Zeit keine regelmäßige Nutzung mehr von Citavi, da auf Wunsch meines jetztigen Betreuers der
Masterarbeit Wechsel zu Endnote“
8
„Viele Studenten, selbst kurz vor der Abschlußarbeit, kennen das Rechercheangebot der TUM nicht.
Dieser Kurs sollte von jedem besucht werden.“
9
„…Ihre Recherchekurse waren beide sehr hilfreich und sollten meiner Meinung nach ab dem 5.
Semester von jedem Studenten besucht werden.“
10
“… Ich wusste jedoch nicht das er angeboten wird bzw. keiner der Dozenten hat darauf hingewiesen.“
Other topics…
Many students (27%) are not aware that they cannot find journal articles in the library catalogue.
This is not very surprising and might be connected to the fact that some students criticised OPAC
as not being very user-friendly.
“The OPAC catalogue should be revised fundamentally, esp. getting more user-friendly, no
wonder that no one wants to deal with it12
More than 63% of participants said that they do not use different search tools, such as
Truncation. Obviously, this is a very high number, but we realise it might be due to the fact that
the question was not clear. Most students are probably not familiar with the term “Truncation” and
therefore answered no.
Among the results from the Citavi workshop, there are two areas where we received results that
show a similar pattern, but might have different causes. A relatively high proportion of
participants (27%) were undecided about whether generating bibliographies is important, and
another 13% did not answer the question. However, a clear majority (85%) stated that they know
how it works. Apparently, this is a function that is very easy to use, so participants might have
known about it beforehand or regard is as easy to learn, and therefore do not rate it as being as
important as other topics. Similarly, 23% of participants were undecided about the importance of
exporting from databases, and 21% thought it not important. On the other hand, 82% know how it
works. This might indicate the same as before, but it is a rather complex tool, and from immediate
feedback, we know that there are students who do not use databases for their studies. In this
case, we assume that although participants are familiar with the function are not using it in the
long-term.
Participants in the Citavi workshop rated two areas above average: Using Citavi in LateX and
how to use Citavi on several computers.
Implications for our IL programme
There is a clear need to raise awareness of our IL programme across the university. Currently we
use a mix of tools to promote our services. In a brainstorming meeting with staff from marketing,
we will be looking into more student-focused and unconventional ways of promoting our services,
and as suggested use student reps or the student union as disseminators. At the same time, we
plan to contact faculties whose students are less represented at the workshops to promote our
services. Ideally, IL sessions would be embedded in their curriculum.
The results demonstrate that training on internet search is very important to our users. We will
continue teaching it, and acknowledge the need for keeping up to date in this area.
We have recently started developing a paper on referencing, which we seek to agree with the
TUM ombudsperson for the academic code of practice. The aim is to publish a set of generic
guidelines for the faculties at TUM and to encourage faculties to announce generic referencing
guidelines. At the same time, we are preparing an online tutorial/set of elearning materials on
referencing for students to refer to any time.
11
„Vielleicht sollten Sie noch etwas mehr Werbung machen, die beinhaltet, dass es die Kurse regelmäßig
gibt und dass sie nur kurz dauern. Ich kenne einige Kommilitonen, denen der Besuch mehr geholfen
hätte als mir, die aber von dem Kurs gar nichts wussten... Gut wäre z.B. eine !kurze! Rundmail mit dem
Kursangebot zu Semesterbeginn an die Fachschaftsverteiler.“
12
„OPAC Katalog müsste grundlegend überarbeitet werden, und vorallem Studentenfreundlicher werden,
kein Wunder das sich niemand effektiv damit auseinander setzt“
Generating bibliographies has been reduced to the main steps in the current version of the Citavi
workshop. We see no need to change the content regarding exporting from databases and we
see this function as an essential tool for academic writing.
As the integration of LateX has been requested in short term evaluation in past years, this has
since been included in the Citavi advanced workshop. To enable using Citavi on several
computers smoothly, a web-based version would be necessary, which is not in our hands, but the
software developer is aware of the issue.
Lessons learned and further action
For the future, we plan to repeat long-term surveys on our IL programme on a regular basis. Next
time we would like to include other course types such as the workshops for PhD candidates and
for researchers. In order to be able to address more participants, we will include a declaration of
consent regarding contacting participants on the course list / on the registration form.
In order to test skills, we are considering including knowledge tests for participants, but that would
involve pre-tests and testing a control group, which is usually not possible because of the nature
of the workshops.
The pretest for the survey should include more non-library related people to avoid
misunderstandings in the questionnaire.
Conclusion
The long-term evaluation of our IL programme helps to understand students’ attitudes and
perceptions regarding our instructional trainings. It complements the range of efforts to measure
customer satisfaction for TUM library and thus adds a valuable element to our quality
management programme.
We see the results as evidence that our workshops do make a difference for the students. The
mainly positive feedback shows that we are on the right track. We might use some of the results –
esp. from the comments - to raise awareness across the university and to promote the integration
of IL workshops into the curriculum at the faculties. In this respect, it may help strengthen our
position within the university.
We were able to find out more about the long-term effect of our trainings, received suggestions
for improvement, and will adapt content according to our users’ needs.
References
Porst, R. (2009): Fragebogen. Ein Arbeitsbuch (2nd ed.). Wiesbaden, Germany: VS Verlag für
Sozialwissenschaften.
Schilling, K. & Applegate, R. (2012) Best methods for evaluating educational impact: A
comparison of the efficacy of commonly used measures of library instruction. Journal of the
Medical Library Association, 100 (4), 264

Weitere ähnliche Inhalte

Was ist angesagt?

Evaluation models
Evaluation modelsEvaluation models
Evaluation modelsMaarriyyaa
 
Programme Evaluation in Open and Distance Learning
Programme Evaluation in Open and Distance LearningProgramme Evaluation in Open and Distance Learning
Programme Evaluation in Open and Distance LearningSanjaya Mishra
 
Interrogating evaluation 2015 inductionb
Interrogating evaluation 2015 inductionbInterrogating evaluation 2015 inductionb
Interrogating evaluation 2015 inductionbRita Ndagire Kizito
 
Paemst review rubric
Paemst review rubricPaemst review rubric
Paemst review rubricscsumatic
 
Measurment 1 course outline
Measurment 1 course outline Measurment 1 course outline
Measurment 1 course outline Est chua
 
teaching material
teaching material teaching material
teaching material Kadek Astiti
 
Designing and Conducting Formative Evaluation
Designing and Conducting Formative EvaluationDesigning and Conducting Formative Evaluation
Designing and Conducting Formative EvaluationAngel Jones
 
Program Evaluation
Program EvaluationProgram Evaluation
Program EvaluationBurhan Omar
 
CIPP Evaluation Model
CIPP Evaluation ModelCIPP Evaluation Model
CIPP Evaluation ModelCt Hajar
 
Creating Assessments
Creating AssessmentsCreating Assessments
Creating AssessmentsChristina Sax
 
A Paper on Practices of Quality Assurance in Open and Distance Learning: A C...
A Paper on Practices of Quality Assurance in Open and Distance Learning: A  C...A Paper on Practices of Quality Assurance in Open and Distance Learning: A  C...
A Paper on Practices of Quality Assurance in Open and Distance Learning: A C...African Virtual University
 
Christine suh friday
Christine suh fridayChristine suh friday
Christine suh fridayECCSymposium
 
Classroom assessment techniques
Classroom assessment techniquesClassroom assessment techniques
Classroom assessment techniquesHINDUJACON
 
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...EDEN Digital Learning Europe
 
e-Learning Planning Framework review process
e-Learning Planning Framework review processe-Learning Planning Framework review process
e-Learning Planning Framework review processcowieNZ
 
A methodology to design customized learning networks
A methodology to design customized learning networksA methodology to design customized learning networks
A methodology to design customized learning networksHendrik Drachsler
 
Blueprint of exam questions
Blueprint of exam questions Blueprint of exam questions
Blueprint of exam questions Dr.Shahid Hassan
 

Was ist angesagt? (20)

Evaluation models
Evaluation modelsEvaluation models
Evaluation models
 
Programme Evaluation in Open and Distance Learning
Programme Evaluation in Open and Distance LearningProgramme Evaluation in Open and Distance Learning
Programme Evaluation in Open and Distance Learning
 
Interrogating evaluation 2015 inductionb
Interrogating evaluation 2015 inductionbInterrogating evaluation 2015 inductionb
Interrogating evaluation 2015 inductionb
 
Paemst review rubric
Paemst review rubricPaemst review rubric
Paemst review rubric
 
Measurment 1 course outline
Measurment 1 course outline Measurment 1 course outline
Measurment 1 course outline
 
teaching material
teaching material teaching material
teaching material
 
Designing and Conducting Formative Evaluation
Designing and Conducting Formative EvaluationDesigning and Conducting Formative Evaluation
Designing and Conducting Formative Evaluation
 
Program Evaluation
Program EvaluationProgram Evaluation
Program Evaluation
 
CHEST Information Architecture for Live Learning Courses 2016
CHEST Information Architecture for Live Learning Courses 2016CHEST Information Architecture for Live Learning Courses 2016
CHEST Information Architecture for Live Learning Courses 2016
 
CIPP Evaluation Model
CIPP Evaluation ModelCIPP Evaluation Model
CIPP Evaluation Model
 
Creating Assessments
Creating AssessmentsCreating Assessments
Creating Assessments
 
A Paper on Practices of Quality Assurance in Open and Distance Learning: A C...
A Paper on Practices of Quality Assurance in Open and Distance Learning: A  C...A Paper on Practices of Quality Assurance in Open and Distance Learning: A  C...
A Paper on Practices of Quality Assurance in Open and Distance Learning: A C...
 
Christine suh friday
Christine suh fridayChristine suh friday
Christine suh friday
 
Classroom assessment techniques
Classroom assessment techniquesClassroom assessment techniques
Classroom assessment techniques
 
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
 
e-Learning Planning Framework review process
e-Learning Planning Framework review processe-Learning Planning Framework review process
e-Learning Planning Framework review process
 
Functions of evaluation
Functions of evaluationFunctions of evaluation
Functions of evaluation
 
A methodology to design customized learning networks
A methodology to design customized learning networksA methodology to design customized learning networks
A methodology to design customized learning networks
 
Blueprint of exam questions
Blueprint of exam questions Blueprint of exam questions
Blueprint of exam questions
 
Methodology
MethodologyMethodology
Methodology
 

Andere mochten auch

Presentation Teknisa
Presentation TeknisaPresentation Teknisa
Presentation Teknisaguestf98a87
 
Improving line management capability the Grimsby Institute story by Peter J B...
Improving line management capability the Grimsby Institute story by Peter J B...Improving line management capability the Grimsby Institute story by Peter J B...
Improving line management capability the Grimsby Institute story by Peter J B...Acas Comms
 
Analizador sintáctico de Pascal escrito en Bison
Analizador sintáctico de Pascal escrito en BisonAnalizador sintáctico de Pascal escrito en Bison
Analizador sintáctico de Pascal escrito en BisonEgdares Futch H.
 
Creditmanagement en cloud computing
Creditmanagement en cloud computingCreditmanagement en cloud computing
Creditmanagement en cloud computingPiet van Vugt
 
Passivhuse: Udfordringer og muligheder
Passivhuse: Udfordringer og mulighederPassivhuse: Udfordringer og muligheder
Passivhuse: Udfordringer og mulighederBertel Bolt-Jørgensen
 
Luxury presentation
Luxury presentationLuxury presentation
Luxury presentationlmeneley
 
Getting results when working with english result
Getting results when working with english resultGetting results when working with english result
Getting results when working with english resultemege68
 
3 Hr. Workbook - S1031 For Professionals
3 Hr.  Workbook - S1031 For Professionals3 Hr.  Workbook - S1031 For Professionals
3 Hr. Workbook - S1031 For ProfessionalsEdmund_Wheeler
 
Ekonomik i̇stikrar
Ekonomik i̇stikrarEkonomik i̇stikrar
Ekonomik i̇stikrarsadettin
 
Organic fertilization and microbial dynamics
Organic fertilization and microbial dynamicsOrganic fertilization and microbial dynamics
Organic fertilization and microbial dynamicsjuveultra
 
NABE Communication Section Workshop - Event Information & Registration
NABE Communication Section Workshop - Event Information & RegistrationNABE Communication Section Workshop - Event Information & Registration
NABE Communication Section Workshop - Event Information & RegistrationStephanie Abbott
 
Bloggingforbusiness2
Bloggingforbusiness2Bloggingforbusiness2
Bloggingforbusiness2Andre Kleist
 
Actions You Can Take In Volatile Market Linkedin
Actions You Can Take In Volatile Market LinkedinActions You Can Take In Volatile Market Linkedin
Actions You Can Take In Volatile Market Linkedingoldglove41
 

Andere mochten auch (20)

Coke
CokeCoke
Coke
 
Hazed and Confused
Hazed and ConfusedHazed and Confused
Hazed and Confused
 
Wordshop Web Evolution (by Morozov Andrey)
Wordshop Web Evolution (by Morozov Andrey)Wordshop Web Evolution (by Morozov Andrey)
Wordshop Web Evolution (by Morozov Andrey)
 
Presentation Teknisa
Presentation TeknisaPresentation Teknisa
Presentation Teknisa
 
Improving line management capability the Grimsby Institute story by Peter J B...
Improving line management capability the Grimsby Institute story by Peter J B...Improving line management capability the Grimsby Institute story by Peter J B...
Improving line management capability the Grimsby Institute story by Peter J B...
 
Analizador sintáctico de Pascal escrito en Bison
Analizador sintáctico de Pascal escrito en BisonAnalizador sintáctico de Pascal escrito en Bison
Analizador sintáctico de Pascal escrito en Bison
 
Creditmanagement en cloud computing
Creditmanagement en cloud computingCreditmanagement en cloud computing
Creditmanagement en cloud computing
 
Passivhuse: Udfordringer og muligheder
Passivhuse: Udfordringer og mulighederPassivhuse: Udfordringer og muligheder
Passivhuse: Udfordringer og muligheder
 
Luxury presentation
Luxury presentationLuxury presentation
Luxury presentation
 
Morsø erhversråd energimærkning
Morsø erhversråd   energimærkningMorsø erhversråd   energimærkning
Morsø erhversråd energimærkning
 
Getting results when working with english result
Getting results when working with english resultGetting results when working with english result
Getting results when working with english result
 
3 Hr. Workbook - S1031 For Professionals
3 Hr.  Workbook - S1031 For Professionals3 Hr.  Workbook - S1031 For Professionals
3 Hr. Workbook - S1031 For Professionals
 
2014 Workers' Compensation Seminar
2014 Workers' Compensation Seminar2014 Workers' Compensation Seminar
2014 Workers' Compensation Seminar
 
Ekonomik i̇stikrar
Ekonomik i̇stikrarEkonomik i̇stikrar
Ekonomik i̇stikrar
 
Organic fertilization and microbial dynamics
Organic fertilization and microbial dynamicsOrganic fertilization and microbial dynamics
Organic fertilization and microbial dynamics
 
NABE Communication Section Workshop - Event Information & Registration
NABE Communication Section Workshop - Event Information & RegistrationNABE Communication Section Workshop - Event Information & Registration
NABE Communication Section Workshop - Event Information & Registration
 
Cap Editing
Cap EditingCap Editing
Cap Editing
 
Presentation1
Presentation1Presentation1
Presentation1
 
Bloggingforbusiness2
Bloggingforbusiness2Bloggingforbusiness2
Bloggingforbusiness2
 
Actions You Can Take In Volatile Market Linkedin
Actions You Can Take In Volatile Market LinkedinActions You Can Take In Volatile Market Linkedin
Actions You Can Take In Volatile Market Linkedin
 

Ähnlich wie Long term evaluation of IL programme paper

Week 12_Designing Instructional Materials and Conducting Summative Evaluation...
Week 12_Designing Instructional Materials and Conducting Summative Evaluation...Week 12_Designing Instructional Materials and Conducting Summative Evaluation...
Week 12_Designing Instructional Materials and Conducting Summative Evaluation...RalphNavelino3
 
Curriculum Evaluation BY Ahmet YUSUF
Curriculum  Evaluation BY Ahmet YUSUFCurriculum  Evaluation BY Ahmet YUSUF
Curriculum Evaluation BY Ahmet YUSUFأحمد يوسف
 
Evaluation Of A Program Evaluation Essay
Evaluation Of A Program Evaluation EssayEvaluation Of A Program Evaluation Essay
Evaluation Of A Program Evaluation EssayHelp With A Paper Provo
 
Assessment Institute August 21 2008
Assessment Institute August 21 2008Assessment Institute August 21 2008
Assessment Institute August 21 2008middlesex
 
Tig 3 piloting curriculum(1)
Tig 3 piloting curriculum(1)Tig 3 piloting curriculum(1)
Tig 3 piloting curriculum(1)geraldine mendoza
 
Monitoring and Supervising Curriculum & Curriculum Revision and Evaluation
Monitoring and Supervising Curriculum & Curriculum Revision and EvaluationMonitoring and Supervising Curriculum & Curriculum Revision and Evaluation
Monitoring and Supervising Curriculum & Curriculum Revision and EvaluationShaharyarShoukatShou
 
6. international-pen-guidance-notes-for-centres monitoring-and-evaluation
6. international-pen-guidance-notes-for-centres monitoring-and-evaluation6. international-pen-guidance-notes-for-centres monitoring-and-evaluation
6. international-pen-guidance-notes-for-centres monitoring-and-evaluationFolasayo Ayegbayo
 
Assessment Matters Newsletter_November 2015 (3)
Assessment Matters Newsletter_November 2015 (3)Assessment Matters Newsletter_November 2015 (3)
Assessment Matters Newsletter_November 2015 (3)Tom Kohntopp
 
programme evaluation by priyadarshinee pradhan
programme evaluation by priyadarshinee pradhanprogramme evaluation by priyadarshinee pradhan
programme evaluation by priyadarshinee pradhanPriya Das
 
Companion Materials Looking Through the Lens of Rubrics
Companion Materials Looking Through the Lens of RubricsCompanion Materials Looking Through the Lens of Rubrics
Companion Materials Looking Through the Lens of Rubricsanniesyso
 
Silvana Richardson: Weighing the Pig Doesn't Make it Fatter or Does it
Silvana Richardson: Weighing the Pig Doesn't Make it Fatter or Does itSilvana Richardson: Weighing the Pig Doesn't Make it Fatter or Does it
Silvana Richardson: Weighing the Pig Doesn't Make it Fatter or Does iteaquals
 
Assessment System And Model Is Based On The Following...
Assessment System And Model Is Based On The Following...Assessment System And Model Is Based On The Following...
Assessment System And Model Is Based On The Following...Caitlin Wilson
 
Online Assessment and Feedback Seminar
Online Assessment and Feedback SeminarOnline Assessment and Feedback Seminar
Online Assessment and Feedback SeminarJulian Green
 
Administration and Supervision in Evaluation
Administration and Supervision in EvaluationAdministration and Supervision in Evaluation
Administration and Supervision in EvaluationSharon Geroquia
 
Outcomnes-based Education
Outcomnes-based EducationOutcomnes-based Education
Outcomnes-based EducationCarlo Magno
 
Assessing for Improvement: learning outcomes assessment for library instruction
Assessing for Improvement: learning outcomes assessment for library instructionAssessing for Improvement: learning outcomes assessment for library instruction
Assessing for Improvement: learning outcomes assessment for library instructionDiane Harvey
 

Ähnlich wie Long term evaluation of IL programme paper (20)

Week 12_Designing Instructional Materials and Conducting Summative Evaluation...
Week 12_Designing Instructional Materials and Conducting Summative Evaluation...Week 12_Designing Instructional Materials and Conducting Summative Evaluation...
Week 12_Designing Instructional Materials and Conducting Summative Evaluation...
 
Curriculum Evaluation BY Ahmet YUSUF
Curriculum  Evaluation BY Ahmet YUSUFCurriculum  Evaluation BY Ahmet YUSUF
Curriculum Evaluation BY Ahmet YUSUF
 
Evaluation Of A Program Evaluation Essay
Evaluation Of A Program Evaluation EssayEvaluation Of A Program Evaluation Essay
Evaluation Of A Program Evaluation Essay
 
Assessment Institute August 21 2008
Assessment Institute August 21 2008Assessment Institute August 21 2008
Assessment Institute August 21 2008
 
Tig 3 piloting curriculum(1)
Tig 3 piloting curriculum(1)Tig 3 piloting curriculum(1)
Tig 3 piloting curriculum(1)
 
Assessment (11)
Assessment (11)Assessment (11)
Assessment (11)
 
Monitoring and Supervising Curriculum & Curriculum Revision and Evaluation
Monitoring and Supervising Curriculum & Curriculum Revision and EvaluationMonitoring and Supervising Curriculum & Curriculum Revision and Evaluation
Monitoring and Supervising Curriculum & Curriculum Revision and Evaluation
 
6. international-pen-guidance-notes-for-centres monitoring-and-evaluation
6. international-pen-guidance-notes-for-centres monitoring-and-evaluation6. international-pen-guidance-notes-for-centres monitoring-and-evaluation
6. international-pen-guidance-notes-for-centres monitoring-and-evaluation
 
Assessment Matters Newsletter_November 2015 (3)
Assessment Matters Newsletter_November 2015 (3)Assessment Matters Newsletter_November 2015 (3)
Assessment Matters Newsletter_November 2015 (3)
 
programme evaluation by priyadarshinee pradhan
programme evaluation by priyadarshinee pradhanprogramme evaluation by priyadarshinee pradhan
programme evaluation by priyadarshinee pradhan
 
Companion Materials Looking Through the Lens of Rubrics
Companion Materials Looking Through the Lens of RubricsCompanion Materials Looking Through the Lens of Rubrics
Companion Materials Looking Through the Lens of Rubrics
 
Silvana Richardson: Weighing the Pig Doesn't Make it Fatter or Does it
Silvana Richardson: Weighing the Pig Doesn't Make it Fatter or Does itSilvana Richardson: Weighing the Pig Doesn't Make it Fatter or Does it
Silvana Richardson: Weighing the Pig Doesn't Make it Fatter or Does it
 
Module Evaluation Survey
Module Evaluation SurveyModule Evaluation Survey
Module Evaluation Survey
 
Measuring learning outcomes
Measuring learning outcomesMeasuring learning outcomes
Measuring learning outcomes
 
Assessment System And Model Is Based On The Following...
Assessment System And Model Is Based On The Following...Assessment System And Model Is Based On The Following...
Assessment System And Model Is Based On The Following...
 
ARLG 2019: Myer have we made a difference
ARLG 2019: Myer have we made a differenceARLG 2019: Myer have we made a difference
ARLG 2019: Myer have we made a difference
 
Online Assessment and Feedback Seminar
Online Assessment and Feedback SeminarOnline Assessment and Feedback Seminar
Online Assessment and Feedback Seminar
 
Administration and Supervision in Evaluation
Administration and Supervision in EvaluationAdministration and Supervision in Evaluation
Administration and Supervision in Evaluation
 
Outcomnes-based Education
Outcomnes-based EducationOutcomnes-based Education
Outcomnes-based Education
 
Assessing for Improvement: learning outcomes assessment for library instruction
Assessing for Improvement: learning outcomes assessment for library instructionAssessing for Improvement: learning outcomes assessment for library instruction
Assessing for Improvement: learning outcomes assessment for library instruction
 

Mehr von Tina Hohmann

IK Kurs Bibliometrie Current Awareness Networking
IK Kurs Bibliometrie Current Awareness NetworkingIK Kurs Bibliometrie Current Awareness Networking
IK Kurs Bibliometrie Current Awareness NetworkingTina Hohmann
 
Long term evaluation of IL programme slides
Long term evaluation of IL programme slidesLong term evaluation of IL programme slides
Long term evaluation of IL programme slidesTina Hohmann
 
Sixth formers at the University Library
Sixth formers at the University LibrarySixth formers at the University Library
Sixth formers at the University LibraryTina Hohmann
 
Information Skills - Targeting untapped user groups - text
Information Skills - Targeting untapped user groups - textInformation Skills - Targeting untapped user groups - text
Information Skills - Targeting untapped user groups - textTina Hohmann
 
Information Skills – Targeting Untapped User Groups
Information Skills – Targeting Untapped User GroupsInformation Skills – Targeting Untapped User Groups
Information Skills – Targeting Untapped User GroupsTina Hohmann
 
Hohmann liber2006text
Hohmann liber2006textHohmann liber2006text
Hohmann liber2006textTina Hohmann
 
Hohmann Learning spaces Warwick english
Hohmann Learning spaces Warwick englishHohmann Learning spaces Warwick english
Hohmann Learning spaces Warwick englishTina Hohmann
 
Learning und Research Grids Hohmann deutsch
Learning und Research Grids Hohmann deutschLearning und Research Grids Hohmann deutsch
Learning und Research Grids Hohmann deutschTina Hohmann
 
Planungs- und Bewertungskriterien fuer Bibliotheksbauten
Planungs- und Bewertungskriterien fuer BibliotheksbautenPlanungs- und Bewertungskriterien fuer Bibliotheksbauten
Planungs- und Bewertungskriterien fuer BibliotheksbautenTina Hohmann
 

Mehr von Tina Hohmann (10)

IK Kurs Bibliometrie Current Awareness Networking
IK Kurs Bibliometrie Current Awareness NetworkingIK Kurs Bibliometrie Current Awareness Networking
IK Kurs Bibliometrie Current Awareness Networking
 
Long term evaluation of IL programme slides
Long term evaluation of IL programme slidesLong term evaluation of IL programme slides
Long term evaluation of IL programme slides
 
Sixth formers at the University Library
Sixth formers at the University LibrarySixth formers at the University Library
Sixth formers at the University Library
 
Information Skills - Targeting untapped user groups - text
Information Skills - Targeting untapped user groups - textInformation Skills - Targeting untapped user groups - text
Information Skills - Targeting untapped user groups - text
 
Information Skills – Targeting Untapped User Groups
Information Skills – Targeting Untapped User GroupsInformation Skills – Targeting Untapped User Groups
Information Skills – Targeting Untapped User Groups
 
Hohmann liber2006
Hohmann liber2006Hohmann liber2006
Hohmann liber2006
 
Hohmann liber2006text
Hohmann liber2006textHohmann liber2006text
Hohmann liber2006text
 
Hohmann Learning spaces Warwick english
Hohmann Learning spaces Warwick englishHohmann Learning spaces Warwick english
Hohmann Learning spaces Warwick english
 
Learning und Research Grids Hohmann deutsch
Learning und Research Grids Hohmann deutschLearning und Research Grids Hohmann deutsch
Learning und Research Grids Hohmann deutsch
 
Planungs- und Bewertungskriterien fuer Bibliotheksbauten
Planungs- und Bewertungskriterien fuer BibliotheksbautenPlanungs- und Bewertungskriterien fuer Bibliotheksbauten
Planungs- und Bewertungskriterien fuer Bibliotheksbauten
 

Kürzlich hochgeladen

Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...raviapr7
 
The Singapore Teaching Practice document
The Singapore Teaching Practice documentThe Singapore Teaching Practice document
The Singapore Teaching Practice documentXsasf Sfdfasd
 
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfMaximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfTechSoup
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfMohonDas
 
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptxClinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptxraviapr7
 
How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17Celine George
 
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptxPISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptxEduSkills OECD
 
Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.EnglishCEIPdeSigeiro
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxDr. Asif Anas
 
General views of Histopathology and step
General views of Histopathology and stepGeneral views of Histopathology and step
General views of Histopathology and stepobaje godwin sunday
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesCeline George
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfMohonDas
 
How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17Celine George
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?TechSoup
 
Presentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a ParagraphPresentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a ParagraphNetziValdelomar1
 
UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE
 
How to Add a many2many Relational Field in Odoo 17
How to Add a many2many Relational Field in Odoo 17How to Add a many2many Relational Field in Odoo 17
How to Add a many2many Relational Field in Odoo 17Celine George
 
5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...CaraSkikne1
 

Kürzlich hochgeladen (20)

Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...
 
The Singapore Teaching Practice document
The Singapore Teaching Practice documentThe Singapore Teaching Practice document
The Singapore Teaching Practice document
 
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfMaximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdf
 
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptxClinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
 
How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17
 
Finals of Kant get Marx 2.0 : a general politics quiz
Finals of Kant get Marx 2.0 : a general politics quizFinals of Kant get Marx 2.0 : a general politics quiz
Finals of Kant get Marx 2.0 : a general politics quiz
 
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptxPISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
 
Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptx
 
Prelims of Kant get Marx 2.0: a general politics quiz
Prelims of Kant get Marx 2.0: a general politics quizPrelims of Kant get Marx 2.0: a general politics quiz
Prelims of Kant get Marx 2.0: a general politics quiz
 
General views of Histopathology and step
General views of Histopathology and stepGeneral views of Histopathology and step
General views of Histopathology and step
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 Sales
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdf
 
How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?
 
Presentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a ParagraphPresentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a Paragraph
 
UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024
 
How to Add a many2many Relational Field in Odoo 17
How to Add a many2many Relational Field in Odoo 17How to Add a many2many Relational Field in Odoo 17
How to Add a many2many Relational Field in Odoo 17
 
5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...
 

Long term evaluation of IL programme paper

  • 1. LONG TERM EVALUATION OF INFORMATION LITERACY PROGRAMME Tina Hohmann Technische Universität München, Germany, tina.hohmann@tum.de Abstract The Technische Universität München library was the first university library in Germany to be certified by TÜV with respect to its ISO 9001:2001 quality management. One of the main objectives of our quality management policy is to measure customer satisfaction not only in the short-term, but also over longer periods of time, in order to constantly develop and improve our services. To this end, TUM library management have decided to conduct a long term evaluation of its information literacy (IL) programme. We regularly ask course participants for feedback immediately after the events and evaluate these yearly. Additionally, we have started to survey long term changes in awareness and sustainability of course content. The library’s IL programme has been in place for over three years, and has thus generated sufficient data for reliable evaluation. The main object of our investigation is the assessment of effectiveness of our IL programme and further guidance as to how we can improve our services. We have been evaluating about 100 IL events, which took place between 6 and 24 months ago. Our study includes the four main types of workshops, which comprise basic and advanced literature search, referencing, and reference management. An online survey contains questions regarding the significance and the application of the skills taught during the courses. We also ask about any identified lack of resources or learning goals, and our e-learning material. The first phase of the evaluation was conducted between February and March 2014. This paper presents our findings from the evaluation project, our conclusions, and recommendations for the further development of the IL programme. Keywords Information Literacy, quality management, long-term evaluation
  • 2. Background / Motivation The Technische Universität München (TUM) library was the first university library in Germany to be certified by TÜV with respect to its ISO 9001:2001 quality management. One of the main objectives of our quality management policies is to measure customer satisfaction in order to constantly develop and improve our services. We have conducted surveys among students and academic staff in the past, but these were more generalised. In terms of our Information Literacy (IL) programme we decided to introduce evaluations over longer periods of time to complement the existing short term evaluation. Immediately after IL training events we give out feedback forms to participants to measure their attitudes to the event. Participants are asked to rate the relevance of the individual topics, to evaluate teaching methods and leave general comments. These provide immediate feedback to the instructor. Once a year course coordinators compile and analyse the data and disseminate the results among the team of instructors. Additionally, we decided to evaluate the relevance of the training content and its impact for participants in the long-term. How sustainable are the skills acquired in the training and which ones would participants apply in their current studies? As library instruction at TUM is usually one off, there is limited opportunity for formative evaluation. We agreed on a survey among participants using self-assessment. To produce comparable results we limited the events to our standard IL programme. As this has been in place for over three years so it has generated sufficient data. Aims and Objectives 1. To find out if the workshops make a difference for former participants and evaluating long term effectiveness of the training content 2. To find out if there are significant differences between short-term and long-term evaluation 3. Suggestions (and inspirations) for improvements We are aware of the limitations of the chosen method as explained by Schilling and Applegate (2012). Self-report surveys do not measure skills or knowledge, but give an idea of students´ perceptions and attitudes. Subject of investigation and timeframe We decided to survey the four main types of workshops of our standard programme, excluding any embedded training. The introductory library workshop entitled “Get ready for your studies” comprises literature search for books and journals, interlibrary loan and internet search. The advanced library workshop “Get ready for your degree” includes developing a search strategy, finding the right database for one's subject, searching in databases and referencing. Further we evaluated the basic and advanced workshop on the reference management software Citavi. In total we have evaluated 96 events. The participants were mainly students, although there are also a number of staff who attended the reference management workshops. The timeframe investigated covered the period between January 2012 to July 2013, which at the time of the survey was between 6 and 24 months ago.
  • 3. The survey The survey was conducted between February and March 2014 as an online survey using the TUM licenced software Evasys. Due to data protection1, we were only able to contact less than half of the participants, which came to a total of 690 emails. It was open for 2 weeks. The survey was prepared in consultation with a librarian adept in surveys and adhering to the main rules for surveys established by Porst (2009). All questions were created with the team of the course leaders and discussed several times. The survey specialist at the university provided us with helpful advice on methodology and technical issues to do with the software. After a pretest, we slightly changed a few questions. The questionnaires comprised topics, which were course-specific as well as generic questions. We had 12-14 scaled and multiple choice questions and five open-ended text questions. The questionnaire was divided into five sections. In the first section, the main objectives and terms of the survey were explained, and the participants had to indicate their academic status and faculty. In the second section, we asked participants to assess the significance of the different topics taught in the course in retrospect. These were scaled and correlated to the questions on the feedback form. They were followed by two open-ended questions for comments and any suggestions for additional content. The third section included a range of questions on self-assessing skills and behavior, self-perception and confidence regarding tools and skills we had identified as learning goals. Again, there was an opportunity to leave comments. The forth section focused on participants´ attending other workshops and the awareness of our e-Learning- material. In the final section we asked participants which topics they found most useful and if they had any suggestions for improvement. The courses were separated into three surveys containing workshops from each half-year period. The reason for this was that we wanted to find out if the return rate would decrease over time. The overall return rate of our survey was 23%; in sum, and 156 questionnaires to analyse. Though these don’t represent the total of all participants, we still think the results are significant and meet our objectives. We decided to omit the results of the advanced reference management workshop as the total number of responses was only 12. For the remaining workshops, we received between 41 and 63 responses each - 144 questionnaires in total. The results - findings The return rate was similar across all workshops. Equally, the assumption of a lower return rate for the workshops in the first half year 2012 did not prove to be true. However, we received some comments mentioning difficulty in remembering details as the training had taken place a long time ago. In the following, I will focus on the results that met (or failed to meet) our objectives… 1. indicating that the workshops do make a difference for the participants 2. showing significant difference to the feedback we get immediately after the workshops 3. Outcomes that are significant or surprising for us 1 In line with data protection regulations we were allowed to address institutional email-accounts only
  • 4. 4. and lead to any conclusions and/or further action on our IL programme 1. Do the workshops make a difference for the participants? 71% (introductory workshop) and 83% (advanced workshop) of the participants acknowledge that their research skills have improved since they participated in the workshop. Over all course types and over all topics, a clear majority of participants rated the content as very important (51%) or important (25%). In the comments section we received quite a few statements about the positive effect of the workshops, e.g. “Great presentation and helpful instructor! This event has supported me to write a great dissertation2 “This workshop was part of my Master’s degree studies. However, the skills I have acquired in this workshop would have been very helpful for my Bachelor’s degree studies…”3 “This was by far the best training at TUM and I received the top grade for my dissertation! The instructor, Ms… was very good! This is a most boring topic, but the training was very interesting! TOP!”4 To find out if specific topics were seen as more relevant than others, we asked “Which topics have you found most useful in retrospect?” Almost a third (29%) of the participants used the opportunity to leave their opinion at this point. The comments included a wide range of topics that were part of the training sessions. The only topic that stood out was bibliographic databases. Participants in the advanced library workshop emphasised their importance for their literature search much above average. 2. Differences short-term and long-term feedback Topic Internet Search Contrary to the feedback we get immediately after the workshops, where participants in the introductory library workshop are not sure about the importance of internet searches we had a consistent high evaluation on this in the long term. On average, only 61% rated this topic as important in the short-term. In the long-term term, 87% rated this topic as being very important (67%) or important (20%). On the other hand, more than 34% of participants said that they are not able to find relevant sources on the Internet. At first, this seems rather contradictory. This could indicate that we do not teach the right content, but we did not receive any comments that would support this presumption. We interpret these numbers as indicating that being more knowledgeable in this area, participants assessed their own skills more critically than before. Topic Referencing On the short-term feedback forms on the advanced library workshop, only 57% of participants rated referencing as important. Additionally, we regularly received feedback about this section as 2 „Toller Vortrag und eine hilfsbereite Referentin! Diese Veranstaltung hat mir sehr geholfen, eine tolle Abschlussarbeit zu schreiben.“ (all translations from German by Tina Hohmann) 3 „Dieser Kurs wurde als Pflichttermin im zuge meines Masterstudiums absolviert. Schon während des Bachelorstudiums hätte mir das in diesem Kurs erlangte Wissen sehr geholfen…“ 4 „Es war mit abstand der beste kurs an der tum und ich habe auf meine abschlussarbeit eine 1.0 bekommen! Die Dozentin frau … war sehr gut! Es ist ein langweiliges nerviges thema aber der kurs war sehr interessant! TOP!“
  • 5. being too extensive. From the current results, it seems this topic is valued more highly over time. In retrospect more than 75% rated this topic as very important (57%) or important (18%). Also, it was mentioned by a few participants as an area where they would have liked even more input. Nevertheless, 40% of participants do not feel confident in this area. Individual comments indicate that there is great uncertainty about non-existent regulations at the university. Equally we receive mixed feedback from instructors that often reflects their own insecurity. “More about correct referencing, please (esp. in connection with the recent plagiarism scandals) – whereas collaboration with faculties is needed in order to develop a standard referencing style across the university or at least the faculties. Currently each chair seems to have its own referencing style…”5 3. Significant or surprising results Reference management software Citavi A relatively high number of participants (30%) of the Citavi workshop indicated that they are currently not using Citavi on a regular basis. At least in some cases this was not their own decision as explained in the comments: “Citavi is great – unfortunately, where I did my Bachelor’s placement, everybody was using EndNote. Inevitably I had to do the same…”6 “At present not regularly using Citavi as I had to switch to Endnote at the request of my current supervisor”7 These comments indicate that EndNote is widely-used among TUM staff and industry, which leads to students switching software. Therefore, we can see the usage statistics for Citavi in a different light. Awareness of the IL programme across university There were a few comments mentioning the lack of awareness across the university and suggesting embedding the workshops as compulsory components in the curriculum. “A lot of students, even in their last year, are not aware of the range of resources on offer. Everybody should attend this workshop.”8 “…Both of your workshops were very helpful. All students from semester 5 should attend these.”9 “… I had no idea (about the workshops) and none of the lecturers has mentioned them.”10 “Maybe you should do more to advertise the fact that you offer workshops regularly and that they are not time-consuming. I know quite a few other students, who would have benefitted from the workshop, but are not aware that you run them… How about a !short! email about the training programme to the students reps.”11 5 „Etwas mehr über korrekte Zitierweise (speziell im Zusammenhang mit den Plagiatsskandalen der letzten Zeit) - wobei dabei eine Zusammenarbeit mit den Fakultäten nötig ist, um einheitliche Zitierstile für die gesamte TUM oder zumindest für einzelne Fakultäten zu erarbeiten. Momentan scheint jeder Lehrstuhl einen eigenen Zitierstil zu haben…“ 6 „Citavi ist toll - leider benutzten bei meiner Bachelorarbeitsstelle alle EndNote. Ich dann zwangsläufig auch ...“ 7 „zur Zeit keine regelmäßige Nutzung mehr von Citavi, da auf Wunsch meines jetztigen Betreuers der Masterarbeit Wechsel zu Endnote“ 8 „Viele Studenten, selbst kurz vor der Abschlußarbeit, kennen das Rechercheangebot der TUM nicht. Dieser Kurs sollte von jedem besucht werden.“ 9 „…Ihre Recherchekurse waren beide sehr hilfreich und sollten meiner Meinung nach ab dem 5. Semester von jedem Studenten besucht werden.“ 10 “… Ich wusste jedoch nicht das er angeboten wird bzw. keiner der Dozenten hat darauf hingewiesen.“
  • 6. Other topics… Many students (27%) are not aware that they cannot find journal articles in the library catalogue. This is not very surprising and might be connected to the fact that some students criticised OPAC as not being very user-friendly. “The OPAC catalogue should be revised fundamentally, esp. getting more user-friendly, no wonder that no one wants to deal with it12 More than 63% of participants said that they do not use different search tools, such as Truncation. Obviously, this is a very high number, but we realise it might be due to the fact that the question was not clear. Most students are probably not familiar with the term “Truncation” and therefore answered no. Among the results from the Citavi workshop, there are two areas where we received results that show a similar pattern, but might have different causes. A relatively high proportion of participants (27%) were undecided about whether generating bibliographies is important, and another 13% did not answer the question. However, a clear majority (85%) stated that they know how it works. Apparently, this is a function that is very easy to use, so participants might have known about it beforehand or regard is as easy to learn, and therefore do not rate it as being as important as other topics. Similarly, 23% of participants were undecided about the importance of exporting from databases, and 21% thought it not important. On the other hand, 82% know how it works. This might indicate the same as before, but it is a rather complex tool, and from immediate feedback, we know that there are students who do not use databases for their studies. In this case, we assume that although participants are familiar with the function are not using it in the long-term. Participants in the Citavi workshop rated two areas above average: Using Citavi in LateX and how to use Citavi on several computers. Implications for our IL programme There is a clear need to raise awareness of our IL programme across the university. Currently we use a mix of tools to promote our services. In a brainstorming meeting with staff from marketing, we will be looking into more student-focused and unconventional ways of promoting our services, and as suggested use student reps or the student union as disseminators. At the same time, we plan to contact faculties whose students are less represented at the workshops to promote our services. Ideally, IL sessions would be embedded in their curriculum. The results demonstrate that training on internet search is very important to our users. We will continue teaching it, and acknowledge the need for keeping up to date in this area. We have recently started developing a paper on referencing, which we seek to agree with the TUM ombudsperson for the academic code of practice. The aim is to publish a set of generic guidelines for the faculties at TUM and to encourage faculties to announce generic referencing guidelines. At the same time, we are preparing an online tutorial/set of elearning materials on referencing for students to refer to any time. 11 „Vielleicht sollten Sie noch etwas mehr Werbung machen, die beinhaltet, dass es die Kurse regelmäßig gibt und dass sie nur kurz dauern. Ich kenne einige Kommilitonen, denen der Besuch mehr geholfen hätte als mir, die aber von dem Kurs gar nichts wussten... Gut wäre z.B. eine !kurze! Rundmail mit dem Kursangebot zu Semesterbeginn an die Fachschaftsverteiler.“ 12 „OPAC Katalog müsste grundlegend überarbeitet werden, und vorallem Studentenfreundlicher werden, kein Wunder das sich niemand effektiv damit auseinander setzt“
  • 7. Generating bibliographies has been reduced to the main steps in the current version of the Citavi workshop. We see no need to change the content regarding exporting from databases and we see this function as an essential tool for academic writing. As the integration of LateX has been requested in short term evaluation in past years, this has since been included in the Citavi advanced workshop. To enable using Citavi on several computers smoothly, a web-based version would be necessary, which is not in our hands, but the software developer is aware of the issue. Lessons learned and further action For the future, we plan to repeat long-term surveys on our IL programme on a regular basis. Next time we would like to include other course types such as the workshops for PhD candidates and for researchers. In order to be able to address more participants, we will include a declaration of consent regarding contacting participants on the course list / on the registration form. In order to test skills, we are considering including knowledge tests for participants, but that would involve pre-tests and testing a control group, which is usually not possible because of the nature of the workshops. The pretest for the survey should include more non-library related people to avoid misunderstandings in the questionnaire. Conclusion The long-term evaluation of our IL programme helps to understand students’ attitudes and perceptions regarding our instructional trainings. It complements the range of efforts to measure customer satisfaction for TUM library and thus adds a valuable element to our quality management programme. We see the results as evidence that our workshops do make a difference for the students. The mainly positive feedback shows that we are on the right track. We might use some of the results – esp. from the comments - to raise awareness across the university and to promote the integration of IL workshops into the curriculum at the faculties. In this respect, it may help strengthen our position within the university. We were able to find out more about the long-term effect of our trainings, received suggestions for improvement, and will adapt content according to our users’ needs. References Porst, R. (2009): Fragebogen. Ein Arbeitsbuch (2nd ed.). Wiesbaden, Germany: VS Verlag für Sozialwissenschaften. Schilling, K. & Applegate, R. (2012) Best methods for evaluating educational impact: A comparison of the efficacy of commonly used measures of library instruction. Journal of the Medical Library Association, 100 (4), 264