Assessing the impact of evidence summaries in library and information studies
1.
2. Lorie Kloda, MLIS, PhD(c), McGill University
Denise Koufogiannakis, MA, MLIS, PhD(c), University of Alberta
Alison Brettle, MLIS, PhD, University of Salford
Canadian Library Association Annual Conference, Ottawa, 2012
3.
4. Structured abstract
objective – design – setting – subjects –
method – main results – conclusion
Commentary
300-400 words
appraisal of validity, reliability, applicability
significance, implications for practice
4
9. Findings (Phase 1)
Development of
Impact
Assessment
Method
Grad, R., Pluye, P., &
Beauchamp, M.-E.
(2007). Validation of
a method to assess
the clinical impact of
electronic
knowledge
resources, e-Service
Journal, 5(2), 113-
135.
http://iam2009.pbworks.com/
11. Findings (Phase 2)
Survey
175 153 emailed 101
Respondents recruited IAM survey respondents
1 unusable 15
3 bounced
email incomplete
86
21 duplicates 49 no reply completed
IAM survey
153
remaining
101 total
respondents n=62
12. Findings (Phase 2)
USA
Country
Canada
UK
Australia
USA Ireland
Australia Finland
Malaysia
UK Saudi Arabia
Iran
Puerto Rico
Spain
Canada
Brazil
Hong Kong
13. Findings (Phase 2)
50+
25+
Number of 15
Evidence 14
13
Summaries 12
11
Read in Past 10
9
Year 8
7
6
5
4
3
2
1
0
0 2 4 6 8 10
No. of respondents
14. Decline in Reference Transactions with Few Questions Referred to
Librarian when the Reference Desk is Staffed by a Paraprofessional (8)
The Presence of Web 2.0 Applications Is Associated with the Overall
Service Quality of Library Websites (6)
Google Scholar Out-Performs Many Subscription Databases when
Keyword Searching (4)
Statistical Measures Alone Cannot Determine Which Database
(BNI, CINAHL, MEDLINE, or EMBASE) Is the Most Useful for Searching
Undergraduate Nursing Topic (4)
A Graduate Degree in Library or Information Science Is Required, but not
Sufficient, to Enter the Profession (3)
15. Findings (Phase 2)
Reason for Freq. %
Reading For general interest or curiosity 15 24%
Evidence
Summary For personal continuing 18 29%
professional education
n=62 To answer a specific question 21 34%
or address a specific issue in
my practice
Other 8 13%
16. Findings (Phase 2)
When the > 1 year
Evidence 6-12 months
Summary was
3-6 months
Read
1-3 months
1 week-1 month
1-7 days
Today
0 5 10 15 20 25
17. Findings (Phase 2)
Freq. %
Cognitive
My practice was (will be) improved 11 13%
Impact I learned something new 36 42%
I recalled something I already knew 14 16%
It prompted me to investigate more 23 27%
It confirmed I did (I am doing) the right thing 17 20%
I was reassured 13 15%
I was dissatisfied: There is a problem with the 1 1%
presentation of this evidence summary
I was dissatisfied: I disagree with the content 0 0%
of this evidence summary
It is potentially harmful 0 0%
Other 9 10%
18. Findings (Phase 2)
Practice Freq. %
Impact Change my service approach 5 6%
“You reported: Change my approach to collections 1 1%
My practice was
(will be) Change my management approach 4 5%
improved. What
Change my approach to teaching 4 5%
did you (will you)
do differently Change my professional approach 4 5%
after reading the
Evidence Other 3 3%
Summary?”
19. Findings (Phase 2)
Community None
Impact
Hypothesized future/potential
“If reading this
Evidence Summary impacts on users
resulted in some Reinforced cognitive or practice
change to your
individual practice, do
you think it led to an
impacts, not user outcomes
impact on anyone 5 reported actual impact at this
within the community
you serve or level:
environment in
which you work? Change in teaching LIS students
Please explain in the
comment box.” Observed (anecdotal) changes
21. Findings (Phase 3)
13 participants Subsample from survey
respondent
Various settings, education
levels, countries, and professional
positions
22. Proposed Impacts Suggested Confirmed
Phase 1 Phase 2 Phase 3
C1 Practice was improved ? ✗
C2 Learned something new ✔ ✔
C3 Recalled something ✔ ✓
C4 Prompted to investigate ✔ ✔
C5 Confirmed ✔ ✓
C6 Reassured ✔ ?
C7 Dissatisfied – presentation ? 0
C8 Dissatisfied - content 0 0
C9 Potentially harmful 0 0
P1 Service ✔ ✔
P2 Collections ✔ ✓
P3 Management ✔ ✓
P4 Teaching ✔ ✓
P5 Professional practice ✔ ✓
U User Community ? ?
23. Findings (Phase 3)
Potential Discovery
New research
Impacts Interesting topics
Methods
Uncovered Keeping current
Sharing
With colleagues, managers
Report writing
Recommended reading
Assistance
Research
Writing, presentations
Teaching (for professors)
24. Findings (Phase 3)
“Likes” Format of evidence summaries
Descriptive
Accuracy of abstract
Concise
Time saving
Database of Evidence Summaries
25. One evidence summary assessed per
respondent
Cognitive impact comparable to findings in
Grad, Pluye, et al. (2006)
Practice impact – two-tiered
Low community impact
26. How to best assess impact of reading about
research?
Do the items in the tool resonate with you?
How important is documenting the impact of
research on practice?
27. Revise Impact Assessment Tool
Include new or revised impact items
Improve wording of items
Conduct survey of a larger sample of evidence
summary readers
Adapt tool for other forms of research
dissemination
28. Canadian Association of Research Libraries
Research in Librarianship Grant
Roland Grad and Pierre Pluye, McGill University,
for their feedback
All of our survey respondents and interview
participants
Overview:Study background and purposeMethods and findings Discussion of findingsQuestions for the audience
Question to the crowd:Ask them to think about this throughout the talk, and we will raise the question again at the end of the talk with some time for discussion.(Some ideas of our own:We publish a journal, spend time getting ESs in there. We hope they are “used” or at least read, and if so, we think they are helpful. Curious to understand why they are used, how they are used, and what differences they make to the readers.
Ask if they are familiar already. Can flash through this and next slide quickly.Modeled after synopses found in the medical literature, such as those in APC journal club, or InfoPOEMs. There, clinicians can read structured abstracts of clinical research with commentaries, and sometimes a “bottom line” – or, answers the question: what does this change (or not) in my practice?
Develop a tool to assess impact, and validate its use with librarians for evidence summariesDetermine how and why evidence summaries are usedUnderstand how evidence summaries impact knowledge (cognition), practice, users
Phase 1: conducted with editorial team, librarians, authors of original IAM in medicine late fall and winter 2010-2011Phase 2: readers of EBLIP and others invited spring 2011Phase 3: subset of survey respondents fall and winter 2011-2012: Just completed. This is the first time the findings from the full study are being presentedCritical Incident Technique – the survey asks the respondent to identify one ES to assess impact. The interview will follow-up on the same incident. The impact of only ONE ES is being assessed per respondent.Interviews were designed to gather data to corroborate findings from survey respondents, by matching answers to determine if the tool actually captured “impacts” accurately, as well to uncover impacts or uses we had not considered.
Proposed impactsBegan with a tool from health sciences, used to assess impact of summaries delivered on a handheld device for GPs.Modified it, with consultation with librarians
McGill researchers in Family MedicineOriginal IAM tool, refined for different populations – this one is for Canadian Physicians to rate a resources known as eTherapuetics+Also mention JASIST articleWe came up with 3 areas of impact, with multiple items (specific impacts) in each:Librarian’s knowledge (cognition)Librarian’s practiceUser community
Using the tool developed in phase 1, we invited readers of EBLIP journal to participate. Those who expressed interest were invited to complete the survey in early spring of 2011 (i.e. a little over a year ago)
Recruitment from March-April 2011 (continued after that, 7 more recruits) [Survey invitations sent April 19, 2011 (EST)First reminder sent May 9, 2011Second reminder sent May 31, 2011Data collection ended June 3, 2011Data collection took place over 1.5 months] Response Rate: 56% (86 completed usable surveys out of 153 potential participants – self selected) Of 86, 62 have read an ES
Quick overview of the country. Closely parallels the readership of the journal.
Majority of respondents read between 1-10 or so a yearSome read many more. Some of these include our copyeditors.
25 Evidence Summaries were identified[9 did not remember or name an article, but still completed the survey2 named articles other than Ess]While not all about health, all are relevant to health sciences librarianship, and have broad appeal across settings.
Other: research, teaching, peer review for EBLIP, new position responsibilities
The majority had read the ES within the last 3 months
Other: share with others/colleagues; confirmed no/need to read original study; no impact because not relevant (title misleading); confirmatory
Other: impacted research method; impacted reader’s advisory service; general knowledgeSo, the “other” mostly fell into the existing category, with one new possible practice change: research approach (which isn’t exactly “practice” since it is a separate role for a librarian)
Out of 62 respondents, 41 added comments.Anecdotal changes: sensed that the institution saved money; prof reported students did better on course assignment as a result of librarian interventionNo formal assessment reported
We then invited a subset of respondents to talk more about the evidence summary.Thus far, we have interviewed 13 participants and thematically analyzed the interview transcripts. We discussed the same ES as the one used for answering the survey. In some cases there was a large time lapse since they had read the ES. This had advantages and distadvantagesOn the one hand, difficulty with recallOn the other hand, more time for impacts to be perceived
SettingAcademic9Health/hospital (not academic)1Teaching faculty (LIS or health)3 EducationPost-graduate diploma (after college)1MLIS or MA (or >1)9PhD3 CountryCanada6United States4United Kingdom1Australia1Hong Kong1PositionSupport staff1Librarian (non-management)6Manager3Professor3 2 participants were previously evidence summary writers; 1 copyeditor (and others peer reviewers)13/62 participants were interviewed. 21% of survey respondents who have read ESs.13/86 if you count all survey respondents (including those who did not read ESs) = 15%
Not sure what to do with this table….use final column for our “revised” impacts (not including new, suggestion possible impacts/uses). OR:, just include a check mark in the 2nd and 3rd columns to indicate these are confirmed?We could have aCheck markAnd X for a badly defined impactA 0 for a never recorded impact, but we still want to keep it because it is possibleAnd a ? For those that we are still unsure of
In the interviews, participants also discussed what they liked/disliked about the ESs in general, and what other potential impact or uses they perceived these could have on knowledge, practice, and users.Discovery was a theme which emerged describing specific areas of learningThere was also the concept of using information to influence one’s own or others’ decision-making. Not sure where this falls. Two uses: sharing, and DM.
Other advantages or themes which emerged from analysis of the interviews.Liked the format compared to the original articles.
Similar to Grad & Pluye et al’s findings with respect to positive cognitive impactsThe low “community impact” could be due to a misunderstanding of the question (the word “Community” could be interpreted as a change in workplace/colleagues), but possibly also likely due to:lack of measures to assess impact in library and information settingsEnvironmental barriers to assessing (EBP culture not always present)Delay in time to assess impact (unlike patient care, where it is one-on-one and more immediate)LIS is new to the concept of assessment of end users, it will take time for the culture to change and for these results to become observable/measurable.
elicit more detailed answers (corroborate survey findings) and determine why some impacts were not selected
(Also thank McGill for the support to conduct this research while on sabbatical).