How to Improve Your Library Instruction: Assessment in Five Minutes (February 2017)

1. How to Improve Your Library Instruction: Assessment in Five Minutes Sarah Steiner February 15, 2017
2. During Today’s Session, We Will… • Review the purpose and benefits of quick assessment in one-shot instruction sessions. • Draft learning outcomes for a session using Bloom’s Taxonomy (with an eye on threshold concepts). • Compose assessment instruments based on our learning outcomes. • Select appropriate assessment tools based on your instruction goals and population. http://www.flickr.com/photos/topgold/4858921901/
3. What Can Quick Classroom Assessment Do? • It can identify instructional gaps or disconnects. • It can help you determine how you spend class time. • It can build your confidence. • It can provide evidence of efficacy and impact. • It can inspire and motivate your learners. • It can (& should) be a basis for change. http://www.flickr.com/photos/sercasey/324341982/
4. What Should I Assess? • To assess: – Comprehension of integral concepts – Achievement of learning outcomes – Retention • Not to assess: – Your personal shortcomings – Your speaking mistakes – Your hairstyle http://www.flickr.com/photos/toniblay/52445415/
5. Have you ever written learning objectives or outcomes? • Nope, not yet. • Yes, one or two times. • Yes, regularly. • I’m not sure. http://www.flickr.com/photos/dadaistic_fantastic/5089525186/
6. Outcome Versus Objective https://flic.kr/p/57zSfN
7. Learning Outcomes “A Learning Outcome is a statement of what the student should understand and be able to do as a result of what she has learned ... ‘the essential and enduring knowledge, abilities, and attitudes or dispositions’ that enable a learner to practice and apply her learning in the real world.” -Valencia Community College http://www.flickr.com/photos/gblakeley/5583354276/
8. A Good Learning Outcome Will… • Identify the audience. • Set a time frame and a context. • Be jargon-free. • Be measurable. • Be action-oriented. • Be brief. • Be linked to learner needs. • Be appropriately narrow. http://www.flickr.com/photos/auntiep/4310267/
9. Determine the Class Priorities http://www.flickr.com/photos/paloetic/4795592340/ • Choose two to five. – What must the students accomplish? – What must they comprehend in order to accomplish it? – What is likely to trip them up? – What do they already know/find obvious?
10. Write the Learning Outcomes Stem + Verb + Product/Outcome http://www.flickr.com/photos/jjpacres/3293117576/
11. Stem By the end of this class, you will be able to… http://www.flickr.com/photos/47108884@N07/4594962925/
12. Verb: Bloom’s Taxonomy http://www.flickr.com/photos/tryingmyhardist/1366874433/
13. Choose Verbs: Knowledge Students will… define, describe, identify, know, label, list, match, name, outline, recall, recognize, reproduce, select, state http://www.nwlink.com/~donclark/hrd/bloom.html http://www.flickr.com/photos/reway2007/4693202935/
14. Choose Verbs: Comprehension Students will… comprehend, convert, defend, distinguish, estimate, explain, extend, generalize, give an example, infer, interpret, paraphrase, predict, rewrite, summarize, translate http://www.flickr.com/photos/bsmith4815/112307904/
15. Choose Verbs: Application Students will… apply, change, compute, construct, demonstrate, discover, manipulate, modify, operate, predict, prepare, produce, relate, show, solve, use http://www.flickr.com/photos/28478778@N05/5729009434/
16. Choose Verbs: Analysis Students will… analyze, break down, compare, contrast, diagram, deconstruct, differentiate, discriminate, distinguish, identify, illustrate, infer, outline, relate, select, separate http://www.flickr.com/photos/billselak/2709756134/
17. Choose Verbs: Synthesis Students will… categorize, combine, compile, compose, create, devise, design, explain, generate, modify, organize, plan, rearrange, reconstruct, relate, reorganize, revise, rewrite, summarize, tell, write http://www.flickr.com/photos/littleredelf/3081832532/
18. Choose Verbs: Evaluation Students will… appraise, compare, conclude, contrast, criticize, critique, defend, describe, discriminate, evaluate, explain, interpret, justify, relate, summarize, support http://www.flickr.com/photos/christinamatheson/108818379/
19. Add Product or Outcome What do the students need? • A thesis statement • Keyword search strategies (basic or advanced) • Understanding of the research process • Ability to critically analyze a source What is the context? • A class assignment • A real-world scenario http://www.flickr.com/photos/fdecomite/2907794565/
20. All Together Now! Stem + Verb + Product/Outcome By the end of this class, you will be ready to construct a usable and adaptable thesis statement for your class paper. This session will equip you with the tools you need to critically analyze a scholarly work. Today, we will explore methods that you can use to select the best resources for you through the article collection called PsycINFO. By the end of class, you will be able locate scholarly articles on your topic and will choose to seek help from your librarian, if you struggle. http://www.flickr.com/photos/revlimit/2186765034/
21. What’s Wrong Here? Today I’ll talk about… 1. The library website. 2. Database searching for peer-reviewed journals using boolean logic and the subject heading thesaurus. 3. Where you click to search the catalog, search the databases, and place an ILL request, and every step you’ll take to accomplish these processes. http://www.flickr.com/photos/mythoto/2604818537/
22. Question Time http://www.flickr.com/photos/sugarhiccuphiccup/5485691671/
23. Have you ever assessed student comprehension or learning in class? • Nope, not yet. • Yes, but not regularly. • Yes, regularly. • I’m not sure. http://www.flickr.com/photos/dadaistic_fantastic/5089525186/
24. Today’s Assessment Tools • Polls • Interactive exercises • Quizzes/questionnaires (pre and/or post) • Minute papers & muddiest point http://www.flickr.com/photos/azarius/225340667/
25. Polls
26. Poll Example Questions • How many of you have had a library instruction class here before? • Is this source peer-reviewed? / Is this source scholarly or popular? • Who is the author of this book? • Are you confused about topic x? http://www.flickr.com/photos/rakka/3451170932/
27. Quizzes/Questionnaires
28. Questionnaire Example Questions
31. The Good, the Bad, the Ugly
32. Minute Papers & Muddiest Point http://www.flickr.com/photos/holtsman/4331034955/
33. Minute Paper Example Questions
34. Best Practices for Question Drafting • Convert your learning outcomes to questions. • Include three to five “meaty” questions, two or three general questions. • Eliminate overly easy or “all of the above” answers. • Avoid nebulous or complicated questions. • Work in terminology to ensure it’s understood. http://www.flickr.com/photos/djmccrady/6118626233/
35. Best Practices for Question Drafting • Get a reviewer. • If at first you don’t succeed…. http://www.flickr.com/photos/djmccrady/6118626233/
36. Review Responses • With the class? Yes! • Look for… • Percentage of correct answers. • Trends in missed answers. • What to change next time. • Keep your chin up. http://www.flickr.com/photos/xingty/3421423768/
37. Verbal Setup http://www.flickr.com/photos/twenty_questions/2348686991/
38. Question Delivery Tools https://flic.kr/p/qVNyBQ
39. Question Delivery Tools • Kahoot! http://getkahoot.com • SurveyMonkey: http://www.surveymonkey.com • Poll Daddy: http://polldaddy.com/ • Poll Everywhere: http://www.polleverywhere.com/ • Google Forms: http://www.google.com/google-d-s/forms/ http://www.flickr.com/photos/litlnemo/5045121567/
40. Kahoot!
41. SurveyMonkey
42. PollDaddy
43. Poll Everywhere
44. Google Forms
45. Review • Write learning outcomes. • Base your questions on the outcomes and on your needs. • Review and reflect on the answers you get. http://www.flickr.com/photos/demonbaby/4228327167/
46. Thank you! Contact me at sksteiner@wcu.edu
47. Other Resources • Ambrose, Susan et al. How Learning Works: Seven Research-Based Principles for Smart Teaching. Jossey-Bass, 2010. • Bloom et al.’s Taxonomy of the Cognitive Domain. http://www.edpsycinteractive.org/topics/cognition/bloom.html • Cahoy, Ellyssa Stern, and Robert Schroeder. “Embedding Affective Learning Outcomes in Library Instruction.” Communications in Information Literacy, 2012. 6(1): 73-90. • Clay, Ben. Is This a Trick Question? A Short Guide to Writing Effective Test Questions. http://www.k-state.edu/ksde/alp/resources/Handout-Module6.pdf • Educational Oasis, “Framework of Learning Outcomes.” http://www.educationoasis.com/instruction/bt/learning_objectives.htm • London Deanery, “Setting Learning Objectives.” http://www.faculty.londondeanery.ac.uk/e- learning/setting-learning-objectives • Oakleaf, Megan. “A Roadmap for Assessing Student Learning Using the New Framework for Information Literacy for Higher Education.” In The Journal of Academic Librarianship, 40.5, 510-514. http://www.sciencedirect.com/science/article/pii/S0099133314001451 • The Teaching Center, “Asking Questions to Improve Learning.” http://teachingcenter.wustl.edu/strategies/Pages/askingquestions.aspx#.VPXu1uHG_nM • Utah State University Library, “Assessment.” http://library.usu.edu/instruct/assessment/index.php

Hinweis der Redaktion

All images are creative commons, cited in the bottom right corner.
Choose your questions and model based on the outcomes you need to meet.

Everyone has been watching the new Information Literacy framework develop. I agree with Megan Oakleaf that the framework doesn’t overthrow the importance of learning outcomes, though it might shift the way you were writing them if you were already doing them.

We will incorporate the framework’s tenets today and talk about how you can assess the threshold concepts that they’re comprised of.
Example: terminology—article, journal, database

This type of assessment is not an end in itself—it supports change and best practice in teaching.
Did they take away what you thought they were going to take away? If not, why?
Threshold concepts: “core tenets in a particular discipline that are transformative, irreversible, integrative, bounded, and potentially troublesome” MEYER and LAND

They tend to be highly constructive
I don’t care what scenario you did it in.

An objective states what we aim to do. Outcome describes what the student will be able to do at the end.
In many cases, these words are used interchangeably. Some people are moving in favor of Outcomes over objectives, because the term objective has become muddled over the years, so that now, objectives are sometimes written as teacher-centric instead of user-centric.

That means that the language is geared toward what I will do (as the teacher), rather than what you will be able to do (at the end of the class session).
Our first step in good, quick assessment is to write learning outcomes. The assessment questions which you choose will, in general relate specifically to those learning outcomes. They will also provide a roadmap for you, as the instructor, in terms of what you must cover. Whatever assessment method you choose later will tie back to these.

A good learning outcome will state what the student will be able to do, and WHY. It will use Bloom’s taxonomy verbs of various cognitive levels.

http://valenciacc.edu/facultydevelopment/tla/Candidate/documents/ExamplesandDefinitionsofLOs.pdf
First step: consider the top three or four class priorities. Write them down. If you are going to do some basic things then you can have more, like four or five. If you want to start mixing in larger concepts—things that are likely to trip them up and require some deep thought to understand--then you may find that you can’t fit in as many—but also, you may find that the way you teach is transformed, and that those more specific pieces aren’t necessary. (I’m still in my own liminal space with transforming to the thresholds, and a lot of others are, so if you feel nervous, you’re not alone.)

Think more broadly than “they will know which ten clicks will lead them to a perfectly refined search. Rather, help them understand information types and why they care about them in the first place.
Don’t be afraid to focus on “Big ideas and enduring understandings”
Do they understand why they would use a scholarly article search versus a Google search?
Do they understand why the scholarly conversation is critical?
Can they identify a scholarly article and its parts?

Learning outcomes should stem from a needs assessment. The needs assessment will determine the gap between the students’ existing condition and a desired condition—that condition is usually an increase in knowledge, skills, or attitudes.

Remember that these are just a pithy starting place for you. One outcome may take almost the full hour to unpack.
Future tense—timeframe
Identifies learner (some people say “students”, some like we)
Benjamin Bloom, an American educational psychologist, helped to write these in 1956. Domains of learning.
Cognitive domain.

I want to go through these in depth a little, because they will directly inform your assessment questions. So, as you choose verbs, consider them not just as objectives, but as assessment questions for your post test.
“Knowledge is the act of remembering previously learned material. It’s the lowest level of learning outcomes in the cognitive domain. “
“Comprehension is the ability to grasp the meaning of material. Students can demonstration comprehension through interpretation or summarization of material. These learning outcomes are a step beyond just remembering. They represent the lowest level of understanding.”
“Application refers to the ability to use learned material in new and concrete situations.”

We do this a lot in one shots.
“Analysis is the ability to break down material so that its organizational structure can be understood. Differentiation, comparing and contrasting, identification.

At this point you may notice that some of these verbs are on more than one list.
“Synthesis is the ability to put parts together to form a new whole.”
It can be hard to get to this in an hour. Our students often do this at the course level—they have to write a paper, a speech, a bibliography, a literature review. Synthesis is the ultimate goal, in many cases.
“Evaluation is the ability to judge the value of material in a particular scenario. The judgments are to be based on definite criteria.

Learning outcomes in this area are highest in the cognitive hierarchy because they contain elements of all the other categories, plus value judgments.
This should include context as often as possible without being too wordy or jargon-filled.

So, for example, in this last I’ve said “advanced techniques” rather than saying boolean logic, subject header searches.
We’ll get to those things, but not in our outcomes.

You don’t have to have the context, but I like it. If you don’t include it in your written outcomes, still think about it and be able to articulate the context to the students.
Stem and outcome
Demonstrable/assessable
Bloom’s cognitive domain—high level
No jargon

Often INCREMENTAL—step-based. They fit together to make a whole. They also, if you show them, provide a sort of roadmap for the students. Giving them a meaningful outline of the topics you’re going to cover is shown to help students engage with the class more thoroughly.

Let me again address the threshold concept piece of this. I hope it’s clear from each of these how they can be jumping off points for in-depth and transformative conversations, rather than point and click sessions. You can use them as a basis for teaching them the WHYs and WHENs of applying these skills, rather than just the HOWs.
Learner not identified—teacher focus
Not assessable
Poor verb
No verb
Jargon, too specific
HUGE in scope
Again, I don’t care about the scenario in which you did this. IF you did it at some point, it counts—with other librarians, in a class, whatever. You maybe did it verbally, or with a poll, or with a quiz, or with an essay, or even with analysis of papers.
Just read through, no elaboration

We’re going to talk about each and which might be best for you, given your setting.
Polls are good for feedback throughout the session--this kind of assessment is called formative assessment OR just at the end, for summative assessment. Formative assessment is anything you use it in the moment to immediately determine where the students are in their understanding. This type of assessment helps you as the instructor to identify material which needs to be clarified or covered in a different way—or who might need more time to think about the concepts you’re exploring.

I’m going to talk about the technical way to do these, but I also often just ask these kinds of questions verbally and skip over the techy part, especially if they are more diagnostic in nature—ie what skills do you already have.

Polls are great if you have information that builds and is cumulative. So, you can get them to answer a question or two before you move on to the next topic. They’re also a fun active learning exercise and they give you immediate feedback.

The questions have to be simpler, with no short answer or essay, but you can have more of them.

You can set them up to be embedded on a site/LibGuide (if your students have computers), or use free tools to set them up as text messages. If you have a clicker system, you can use them as well. Not too expensive, but they do cost $$$. (1000 to 2000... Other?) You can take those polls and embed them into PowerPoints, and then the answers will show up there (PollEverywhere). They can also just raise their hands, of course!

You can have students work together to submit a group answer, or you can give each individual a clicker.

Drawback—can be distracting, can have technical problems, can take up a lot of class time.
Regardless of what method you choose to deliver the content (phone, embedded poll, clicker) you can have different question types. Multiple choice, yes/no.

These probably seem basic, and that’s because they are, but they are often the basis for everything you’re trying to share with your class. If they have a shaky foundation, it means you can’t build a house on it and your efforts will be a failure, so these aren’t always exciting, but they’re critical.

Substantive or friendly—can also help you to set the stage for what you’ll cover in the beginning. I know these aren’t hifalutin, but it’s hard to build on a shaky foundation. These can be your foundation questions.

These can be great, because they are anonymous in the moment. So, if someone feels shy about raising a hand to say s/he is confused, this can help them to do it without drawing attention to themselves.
questionnaires tend to be my weapon on choice, especially when I was first starting with quick assessment and trying to determine where my basic disconnects with the students were.

You can position these however you like in your class time. I used to deliver them about ten minutes from the end of class, give the students five minutes, and then use the last five minutes to go over the answers quickly and take one last stab at fixing any confusion issues that arise.

Lately, I’ve been moving to a different model where I have them answer the questions right when they come in, then we discuss what they wrote as a group. I have found that this approach has a very different feel. It’s a great way both for you to get a sense of where the students are from the start, and to help them apply your teaching to something they have written in front of them.

You can also do a pre and post questionnaire.

With both quizzes and polls, you can easily convert your learning outcomes to question, if they’re carefully drafted. So, “You will know how to construct a search statement becomes “construct a search statement.”

Not more than five “meat” questions, maybe a couple of close out questions that are a bit more free-form.
This is a new one I’ve been working on and I want to tell you a bit about it. I tweak it every time, and if you adopt it you probably will too.

The intent of this first question is to learn what they’re anxious about and what I should focus on.
The second one is to hear their thoughts on why the library is important. Often they come up with abstractions like “my teacher says I have to” and that can actually be a fabulous jumping off point for a discussion about source credibility.
The third one tends to go much the same—they say things like “I click the peer-reviewed button.”

When covering these, I’m always careful not to shut them down when their answers are off target—rather, I talk about how we are now operating in a different framework—the research framework—and that that is different from the open web.

The fourth one has a specific function, and that is to ask them to apply what we are about to talk about to something already in their minds. This kind of anchoring technique can help them to relate what you’re saying to their reality,a nd therefore, find more meaning in it.
Pre and post, or just post. Pre and post can gauge learning, post can gauge what they’re walking out with. If you can get teacher cooperation, you can also do them again as a follow up, to gauge retention on various time scales.

Some of these are open ended, some multiple choice. It’s a good idea to have a variety. Not too many easy ones, or you may not learn anything useful.

You can also format these as quick assignments. Great for classes with one set task—evaluate and cite websites, for example.

You can also treat these like scavenger hunts, if you have enough time, and ask the students to get up and locate things.
You don’t need to be perfect your first time. If your questions aren’t perfect the first time or your instruction isnt perfect, that’s okay

You will realize even as you get more accomplished, that you sometimes don’t say things perfectly! You can correct that on the spot.

You can use answers as an opportunity to congratulate them for their awesomeness.
What is a minute paper? You have them write or type for just one minute—only two real questions (and maybe some basic information).
On paper or online is your choice. It’s easier to tabulate things online though.

Drawbacks:
It’s hard to evaluate these in the moment, but you can.

It’s also very hard to tie these in with your objectives—however, they are useful. I like to do a combination of hand-raising polling, with a post quiz that has a minute paper attached.
Gives them space to clarify and explain, if they want to. You can also invite them to leave contact info.
With polls and quizzes, if they’re brief, it’s easy to tabulate and review responses immediately. I like to do this. Skim over the answers and address problems right there. “It seems like I was unclear on Point X. Can someone describe the answer?

HAVE YOUR ELEVATOR SPEECHES READY.

This is the most important step. If you don’t integrate the information, it’s not useful to anyone.

We’re not perfect—this kind of thing can be jarring at first.

Change can relate to the way the questions are worded, ordered, or what is included, and also in the ways you teach.
Create a warm and welcoming environment by walking around in the beginning of class and during. Insure the students that the process is difficult and messy, and the assessments are intended to insure that they have what they need and have a chance to ask questions while you’re there to help.

Let them know the things you learn from their struggles help you to improve the classes for future students. Appeal to altruism=awesome.
Most of these can be used for both.
Absorbed Zoomerang

Basic: free
10 questions per survey
100 responses per survey
No fancy things like skip logic

Select: $26 per month ($312 per year)
Unlimited questions
Unlimited responses
Custom urls
Skip logic etc.
Can handle short answer, you can embed images, and it does quizzes and surveys in addition to polls.

If you’re asking what the difference is between a quiz and a survey in terms of this software, if you structure your setup as a quiz, you can set it to know the correct answers in advance.

Free account:
Used to be limited to 100 responses per month, but they took that away. So, the free version should be everything you want unless you want custom urls.
This is usable with any computer, tablet, or phone—even old ones, and it can handle short comment answers. You can embed in PowerPoint.

Free version
25 responses per poll or 40 responses per poll if you are in K through 12 or higher education. They don’t love public libraries, apparently.

The pricing beyond that point gets kind of complicated based on your situation, but it’s $14 per month and up.
Free, but somewhat clunky to use.
List of verbs: http://www.acu.edu/academics/adamscenter/course_design/syllabus/verbs.html

How to Improve Your Library Instruction: Assessment in Five Minutes (February 2017)

Du liest eine Vorschau.

Hier ansehen

How to Improve Your Library Instruction: Assessment in Five Minutes (February 2017)

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie How to Improve Your Library Instruction: Assessment in Five Minutes (February 2017) (20)

Weitere von ALATechSource (20)

Aktuellste (20)

How to Improve Your Library Instruction: Assessment in Five Minutes (February 2017)

Hinweis der Redaktion

How to Improve Your Library Instruction: Assessment in Five Minutes (February 2017)

Du liest eine Vorschau.

How to Improve Your Library Instruction: Assessment in Five Minutes (February 2017)

Empfohlen

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie How to Improve Your Library Instruction: Assessment in Five Minutes (February 2017) (20)

Weitere von ALATechSource (20)

Aktuellste (20)

How to Improve Your Library Instruction: Assessment in Five Minutes (February 2017)

Hinweis der Redaktion