IMLS WebWise 2014 Help Wanted In Search of the Right Evaluator
•Als PPTX, PDF herunterladen•
1 gefällt mir•291 views
Whether you are eager to learn more about existing or potential audiences (online and off), prototype a digital product, roll out a new website, or apply for grant or foundation funding, you may find yourself seeking the help of an evaluator. How do you know whom to choose? What qualities should you look for to best meet your needs? Can you afford it? What are the steps for finding the right fit? Not only will workshop participants leave WebWise knowing what to look for in an evaluator for their current or future project, they’ll have written the want ad. Mini-workshop given in Baltimore Feb 11, 2014
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Kürzlich hochgeladen
Kürzlich hochgeladen (6)
Empfohlen
Empfohlen (20)
IMLS WebWise 2014 Help Wanted In Search of the Right Evaluator
Hinweis der Redaktion
- Session description:Whether you are eager to learn more about existing or potential audiences, prototype an exhibit, or apply for grant or foundation funding, you may find yourself seeking the help of an evaluator. How do you know whom to choose? What qualities should you look for to best meet your needs? Can you afford it? Join in on a discussion about finding the right evaluation fit.[GROUP: Everyone have the twitter info down?]
- When I say ‘evaluator’ that’s because it’s my primary lens…But I mean it really broadly–What we cover today/here can be applied to anytime you are considering using a consultant—Anything from more traditional ‘visitor studies’ to user testing, learning research, market research… basically any intel about audiences to help us make informed decisions.
- [GROUP: Who's who in the audience?Why did youchoose the session?]
- Introduction (EYM, DMNS)I told IMLS I was up for doing this workshop because I’m sick of something I love— something I believe has the potential to make our work better and smarter– being something most people either actively hate or passively dismiss
- But I understand why…I have been on the “inside” of projects where I hire evaluation consultants, and now I AM an external consultant… so I can see both sides.
- [GROUP: Without naming names, are any of you up for sharing an experience where you were disappointed with how an evaluation went– be it the process, the methods, the communication, the evaluator her and himself… anything?]
- [GROUP: And what about the opposite?Any stories about when evaluation went exactly right and provided you with the information you truly needed to improve your work?]
- The good news is, regardless of your experiences with evaluation to date, you have some options.You have a choice.I’ll walk you through some of them.
- You can do it yourself– and many of you do this already. My advice to you? Keep doing it. Don’t stop. We are all evaluators!There are some real benefits to this:You can, in theory, start todayYou know the environment, the organizational climate, the playersYou may be more trusted, because you’re an insiderEvery time you do it, you’re likely to learn from it and get betterOn the other hand….It can be time-consuming. One more thing on top of all the things you already do.You may not have internal staff with the tools or resources to do the work.You also may be too close to the project, and it can be hard to evaluate your own work or your colleagues’ work.
- There are benefits that come with an outsider too…May be good to have an external perspective, someone who isn’t as close to the project or the playersAn outsider may be able to get information you can’tSometimes external evaluators can be perceived as more credible, with more experience and training and more tools to assess what’s going onSometimes it’s not optional; a funder might require it being an outsider.On the other hand…External evaluation can be more expensive– depending on the size and scope of the evaluation and how much the given evaluator chargesAnd just because you are paying them to do the work doesn’t get you off the hook for managing them, getting them what they need, up to speed, etc.
- An external evaluator who will work with or train your internal team to build your capacity, do some skill-building, and mobilize your internal staff to do data collection is often a great mix of the best of both approaches.This can lead to skills that build more evaluative thinking and an evaluative culture in your institution, with longer-lasting effects than a “one off” study.You can have them do the heavy lifting on the pieces you can’t or don’t want to do– like creating the instruments or analyzing the data.What’s also great about this is it builds a tremendous sense of internal buy-in and ownership. It can also bring down costs.
- Take about 5-10 mins on your own to work on the INTERNAL OR EXTENAL EVALUATION: WEIGHING YOUR OPTIONS handout that’s coming around.For your current project, or one you’re thinking about pursuing, weigh your options. For each of the listed factors, choose if internal evaluation is a strength or a weakness. List a couple of reasons why. Try to fill out each of the four columns for every factor. For example, availability. An internal strength is that you’re staff are located right there, onsite, and can fit the evaluation in between other things. A weakness is it takes them away from other work and is “one more thing” to add to the list. An external strength might be that if it’s a grant project, for example, they have allocated a percentage of their time to it already, so they’re locked in and dedicated. A weakness may be that they have competing projects that have to be factored in in terms of schedule, etc.Now I want you to pair up with someone nearby; ideally someone who is NOT from your own institution or on your project. Talk through the worksheet together for the next 5 min or so and compare how you weighed the strengths and weaknesses of internal versus external evaluation.[GROUP: What did you learn from this activity?]This is a great worksheet to use within your own teams as you are thinking about the evaluation component of your project. I will give you a link at the end so you can get it online.
- Let’s say for your project you decided you need external evaluation help; either to do the evaluation or to help get you started or train you in a hybrid-type model.
- What do you need to think about in order to find the right consultant for you?You may or may not be aware that you have a TON of choices, which means you get to shop around and actually find the best fit for you to help ensure you learn from– maybe even enjoy– evaluation.
- A great starting place is thinking about the stakeholders for this project.Who cares about this evaluation? Who has a stake in the results? Who’s going to use the outcomes to move the work forward?(Internal? Donors? Senior Leadership? Education staff? Curators? Colleagues? Public?)You want to start here so you can consider how particular evaluators might communicate with your unique audience.Communication is absolutely the key to evaluation results being heard, remembered, and acted on.
- A good next step is to ask yourself:What do we want to learn--really? Don't worry about how to ask the specific questions or how to word them… just lay out what you want to know and why.You could do this with a theory of change, or an user experience map… the idea is to get on paper what you know and where the holes are.What questions do you need the answers to now to help you tweak, improve, or change what you’re doing?What would be lost if you don’t evaluate?Evaluators definitely have different approaches and specialties. Being clear about your needs will help you find the best fit.
- A huge mistake I see happening is people just going with a name they’ve heard a lot, someone who authors a lot of journal articles, or someone they’ve seen present at a conference.I made this mistake. You’d think I would know better, but I chose someone I like very much on a personal level and had spent social time with at conferences-- and that person turned out to be a really bad fit for the evaluation, especially when it came to communicating the results clearly and effectively to my stakeholders. It was definitely a horror story. You have a choice!There are a lot of evaluators and audience consultants out there, and they’re all really different.[GROUP: Where might you look to find an evaluator?]Students, local evaluation firms (whether or not they specialize in museums), Visitor Studies Association, American Evaluation Association, neighboring culturals, etc.Ask around. Who have your favorite colleagues worked with? What were their experiences, good and bad?And hey– ask evaluators! Those who specialize within museum and library community tend to all know one another– it’s not that big!We will tell you what we’re good at, what we’re after, what we would rather not touch, and brainstorm ideas with you. I will give you some resources about where to look to start making your short-list.You’ll want more than one and you’ll want to know when each one is the right fit for different projects.
- Once you have your list…It’s not a lot different than a job interview.Evaluators are not all the same! Once you locate a few, call them up or ideally Skype. Don’t take email for an answer.As them what they love, what some of their most memorable projects have been, what methods they useIf you want to build capacity internally through working with an evaluator, ask them how they see that happeningAsk them about their philosophy– ask them what they care aboutAsk them how they deal with change, with creativity, with innovationAnd do your homework! Ask for work samples, references, whatever.Keep asking until you feel really good.Make contact early. Good evaluation consultants are usually busy, so give them enough lead-time to fit in their schedules.You don’t want to wait and choose someone out of desperation!Like any partnership, get to know them early, well before you’re even sure what you will need their help on.
- This next activity… Take some time on your own to write down what qualities you’re looking for in an evaluator.Does it depend on the project?Also have a look at the next steps and the questions you might want to ask evaluators you’re considering working with.Star the ones you like, re-write the ones you don’t, or come up with your own. In a few mins I will have you pair up again to discuss what you’ve come up with. Now I want you to pair up with someone nearby; ideally someone who is NOT from your own institution or on your project. Talk through the worksheet together for the next 5 min or so and compare what you’re looking for in an evaluator… and how you might find the right fit. [GROUP: What did you learn from this activity?]This worksheet is a good one to keep on hand as you start thinking about what you need for current or new projects… and how you might find it.
- They aren’t the only one you need to think about.How involved do you want to be? Do you want to learn through doing, be super involved, function more as a gatekeeper, or stand back until the end?If not you, who? Do you have someone who can take this role on?Some evaluators can be more or less demanding. Ask them about this! Ask, “what do you need from a client you’re working with?”
- You can have something on paper, and that’s a great start, but more important is the action of planning… Putting the time in at the beginning will save you immensely later onIt will help you, and whoever you work with, stay on task.Be willing to adjustShowing leadership a plan about what holes evaluation will fill, what questions will be answer that lead to actual behavior change and informed decision-making… That is convincing.Planning early and investing in evaluation can save you immensely. It pays off. For example, usability testing on your website can prevent…misleading navigational cues, poorly designed pathways, pages that are so dense they take a long time to use, etc.This method was first published by Clare-Marie Karat of IBM who used it to show a 100-fold return on investment for a particular software product. In that case, spending $60,000 on usability engineering throughout development resulted in savings of $6,000,000 in the first year alone.
- One thing that is so amazing, which I have learned from transitioning from an internal staff role to a consultant, is how much it matters when the point person at an organization is really organized and takes good care of you. It’s the little things, like showing you where all the bathrooms are and how to make the coffee maker, finding an empty desk where you can sit, introducing you to the staff, telling you the best places to grab lunch. That stuff really matters.
- This really varies, and can depend on experience level as well as philosophy (a grad student versus someone with 20+ years experience and published books).Usually anywhere from $50-75 an hour on the low end to $200-300 on the high end.Another good rule of thumb is to allocate somewhere close to 15% of your total project budget (or grant budget) to evaluation. Most evaluators will work with you on this. If they know you’re budget and what you hope to accomplish, they can let you know what they can realistically do within that range. I know it can feel exorbitant, but remember their projects are often sporadic and they’re paying for the own insurance, etc. etc.I can honestly say I don’t know a single evaluator I feel rips her or his clients off.
- Different ways you can go about working with an evaluator…A request for proposals can get you some great information. The hard thing about RFPs is the amount of time and efforts they can take, as well as the veil of secrecy often built in. What I mean is, the organization wants to keep things vague enough so a to get a range of responses, but the respondent has trouble knowing how to make something work with so much grey area. Leaving a window to have potential proposers ask questions and get answers helps, as does a budget range or ceiling.Indefinite duration, indefinite quantity. Essentially, you could have 4-5 people “on hold”, all with different skill sets well-matched for the kind of projects you might want them to do. Contacts and other paperwork done up front, and you call them when you need them. This is not a bad approach unless the person you call is already booked.The retainer model is something I have seen work really well for smaller institutions with limited budgets. It’s like having customer support in evaluation. You offer a set amount, say $5000, upfront, and then spend it down when you need it. Get a survey reviewed or designed; have them come in when you are working on a grant proposal and you need a logic model. Whatever. It’s small and flexible, so you have it when you need it and they can fit it in between bigger gigs.
- This will link to various resources like “find an evaluator” databases, lists of questions you can (and should) ask when interviewing evaluators, articles that expand on what I’ve talked about today.Also, for a lot of the work you do it may be well worth it to contact mobile or digital consultants, UX designers, or usability testers on the front end. Truthfully, not a lot of “museum evaluators” have that skill set or passion…(Some of us do, but not many.)Again, it’s about asking the right questions to find the right fit.Questions? Thanks for your time!