2. What?
AUDIENCE RESEARCH
EVALUATION
VISITOR STUDIES
2
3. What? Observe
Data collection
Analysis
Polling
Research
Reporting
Surveys Feedback
Opinions
Understand
Improve
Who visits? 3
Marketing
4. What?
IN GENERALâŚ
Audience Research or Visitor Studies are
terms commonly used in the museum field to
describe the process of systematically
obtaining knowledge from and about
museum visitors, actual and potential, for the
purpose of increasing and utilizing that
knowledge in the planning and execution of
activities that relate to the public.
AAM Committee for Audience Research & Evaluation
4
5. What?
IN GENERALâŚ
Audience Research or Visitor Studies are
terms commonly used in the museum field to
describe the process of systematically
obtaining knowledge from and about
museum visitors, actual and potential, for the
purpose of increasing and utilizing that
knowledge in the planning and execution of
activities that relate to the public.
AAM Committee for Audience Research & Evaluation
5
6. What?
IN ITS SIMPLEST TERMSâŚ
Evaluation is a process that
helps us answers 3 questions:
ď˘ What?
ď˘ SoWhat?
ď˘ Now What?
6
7. What?
WHATâS YOUR VIEW?
Knowledge
Experience
7
8. What?
WHEN TO EVALUATE
Front-End
Formative
Remedial
Summative
8
Concept Development Preview Post-Opening/
Implementation
9. What?
EVALUATION METHODS
ď˘ Types of methods include:
ď Questionnaires/Surveys
ď Interviews (structured ď semi-structured ď structured)
ď Focus Groups
ď Focused Observations
ď Tracking & Timing Studies
ď Concept Mapping
ď Document Analysis
ď˘ There is no âbestâ method â we often use
mixed methods 9
11. So what?
OUTPUTS
⢠Projects
⢠Exhibits
⢠Collections
⢠Research
⢠Budgets
11
12. So what?
OUTCOMES. . . AKA IMPACTS . . .
ď˘ I learned about . . .
ď˘ My family is going to do . . .
ď˘ I am more interested in . . .
ď˘ I had fun. . . . We had fun. . . .
i.e. if successful, how will your audience be
12
different as a result of their experience?
13. So what?
BIG QUESTION:
HOW DO MUSEUMS MAKE A DIFFERENCE?
13
And how do we know?
14. So what?
NSF IMPACT CATEGORIES
ď˘ Awareness, knowledge or understanding (of)
ď˘ Engagement or interest (in)
ď˘ Attitude (towards)
ď˘ Behavior (related to)
ď˘ Skills
GENERIC LEARNING OUTCOMES
ď˘ Knowledge and Understanding
ď˘ Enjoyment, Inspiration, Creativity
ď˘ Attitudes and Values
ď˘ Activity, Behavior and Progression
ď˘ Skills
14
15. So what?
AWARENESS, KNOWLEDGE OR
UNDERSTANDING
ď˘ Knowing what or about something
ď˘ Learning facts or information
âVisitors will be aware that their
ď˘ Making sense of something
daily actions have an impact on the
ď˘ Deepening understanding
marine environment.â
ď˘ How museums, libraries and archives operate
ď˘ Making links and relationships between
15
16. So what?
ENGAGEMENT OR INTEREST
ď˘ Participation, engagement, interest are
prerequisites for other types of learning
ď˘ Having fun
âChildren who participate in the
ď˘ Being surprised
ď˘ Innovative thoughts more engaged in
program will be
viewing contemporary art.
ď˘ Creativity
ď˘ Exploration, experimentation
ď˘ Being inspired
16
17. So what?
ATTITUDES & VALUES
ď˘ Feelings
ď˘ Perceptions
ď˘ Opinions about ourselves (e.g. self esteem)
ď˘ âAdults will express a positive
Opinions or attitudes towards other people
ď˘ Increased capacity for the importance
attitude towards tolerance of
ď˘ Empathy play in their livesâ
ď˘ Increased motivation
ď˘ Positive and negative attitudes in relation to an
experience 17
18. So what?
BEHAVIOR
ď˘ What people do
ď˘ What people intend to do
âAdults will avoid eating foods that
ď˘ What people have done
have a substantially negative
ď˘ Reported or observed actions
impact on the way people manage their
ď˘ A change in
the environment.â
lives
18
19. So what?
SKILLS
ď˘ Knowing how to do something
ď˘ Being able to do new things
ď˘ Intellectual skills
âAdults will be able to document
ď˘ Information management skills
observations of plants that are
affected by climate change.â
ď˘ Social skills
ď˘ Communication skills
ď˘ Physical skills 19
20. Now what?
WHAT DOES SUCCESS LOOK LIKE?
DEVELOPING INDICATORS
20
21. Now what?
FRYE ART MUSEUM
Impact Possible Indicators
1. Visitors will point, gesture, step
forward, nod, raise hand during
Visitors will be Engage physically tour program.
and verbally with the artwork
2. Visitors will ask questions,
make statements regarding the
artwork, and share opinions
with gallery guide or other
visitors
Evidence of success: % of visitors who âŚ.
21
22. Now what?
YOUR TURN!
PROPOSED EXHIBIT TOPIC:
âHOMELESSNESS IN THE PACIFIC NORTHWESTâ
1. Develop an impact statement, one for each category
2. Write your impact statements on the board
22
23. Now what?
YOUR TURN!
PROPOSED EXHIBIT TOPIC:
âHOMELESSNESS IN THE PACIFIC NORTHWESTâ
1. As a class lets discuss some potential indicators of
successâŚ.
2. What evidence would show you that your desired impact
has been achieved.
23
24. Now what?
USE IT!
Australia Museum â âMistyâ Cases
24
http://australianmuseum.net.au/Audience-Research-Blog/
25. Now what?
USE IT!
Seattle Aquarium â Harbor Seal Exhibit
25
26. SUMMARY
â˘Evaluation is a systematic process of collecting
information from your audience.
What?
â˘Evaluation should be included in all phases of
development.
â˘Museums should clarify their desired impacts
before they create their outputs.
So what?
â˘Evaluation can be use to help assess whether
desired impacts are achieved.
â˘Use evaluation to paint a picture of the museum
landscape, in decision making, and
Now what?
improvement of exhibits & programs.
26
â˘Donât do it unless youâre ready to use it!
27. More!
RESOURCES
Visitor Studies Association (VSA)
www.visitorstudies.org
American Evaluation Association
www.eval.org
Committee for Audience Research & Evaluation (CARE)
www.care-aam.org
Inspiring Learning (GLO)
www.inspiringlearningforall.gov.uk
National Science Foundation
http://www.nsf.gov/pubs/2002/nsf02057/nsf02057.pdf 27
28. More!
JOURNALS & WEB
www.informalscience.org - evaluation reports
Visitor Studies
Curator: The Museum Journal
Journal of Interpretation Research
Journal of Museum Education
Museums & Social Issues
Informal Learning Review
28
Introduction, Me (MSU Museum, NYA, Museology & New Directions). My interest in VS, I want to know why things happen the way they doâŚand how we can improve our practiceâŚAlso, Seek to improve or practice through evidence. Evidence-based practice. SIMPLE CONCEPTS! Story about New York Aquarium, developing sea lion shows âyou are an integral part of the environment around you and your actions can help or hurt that environmentâ⌠everyone always said âwe have such an amazing conservation messageâ⌠my response: âhow do we know thatâs true?, how do we know that anyone is taking that away?â âŚ. Theyâd say âwell, if just one person gets it, weâve done our jobââŚ. After putting in so much time, money and effort into a program/exhibit⌠do we really want to just effect one person?.... I think we can shoot higher! And the way we do that is through gathering data, gathering evidence that what weâre doing is relevant, something visitors actually are responding to.
Weâll use these terms somewhat interchangeably.Ask group what comes to mind when they think of the term âEvaluationâ.
These are some of the words you used when I asked about what you thought people who work with visitor studies do. Add in understand, and improve. Museum evaluators seek to understand and document the experiences of our visitors, and learning that occurs
The wordy definition ď âŚâŚ AAM, committee for Audience Research & Evaluation
Important part here: systematic collection of information, visitors looking at actual and potential audiences, using that knowledge, and planning activities based on the data you collect.
Kathleen Tinworth, Audience Research Director and Denver Museum of Science & Nature. And easy philosophy to live by when thinking about starting to use evaluation in your workâŚ. Easy for people to get without using so much jargon.
These are your results of your rated knowledge and experienceâŚ. Opinion Line: Not everyone has the same âknowledgeâ or âexperienceâ with VS, but everyone comes with some perspective or opinion, itâs important to know how we feel, our own philosophy, before we undertake VS, or do it in our museums! Go through prompts.. Discuss responses â tell me more about why you think this, any brave souls want to play devilâs advocate? âŚ. Evaluation is a valuable tool to determine an organization or institutions effectiveness.Evaluation should be everyoneâs responsibility.Evaluation is best done by professionals with training and skills.As long as we provide a great visitor experience, we donât really need evaluation.Politics determine decision making, not information we get from evaluation.Some data is better than no data.Evaluation take staff time and resources.Itâs common to think evaluation is something we just tack on to the end of a project, or only do when we have extra time. But evaluation should be incorporated to the whole project development processâŚAlso, Seek to improve or practice through evidence. Evidence-based practice. SIMPLE CONCEPTS! Story about New York Aquarium, developing sea lion shows âyou are an integral part of the environment around you and your actions can help or hurt that environmentâ⌠everyone always said âwe have such an amazing conservation messageâ⌠my response: âhow do we know thatâs true?, how do we know that anyone is taking that away?â âŚ. Theyâd say âwell, if just one person gets it, weâve done our jobââŚ. After putting in so much time, money and effort into a program/exhibit⌠do we really want to just effect one person?.... I think we can shoot higher! And the way we do that is through gathering data, gathering evidence that what weâre doing is relevant, something visitors actually are responding to.
When should you do evaluation? Stages of program or exhibit development. Evaluation isnât just something you do at the end of a project⌠itâs an integral part of project development:Front End: conceptual, before the start, exploratory, visitor understanding, interest, misconceptions, paints a picture of the landscape before project begins, often called âneeds assessmentâ (especially for grant proposals).Formative: Assess ongoing project activities as a program/exhibit is being developed (often iterative),Results are intended to offer direct, concrete, and practical ways to improve a project,Must be open to making changes midstream based on something that does not seem to be working.Remedial:Small studies that take place when program/exhibit is up and running,Tweaking and modifications, maybe after a âsoftâ open.Summative: To assess whether or not a project achieved its goals and objectives. Did the program/exhibit do what it was intended to do? Determine the efficacy of the program/exhibition,Serves to inform future exhibit and program developmentStory: back to aquarium, we wanted to do a show about votingâŚ. But they said ,âmaybe we shouldnât because people already know that voting is a conservation action. Well, lets ask them! We did some front-end evaluation⌠they rated voting as lower importance so, we know that thatâs an area we can beef up..⌠after the show, they rated it higher. A good example of evaluation in practiceâŚ.
Breezing through this, some methods that can be used in evaluation. This is overly emphasized by museums looking to do evaluation. I.e. if we want to do evaluation, a common mistake is to jump right into methods without taking time to plan, âlets evaluate this exhibit, we should make a survey and hand it out to visitorsâ⌠. The method used should be dictated by the questions that we ask or the type of information we want to know.
SO WHAT!? Lets take a step back for a second, Brainstorm as group and write list on board of what museums do, is there a difference in the items on the board? Some about what we do (activities) some about the difference we make (outcome)
Outputs are the activities that museum undertake⌠they are the âproductsâ or the âthingsâ that museums do or create. Itâs great to design and put up an exhibit, but âWhyâ would a museum put up an exhibit, develop a program?... Thatâs more important
Impacts (also referred to as outcomes) - The results of the outputs, the results of our activities⌠how are you effecting some sort of change in your audience? How do they think, feel, act differently⌠âat the end of the day I know this exhibit was successful whenâŚ.âGo back to board. . . Which things were outputs, which were impacts? Describe this images (kids playing in a giant robinâs nest, what impact might the team have desired in creating this exhibit?
Impact categories & frameworkâŚ. National Science Foundation CategoriesGeneric Learning Outcomes: From the Museums, Library and Archive Council in the UK: âInspiring Learningâ meant to help museums improve their learning activities
What are some of the outcomes we want for awareness, knowledge or understanding?
Similar to engagement, interest⌠but typically refers to more stable constructs like âempathy towards animalsâ⌠or happiness ratings scales, âagree-disagreeâ scale items⌠(like our opinion line)
Relevant to Zoo, Aquariums, Botanical Gardens â commonly, getting people to take part in conservation actionsâŚ.For example, we might say we want to change behavior by saying youth who participate in the project will choose to spend more time outside in the next 3 months
Measurable demonstration of new skills or the reinforcement or practice of developing skills. (procedural aspects of knowing)
How do we know weâre making a difference, and how do we know weâre achieving these impacts? What does success look like? This brings us back to evaluation⌠evaluation comes in by identify Indicators of successâŚ
With the person next to you, or at your table, develop impact statements for a proposed exhibit with the topic of homelessness in the pacific northwest
With the person next to you, or at your table, develop impact statements for a proposed exhibit with the topic of homelessness in the pacific northwest
Donât conduct evaluation if youâre not prepared to use it! Some examples from practice⌠Australia Museum âAudience Research Blogâ â Misty Cases example. Audience research there showed that people were sometimes disheartened by seeing multiple stuffed (taxidermied) animals in a row, such as these birds⌠exhibits was able to take that information and develop these âmistyâ cases where visitors can pass by without seeing them, or stand close to view just one or two⌠from a distance the cases appear frosted over.
New Directions team, doing a front-end evaluation of the current harbor seal exhibitâŚ. Exhibit will change regardless, but an example of assessing visitor knowledge and attitudes toward harbor seals, and attention to the exhibit and itâs interpretive elements⌠Will be a great âbaselineâ or start, once renovation is complete, will re-do the evaluation â a summative â to see whether or not there is a change.
Summarize the discussion:
With the person next to you, or at your table, develop impact statements for a proposed exhibit with the topic of homelessness in the pacific northwest
With the person next to you, or at your table, develop impact statements for a proposed exhibit with the topic of homelessness in the pacific northwest
Contact information. Pass out sheets, quick evaluation, âwhat is one thing you took away, or one thing you think youâll remember from our discussion today?â