Workshop 2 methods, video and interviews

Evaluation workshop 2
Aims for participants:
1.1 To understand evaluation context and key ideas
1.2 To share evaluation case studies
1.3 To learn about evaluation research and examples
2.1 To recap evaluation methods
2.2 To carry out (video) interviews
2.3 To consider data analysis
3.1 To review evaluation reports
3.2 To present demographic information using mapping
software
3.3 To consider evaluation strategies
3.4 To present evaluation information

Workshop 2 activities
Workshop 2 Aims:
2.1 To recap evaluation methods
2.2 To carry out (video) interviews
2.3 To consider data analysis

Day 2
Methods I have already used Never used
Evaluation Methods notes
Format:
•Research examples
•Methods to try out
•Recommended reading

1.3 To learn about evaluation research and examples
-Research about evaluation and examples (GK)

Examples of front end evaluation
1. National Air and Space Museum front-end evaluation,
Korn and Associates, 1992
http://informalscience.org/images/evaluation/report_32.pdf
2. Natural History Museum Meterorites gallery, 2005
http://www.nhm.ac.uk/resources-rx/files/meteorites-evaluation-11677.pdf
3. Havard-Smithsonian Center for Astrophysics:
Front End Evaluation of EarthScope Panorama online game
MacPherson and Associates, 2007
http://informalscience.org/images/evaluation/report_264.PDF

Learner Voice
Hart’s ladder of engagement, 2008
Working with children importantly develops their view of democratic change,
and they bring refreshing insight into your programme plans.
Examples:
• Sharing artwork to be shown in the Images of Nature gallery at the Natural
History Museum. Activities in a local school explored children’s views about
how best to display complex artworks in ways that would appeal to younger
visitors. Outcome: child level questions prompted children to compare
pictures with well known and straightforward icons, and to look out for
motifs.
• Sensation Science Centre in Dundee, Scotland, facilitated children to writing
labels for the Discovery Boat exhibition.

To try: consulting with pupils in advance of a programme

Research ethics: video permission

To try: Teacher panel session
• Invite teachers through advertising on general websites e.g. TES
connect, teacher unions
• Share programme plans in a structured format
• What’s in it for them? Combine with evening event

Teacher Panels
Working closely with teachers at early stages in programming develops
relationships and ensures that programme content meets a need and is
aligned with curriculum goals.
Example: Darwin Centre Teacher Panel, Natural History Museum, London
Format: Teachers were invited to meetings before member events.
Individuals within the organisation submitted activities they required input for, at
the start of the academic year.
These were scheduled according to project timescales, with three topics
evaluated per panel.
The Teacher Panel culminated in a celebration event at the opening of the Darwin
Centre.

Recommended reading: Participatory
Approaches
Simon, N (2011) ‘The Participatory Museum’. Freely
available online.
http://www.participatorymuseum.org/
Innovation in front-end evaluation
Exploring Social Media for front end evaluation, Jensen and Kelly 2009
Click image for hyperlink

Key questions:
•What are your learning goals?
•What are the requirements for reporting demographic data?
•What is the scope for change?
•Drop in events require formative evaluation to monitor audience segment
attendance.
•Pre booked events should have this data already collected.
•When deciding how to ask for improvements, consider what is going to happen
with the data.
•Who will write it up?
•Who needs to know?
•What will they do with it?
•Is there capacity for change?
•Customer service- marketisation

Examples of formative evaluation
1.Science Café Online workshop evaluation
Nanoscale Informal Science Education
Le-Compte Hinley and Pattison, 2011
http://www.nisenet.org/sites/default/files/catalog/eval/uploads/2011/05/1353/sc
2. Investigating The Implementation of the Be A Scientist! Project in New
York City and Los Angeles Formative Evaluation – Year One
Center for Children and Technology Education Development Center, Inc.,
2011
http://informalscience.org/images/evaluation/BAS_YR1_Eval_Report_Final.pdf
3. Data Visualizations: Year 1 Formative Evaluation
Institute for learning innovation, 2011
http://informalscience.org/images/evaluation/Data_Viz.Sept_2011.Formative.fina

To try: Using google forms for an online survey
1
2
3
Benefits?
Disadvantages?

To try: participant observation
1. Structure according to physical area of interest

2. Record teacher or pupil activity

3. Make a map of your space
and tally behaviour
http://www.familylearningforum.org/evaluation/types-of-evaluation/timing-tra

4. Record classroom activity

Recommended Reading
http://psychology.ucdavis.edu/sommerb/sommerdemo/mapping/behmap.htm
Links with marketing and psychology:
http://www.mapping-museum-experience.com/en/about-project/project-descri
“Psychogeography”
http://www.hypothesisjournal.com/wp-
content/uploads/2011/09/vol8no1-hj012.pdf
Pearson, Barriault and Cochrane, 2010
Analysis of science communication engagement
Innovation: Use of iPads in understanding children’s learning
http://www2.2simple.com/apps?
virtuemart_product_id=101&virtuemart_category_id=4

Work Package Objectives -> Evaluation
-work package objectives were collaboratively developed into domains of
learning for summative evaluation (March 2012)
-This led to the development of the EU-UNAWE summative evaluation
framework (August 2012)
-National Project Co-ordinators improved the evaluation framework
(October 2012)

1.1 The Program is addressed specially to disadvantaged children in different social
environments.
1.2 The program intends to stimulate via images, playful activities, experiments, models
and artistic activities the children´s awareness of astronomical objects and the place
of the Earth as part of the Universe in a way of broadening the children´s view of the
world. In doing so try to use throughout daily-life materials in the activities, experiments
and models such that children can reproduce them at home.
1.3 Awaken the awareness of the uniqueness of the Earth and of the need for
environmental preservation.
1.4 Motivate children to appreciate other cultures via stories and twinning activities
with children living in other countries, making them aware of the place in which other
children live, their common cultural features and differences.
1.5 Throughout stimulate children´s curiosity giving them the occasion and time to
formulate their own questions related to the Universe.
1.6 Develop EUNAWE-contents to be linked to daily school curricula via training of
the teachers.

National Science Foundation
Informal Science Education
http://caise.insci.org/uploads/docs/Eval_Framework.pdf

Rationale for NSF Informal Science Education model
1. The variety in types of programs is expansive. Informal education and outreach activities
can take place in schools, museums, the community, the media, and various other locations
where people gather information and experience the world. Almost all ACC agencies had some
type of program that was designed to generate awareness and engage the public in the agency’s
work. Further, the types of activities varied considerably across programs.
2. The nature of these programs makes it difficult to conduct rigorous evaluation because,
among other reasons:
(1) the audience for these programs is diffuse and difficult to identify;
(2) the multiple factors affecting and affected by these activities cannot be isolated for
assessment;
(3) the modest scale of these efforts does not warrant a costly assessment approach.
There are examples of pre- and post- quasi-experimental evaluations of these programs, but it is
extremely challenging to carry out rigorous studies to identify causality in these programs.
3. Third, despite all of these complexities many programs share the same or similar goals. Though
the programs are varied, the Informal Education and Outreach group agreed on the importance
of better interagency coordination and information sharing. Sharing best practices across
agencies could offer significant benefits as these programs often bear a closer relationship to
outreach and informal education programs in other agencies than they do to K–12 or
postsecondary education programs in the same agency
(U.S. Department of Education. [DoEd], 2007, p. 26.).

EU projects have the same issues
“Furthermore, due to budget and time constraints,
the evaluation of initiatives is often
limited. The interconnections between initiatives
are very rare which effectively eliminates
the possibilities for scaling-up and for the
dissemination
of the new ideas: the dynamics of
“economy of scale” and the huge potential for
real impact are simply not being exploited”.
http://ec.europa.eu/research/science-
society/document_library/pdf_06/report-rocard-on-
science-education_en.pdf

European model: framework for informal education

March 2013: Evaluation guide published!

To try: interviewing pupils

To try: interviewing teachers

Your task:
to interview a colleague about the impact of participating in the UNAWE
programme

Day 2
Interviewing
What has been the
impact of being
involved in EU
UNAWE?

Your questions
Question Notes on answers
Name

Your questions
Question Notes on answers
Name
Knowledge
Skills
Motivation
Intercultural attitudes
Legacy
Guestnet
Username: conf1310
Password: jooraeth

Day 2
Aims for participants:
1.1 To understand evaluation context and key ideas
1.2 To share evaluation case studies
1.3 To learn about evaluation research and examples
2.1 To recap evaluation methods
2.2 To carry out (video) interviews
2.3 To consider data analysis
3.1 To review evaluation reports
3.2 To present demographic information using mapping
software
3.3 To consider evaluation strategies
3.4 To present evaluation information

Workshop 2 methods, video and interviews

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Workshop 2 methods, video and interviews

Similar to Workshop 2 methods, video and interviews (20)

Recently uploaded

Recently uploaded (20)

Workshop 2 methods, video and interviews