1. Exploring how evaluation practice changes
over time
Postgraduate Conference, 25th April 2015
Joana Zozimo, Educational Research, Lancaster University

2. Why researching evaluation?
• Experience evaluation practice
in four countries: Portugal;
Spain; Mozambique and UK
• Short term experiences: Guinea
Bissau; Cape Verde; Angola and
Senegal
• PCM cycle main expertise:
evaluation stage
• Evaluation resistance,
misunderstanding, struggling
• Bed time story explaining
researcher/evaluator

3. Research Problem
• Performance measurement framework places certainty and
attribution (Kusek and Rist, 2004) over reflective practice,
uncertainty and contribution (Mayne, 2008; Patton, 2010;
Saunders et al., 2005)
• Development education (Dev Ed) sector knows very little about
how evaluation is being practiced, by whom and where it is
being practiced and to what effect (Henry and Mark, 2003).
• Dev Ed ultimate goals are attitudinal change; social change
and/or change behaviour of its beneficiaries.
• Scholars emphasised that Dev Ed changes are more likely to be
captured through a practice led-evaluation rather than a
performance led-evaluation (Bourn, 2014).

4. What is Development education?
• active learning process,
• founded on values of solidarity, equality, inclusion and co-operation.
• enables people to move from basic awareness of international development
priorities and sustainable human development,
• through understanding of the causes and effects of global issues
• to personal involvement and informed actions.
• fosters the full participation of all citizens in world-wide poverty eradication, and the
fight against exclusion.
• seeks to influence more just and sustainable economic, social, environmental,
human rights based national and international policies (DARE Forum, 2004).
Dev Ed Research context : funding dependent; volatile human resources; short term
activities based.

5. Research Problem (cont.)

6. My approach
• Overall research aim: explore how evaluation practice changes over time
within developmen education sector.
• Research context: Youth Project evaluation practice timeline; three-year
project; run by NDEC (3 groups of participants; funder-recipient context
based)
• Research questions: How do development education organisations evaluate
their interventions?
- What do the Youth Project stakeholders do when evaluate it?
- What are the internal and external influences that shape NDEC’s evaluation practice,
particularly in the Youth Project?
• What is Evaluation? What is a social practice?

7. What is Evaluation?
• sets or clusters of behaviours
• conceptualised as social practices
• forming ways of ‘thinking and doing’
• associated with undertaking evaluative activity
• attributing value, complexity and appropriateness to social
interventions (Saunders et al., 2011)
• I will use the term “Evaluative Practices”
• In other words, to evaluate is to attribute value …

8. How Evaluation is broadly perceived

9. What is a social practice?
• routinized type of behaviour,
• forms of bodily activities,
• forms of mental activities,
• ‘things’ and their use,
• a background knowledge in the form of understanding,
• know-how,
• states of emotion and motivational knowledge (Reckwitz, 2002)
In everyday’ s language, a practice is:
• a way of cooking, of working,
• of investigating, of taking care of oneself or of others,
• of evaluating

10. What is a social practice? (cont.)
Elements of Practice : Materials; Competence; Meaning
Dynamics of social practice: co-occurrence of practices; how they are carried by
practitioners (Shove et al., 2012)
• Overlap
• Interact
• Compete
• Collaborate
• Integrate
• Dominate
• Or what else?..
Social Practice Theory by Reckwitz (2002) and Shove et al (2012) applied to
Evaluation by Saunders (2011) in a Development Education sector.

11. Methodology / Research design
 Social constructivist perspective;
 Case study research as approach , not only method! (Simons, 2009); Single in-depth
case study;
 Qualitative inductive approach, purposeful sampling to allow rich case to be
explored;
 The case: Evaluation practice of the Youth project (three-year project run by NDEC).
 The case design: Units of analysis/ Embedded cases (Yin, 2003):
1. Coordinators
2. Staff
3. Funders
Research decision: ethical decision to design three embedded cases to preserve
indviduals’ anonynity and confidentiality.

12. Data collection overview
• Pilot study with 5 informal conversations; 3 interviews
• Fieldwork in practice: 7 hr of participant observation
• Observation notes; Informal conversations (n=3); documents (n=10);
Phase 1
Planning the
Evaluation
Feb till May
2012
• 15 in-depth semi structured interviews,
• 95 hr of participant observation spread over 3 months (weekly basis)
• 24 hr of non participant observation spread over 2 months (event basis)
• Interviews notes (n=15); observation notes;
Phase 2
Evaluation off
the ground
Sept-Nov 2012
• 3 in-depth interviews; 3 informal conversations
• 4 documents analysed
• Interview notes (n=3); Informal conversations (n=3)
Phase 3
Evaluation
reporting
March-April
2013

13. My Data set/ Approach to analysis
 6 Informal conversations notes/research diary
 18 interviews transcripts (fully transcribed)
 14 documents
 126 hr observational field notes /research diary
Case “story” account (Simons, 2009); Transforming data (Wolcott, 1994)
“Thick” description (Geertz, 1973); Longitudinal analysis (Saldana, 2002)
 Atlas TI software; coding and categorisation; seek patterns and connections;
 Three cycles of coding - Content analysis, value coding analysis, longitudinal analysis;
 Social practice theoretical lenses applied to focus on what different groups do on a
daily basis – coordinators; staff and funders;
 Informed by literature on:
Moments of evaluation (McCluskey et al, 2008)
Provisional stabilities (Saunders, 2005)
Dynamics of practice (Shove et al., 2012)

14. The Case Evaluation Practice
Timeline (WIP)
Example image
Youth
Project
approved
DfID’s
official
letter and
guidelibes
Stage 1
Planning
Evaluation
GMA’s
evaluation
mandate
Practice
change
episode
s (PCE)
DfID’s
evaluati
on
reviewe
d
My
presence
in the
fieldwork
Stag
e 2
Gett
ing
Eval
uati
on
“off
the
grou
nd”
Stage 3
Reporti
ng
Evaluat
ion
Dev
Ed
Politi
cal
conte
xt
chan
ge
PCE - Funders
contrasting
evaluation
approaches
2009/2010
2011
3/12 9/12 1/13

15. The Case Evaluation Practice
Timeline (WIP)
Example image
Finding
Designing
the
Project
Stage 1
Planning
Evaluation
Practice
change
episodes
Researcher
presence in
the fieldwork
Stage 2
Getting
Evaluation
“off the
ground”
Stage 3
Reporting
Evaluation
2009/2010 2011 3/12 9/12 1/13

16. Main Themes
• Absence of evaluation culture (dishonesty, fear, scepticism)
• Retrospective reflection
Finding
Designing the
Project
• Evaluation language/discourse
• Forced/Imposed practice
Stage 1
Planning
Evaluation
• Resistance
• Power (funders’ relationship)
• Managerialism
• Co-occurrence of practices
Stage 2
Evaluation
“off the
ground”
• Expectations
• Fragmented evaluation practice
• Competence
Stage 3
Reporting
Evaluation

17. My account/ research story
• The Grant Management Agency only administered the grant,
they weren’t development education specialists. It’s like asking
your bank what are the best sausages, you know...they don’t
know [laughs]. They might eat it, but they don’t know; you
know what I mean...they are not education people; they are
consultants who are administering the grants. (Mary, project
worker, 7:77, emphasis added)

18. My account/ research story
Reflection enables practitioners to notice how evaluation practice has changed over time;
and to learn from it !

19. My account/ research story
When a new actor [GMA] appear within the funders/recipient context, evaluation practice
changes (the way a practice is carried by the practitioners is altered, because new actors
have occupied that space in the field).

20. My account /research story
A fragmented practice changes the way evaluation is practiced over time.

21. My account /research story
Research participants are expected to have competence in evaluation, to respond to
funders mandatory requirements

22. My contribution is to the evaluation
practice debate
 Theoretical contribution – extending evaluation practice theory by using an
advance theoretical framework of SPT applied to a new domain of Dev Ed.
 Implications for Practitioners- add fresh insights on evaluative practices
within Dev Ed domain; dynamics of social practice places reflective
practice at the center of evaluation practice.
 Research Impact – gain momentum of 2015 Year of Evaluation
http://mymande.org/evalyear/Declaring_2015_as_the_International_Year
_of_Evaluation

23. My reflective space / Research
journey
 Writing as a sense making tool – despite the use of Atlas, it was only when
I start writing on a daily basis that I deeply engaged with my materials.
 Writing as interpretation (Richardson, 1994).
 Writing retreats (Rowena Murray) – learning the dynamics of academic
writing; create habits; practice writing in a peer group; supportive and
contained environment; create your own writing group.
 More resources:
• Rowena Murray work http://www.rowenamurray.org/
• FASS writing retreat http://www.lancaster.ac.uk/fass/gradschool/training/
• Thesis whisperer http://thesiswhisperer.com/

24. References
Bourn, D. (2014) The Theory and Practice of Global Learning. Development Education Research Centre.
DARE Forum. (2004). What is DARE. DEEEP - Developing Europeans’ Engagement for the Eradication of Global Poverty.
Retrieved from http://www.deeep.org/what-is-dare-.html
Geertz, C. (1973) The Interpretation of cultures. [New York (N.Y.)]: Basic Books.
Henry, J., Mark, M.M. (2003) Toward an agenda for research on evaluation. In C. A. Christie, ed. The practice-theory
relationship in evaluation. Jossey-Bass.
Kusek, J.Z., Rist, R.C. (2004) Ten Steps to a Results-Based Monitoring and Evaluation System: A Handbook for Development
Practitioners. World Bank Publications.
Mayne, J. (2008) Contribution analysis: An approach to exploring cause and effect.
McCluskey, A. (2011). Evaluation as deep learning: a holistic perspective on evaluation in the Pallete project. In
Reconceptualising evaluative practices in HE (Saunders et al.). Open University Press.
Reckwitz, A. (2002). Toward a Theory of Social Practices: A Development in Culturalist Theorizing. European Journal of
Social Theory, 5(2), 243–263
Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Guilford
Press.
Saldana, J. (2002) Analyzing Change in Longitudinal Qualitative Data. Youth Theatre Journal 16(1). 1-17

25. References
Saunders, M., Charlier, B., Bonamy, J. (2005) Using Evaluation to Create ‘Provisional Stabilities’. Evaluation. 11(1), 37–54.
Saunders, M., Trowler, P., & Bamber, V. (2011). Reconceptualising Evaluative Practices in HE (1st ed.). Open University
Press.
Simons, H. (2009) Case study research in practice. Los Angeles ; London: SAGE.
Shove, E. (2012) The dynamics of social practice: everyday life and how it changes. Los Angeles: SAGE.
Wolcott, H.F. (1994) Transforming qualitative data : description, analysis, and interpretation. Thousand Oaks, Calif.: Sage
Publications.
Yin, R.K. (2003) Case study research: design and methods. 3rd ed. Thousand Oaks, Calif: Sage Publications.

26. Thank you! What resonates with you?
Welcome your questions.
j.zozimo@lancaster.ac.uk