1. Ensuring rigour and reliability
Laura Camfield, University of East Anglia
l.camfield@uea.ac.uk

2.  Qualitative methods in modern
impact evaluation
 Who’s doing it?
 How are they doing it?
 How is its quality being assured? And
are existing checks sufficient?
 Ethical codes, peer review
 Archiving
 Restudies
2

3.  Endorsement
from DFID (funded
reports by Garbarino and
Holland, Stern)
 And from
NONIE, 3ie,
World Bank IEG,
Ford
Foundation, etc.

4.  Growing interest in approaches to
causation that incorporate
qualitative methods
 Small n/ case study
 Theory Based
 Realist
 Process tracing

5.  Range of research designs limited
 Qualitative research typically post-hoc
rather than ex-ante
 Integrated analyses are rare
 Hard to access full methodological accounts
– including how the analysis was done - or
raw data

6.  Growing numbers of mixed methods
designs, many ongoing
 3ie – 700+ studies, 50 qual or mixed
methods (education, health, social
protection, rural livelihoods)
 But qualitative component often small
or superficial,
◦ e.g. sampling through participatory wealth
ranking or using criteria from PPA, ‘field
visits’, ‘qualitative surveys’

7.  ‘Empowering parents to improve education: evidence from
rural Mexico’
◦ Reports data from 30 focus groups with parents from treatment
and control schools, but no analysis of differences between
groups or how the programme promoted parental participation.
For further information we are referred to an internal report
 ‘Changing Households’ Investments and Aspirations through
Social Interactions’
◦ Mentions ‘focus groups and semi-structured interviews with a
wide set of beneficiaries and other local actors’ which ‘suggests
that aspirations and perspectives towards the future may have
been a key part of program impact’ (p9). The methodology and
data underpinning these claims are in a report that has not been
translated and isn’t available online

8.  ‘Assessing the Impacts of Farmer Field Schools on
Excessive Fertilizer Use in China’
 Detailed description of RCT methodology and sampling -
‘qualitative data will also be collected to see what worked and
how it worked'
 ‘Evaluating the Impact of Technology Development
Funds in Emerging Economies: Evidence from Latin
America’
 Qualitative methods not included in the methodology, but
emerges like a rabbit from a hat ... ‘finally, based on qualitative
evidence from this study, financial support should be
accompanied by the infrastructure and technological services of
research centres and universities’

9. ◦ Data production
 ‘Intellectual biographies’ of researchers and
fieldworkers
 Quality of note taking – capturing embodied
knowledge?
◦ Data management
 Transcription
 Translation
◦ Analysis
 ‘Tertiary’ analysis
◦ Representation
 Moving beyond quotes and text boxes
 Evidencing claims - ethnographic authority?
9

10. 1. Record of publications, presentations etc which explain and /or
draw on the archived data.
2. Research design
a) Brief context and logic of research design (in each data collection
period if appropriate).
b) Is research exploratory or question driven? What are the
questions?
3. What were the sampling decisions and how do they relate to the
research questions?
a) Was the desired sample achieved? How does the sample relate to
wider empirical evidence across the population and/or theoretical
issues?
b) Are there implicit as well as planned ways in which the sample is
structured?
4. An overview of what data is provided as part of the project.
5. A descriptive profile of each participant specifying units of data
(interview; diary; by wave).
10

11.  Rigour, Credibility, Relevance
◦ 1. Was there a clear statement of the aims of the research?
◦ 2. Is a qualitative methodology appropriate?
◦ 3. Was the research design appropriate to address the aims
of the research?
◦ 4. Was the recruitment strategy appropriate to the aims of
the research?
◦ 5. Were the data collected in a way that addressed the
research issue?
◦ 6. Has the relationship between researcher and participants
been adequately considered?
◦ 7. Have ethical issues been taken into consideration?
◦ 8. Was the data analysis sufficiently rigorous?
◦ 9. Is there a clear statement of findings?
◦ 10. How valuable is the research?

12. Validity in Quantitative and qualitative research
12
Quantitative research
Qualitative research
Bryman
Guba & Lincoln “Naturalistic inquiry”
“trustworthiness and authenticity”
Internal validity Legitimate causality claims Do the observations correspond well to the
theoretical constructs.
Can you support claims that you observed
what you claim?
Credibility Multiple accounts of the same phenomenon
researcher validation
respondent validation
triangulation
External validity Generalisability
(representative samples)
Can the qualitative findings be transferred
across social contexts.
Transferability Research on small groups;
Depth vs. breadth
“Thick” description (Geertz)
Reliability Repeatability, replicablity Could the qualitative research be replicated
(access to qualitative inquiry “texts”)?
Researchers adopt similar social roles
Do researchers agree about what they see
and here? – inter-observer credibility
Dependability Trustworthiness by providing “audit trail” –
extensive records of all aspects of the research,
including raw data
Re-studies
Objectivity Objectivity, methodological
rigour
Objectivity in social science is spurious
Values more embedded in qualitative
research
Confirmability Researcher worked in good faith

13.  Possible solutions (Camfield and Palmer-Jones, 2013)
 Ethical codes, peer review
 Data archiving
 Restudies
13

14.  Social science ethical codes typically focused on
care of the subject, but see GoS’s (2007:1)
emphasis on ‘respect, rigour and responsibility’
 ‘act[ing] with skill and care in all scientific work’
 ‘communicating results and intentions honestly and
accurately, and understanding that your work or its
outputs will have an impact on society in its broadest
sense’
 Not all countries/ institutions/ disciplines have
ethical codes or committees
 Ethics committees assess research designs, but do
not monitor data production or presentation

15.  Peer reviewers rarely see the data – no
qualitative data deposit with journals – so
much is taken on trust/ reputation
 Grey literature and working papers may not
receive the same level of scrutiny, but are still
influential
 ‘Audit trail’ obscure in earlier examples

16.  What does it offer?
◦ Methodological insights
◦ Increased use of data for research and teaching
◦ Respect for respondents’ accounts and time
◦ Judging validity of claims
◦ Historical perspective
 Secondary analysis is difficult to do well,
but within impact evaluation, most
qualitative analysis is secondary, or even
tertiary

17.  ‘Scientific replication’
◦ Goldthorpe and Lockwood (1962-3) tested the ‘affluent worker’
hypothesis by taking highly-paid car assembly workers at the
Vauxhall factory in Luton as a ‘critical case’ to investigate whether
everyone was becoming middle-class. They concluded that this
wasn’t the case, albeit on the basis of a small quantitative sample
and (Savage argues) inadequate engagement with the data. Savage
(2005:39) suggests that the conclusion arose because they had
fitted their data into a particular typology which closed off
alternative interpretations.
 Methodological insights
◦ Gillies and Edwards (2011:23) describe how in Townsend and
Marsden’s (1965) study of single mothers, their “physical
attractiveness (or lack of it) is commented on [...and] perceived
intelligence and character was also subject to evaluation”.

18.  Some examples from developing countries, e.g.
ESRC funded restudy of Indian villages studied in
the 1950s by Bailey, Mayer and Pocock; van
Schendel, 1981 (Bangladesh); Breman et al., 1997
(S. & S.E. Asia); Breman, 2007 (Gujarat, India),
 And some controversies, often due to ‘interpretive
overreach’ (e.g. Freeman vs. Mead, Lewis vs.
Redfield, Tierney vs. Chagnon)
 Why restudy?
 (Dis)confirming original findings
 Epistemological/ methodological insights
 Longitudinal perspective
 Running out of field sites...

19.  Constructivist revisits (types 1 and 2) assume the
site being studied at two points in time does not
change so differences are due to the different
relation of the ethnographer to the site (type I) or
theory that the ethnographer brings to the site
(type 2)
 e.g. Weiner's (1976) feminist reconstruction of Malinowski’s
(1922) Argonauts of the Western Pacific highlighted the
importance of mortuary ceremonies in cementing women’s
control over ancestral identity alongside Malinowski’s
celebration of the ‘Kula ring’
 Positionality and assumptions also important in
evaluation

20.  Realist revisits (types 3 and 4) study
historical change. Type 3 revisits focus on
internal processes in accounting for
differences between authors’, while type 4
emphasise external forces
 Hutchinson’s (1996) revisit to Nuerland (now Southern
Sudan), which was studied by Evans-Pritchard (1940),
explicitly to explore the impact of decolonization, war,
Christianity, and transnational capital on the lives of the
Nuer
 Hard to reliably attribute and understand effects of
particular interventions in complex, changing
environments

21.  Would Bemba society (N. Zambia) break down when
men migrated to southern African mines because
the slash and burn agricultural system (citimene)
needed men to cut down the trees?
◦ Richards – yes (conclusions enthusiastically adopted by colonial
administrators/ chiefs for their own reasons)
◦ Moore and Vaughan (50 years later) – no (real threat was Zambian
government’s agrarian reforms which promoted the labour-
intensive cash crop maize)
 But Moore and Vaughan’s analysis may also have
been ‘one-sided, governed by specific feminist and
Foucauldian assumptions’ (Burawoy, 2003:667) and
effects will be seen in future revisits…

22.  What lessons can be learned for
evaluation?
◦ Watch out for interest groups!
◦ Don’t forget about the women
◦ Impacts of interventions are often unintended,
multi-dimensional, and affect every member of
the household, not just the target
 maize income and women’s labour controlled by men,
women diverted from subsistence farming, family
becomes food insecure and women not entitled to take
a share of the income to buy food, children’s health
suffers...

23.  Quality of research is an ethical issue
 Increasing role for qualitative research in impact
evaluation requires increased reflection on current
standards
 Epistemological and methodological problems
cannot easily be detected or influenced through
conventional processes (standards, committees....)
 But qualitative researchers have own strategies for
ensuring rigour and reliability
 Data archiving and restudies contribute to these