A Beginners Guide to Building a RAG App Using Open Source Milvus
Â
IDS Impact, Innovation and Learning Workshop March 2013: Day 2, Paper Session 3 Laura Camfield
1. Ensuring rigour and reliability
Laura Camfield, University of East Anglia
l.camfield@uea.ac.uk
2. ď˝ Qualitative methods in modern
impact evaluation
ď Whoâs doing it?
ď How are they doing it?
ď How is its quality being assured? And
are existing checks sufficient?
ď Ethical codes, peer review
ď Archiving
ď Restudies
2
3. ď˝ Endorsement
from DFID (funded
reports by Garbarino and
Holland, Stern)
ď˝ And from
NONIE, 3ie,
World Bank IEG,
Ford
Foundation, etc.
4. ď˝ Growing interest in approaches to
causation that incorporate
qualitative methods
ď Small n/ case study
ď Theory Based
ď Realist
ď Process tracing
5. ď˝ Range of research designs limited
ď˝ Qualitative research typically post-hoc
rather than ex-ante
ď˝ Integrated analyses are rare
ď˝ Hard to access full methodological accounts
â including how the analysis was done - or
raw data
6. ď˝ Growing numbers of mixed methods
designs, many ongoing
ď 3ie â 700+ studies, 50 qual or mixed
methods (education, health, social
protection, rural livelihoods)
ď˝ But qualitative component often small
or superficial,
⌠e.g. sampling through participatory wealth
ranking or using criteria from PPA, âfield
visitsâ, âqualitative surveysâ
7. ď˝ âEmpowering parents to improve education: evidence from
rural Mexicoâ
⌠Reports data from 30 focus groups with parents from treatment
and control schools, but no analysis of differences between
groups or how the programme promoted parental participation.
For further information we are referred to an internal report
ď˝ âChanging Householdsâ Investments and Aspirations through
Social Interactionsâ
⌠Mentions âfocus groups and semi-structured interviews with a
wide set of beneficiaries and other local actorsâ which âsuggests
that aspirations and perspectives towards the future may have
been a key part of program impactâ (p9). The methodology and
data underpinning these claims are in a report that has not been
translated and isnât available online
8. ď˝ âAssessing the Impacts of Farmer Field Schools on
Excessive Fertilizer Use in Chinaâ
ď Detailed description of RCT methodology and sampling -
âqualitative data will also be collected to see what worked and
how it worked'
ď˝ âEvaluating the Impact of Technology Development
Funds in Emerging Economies: Evidence from Latin
Americaâ
ď Qualitative methods not included in the methodology, but
emerges like a rabbit from a hat ... âfinally, based on qualitative
evidence from this study, financial support should be
accompanied by the infrastructure and technological services of
research centres and universitiesâ
9. ⌠Data production
ď âIntellectual biographiesâ of researchers and
fieldworkers
ď Quality of note taking â capturing embodied
knowledge?
⌠Data management
ď Transcription
ď Translation
⌠Analysis
ď âTertiaryâ analysis
⌠Representation
ď Moving beyond quotes and text boxes
ď Evidencing claims - ethnographic authority?
9
10. 1. Record of publications, presentations etc which explain and /or
draw on the archived data.
2. Research design
a) Brief context and logic of research design (in each data collection
period if appropriate).
b) Is research exploratory or question driven? What are the
questions?
3. What were the sampling decisions and how do they relate to the
research questions?
a) Was the desired sample achieved? How does the sample relate to
wider empirical evidence across the population and/or theoretical
issues?
b) Are there implicit as well as planned ways in which the sample is
structured?
4. An overview of what data is provided as part of the project.
5. A descriptive profile of each participant specifying units of data
(interview; diary; by wave).
10
11. ď˝ Rigour, Credibility, Relevance
⌠1. Was there a clear statement of the aims of the research?
⌠2. Is a qualitative methodology appropriate?
⌠3. Was the research design appropriate to address the aims
of the research?
⌠4. Was the recruitment strategy appropriate to the aims of
the research?
⌠5. Were the data collected in a way that addressed the
research issue?
⌠6. Has the relationship between researcher and participants
been adequately considered?
⌠7. Have ethical issues been taken into consideration?
⌠8. Was the data analysis sufficiently rigorous?
⌠9. Is there a clear statement of findings?
⌠10. How valuable is the research?
12. Validity in Quantitative and qualitative research
12
Quantitative research
Qualitative research
Bryman
Guba & Lincoln âNaturalistic inquiryâ
âtrustworthiness and authenticityâ
Internal validity Legitimate causality claims Do the observations correspond well to the
theoretical constructs.
Can you support claims that you observed
what you claim?
Credibility Multiple accounts of the same phenomenon
researcher validation
respondent validation
triangulation
External validity Generalisability
(representative samples)
Can the qualitative findings be transferred
across social contexts.
Transferability Research on small groups;
Depth vs. breadth
âThickâ description (Geertz)
Reliability Repeatability, replicablity Could the qualitative research be replicated
(access to qualitative inquiry âtextsâ)?
Researchers adopt similar social roles
Do researchers agree about what they see
and here? â inter-observer credibility
Dependability Trustworthiness by providing âaudit trailâ â
extensive records of all aspects of the research,
including raw data
Re-studies
Objectivity Objectivity, methodological
rigour
Objectivity in social science is spurious
Values more embedded in qualitative
research
Confirmability Researcher worked in good faith
13. ď˝ Possible solutions (Camfield and Palmer-Jones, 2013)
ď Ethical codes, peer review
ď Data archiving
ď Restudies
13
14. ď˝ Social science ethical codes typically focused on
care of the subject, but see GoSâs (2007:1)
emphasis on ârespect, rigour and responsibilityâ
ď âact[ing] with skill and care in all scientific workâ
ď âcommunicating results and intentions honestly and
accurately, and understanding that your work or its
outputs will have an impact on society in its broadest
senseâ
ď˝ Not all countries/ institutions/ disciplines have
ethical codes or committees
ď˝ Ethics committees assess research designs, but do
not monitor data production or presentation
15. ď˝ Peer reviewers rarely see the data â no
qualitative data deposit with journals â so
much is taken on trust/ reputation
ď˝ Grey literature and working papers may not
receive the same level of scrutiny, but are still
influential
ď˝ âAudit trailâ obscure in earlier examples
16. ď˝ What does it offer?
⌠Methodological insights
⌠Increased use of data for research and teaching
⌠Respect for respondentsâ accounts and time
⌠Judging validity of claims
⌠Historical perspective
ď˝ Secondary analysis is difficult to do well,
but within impact evaluation, most
qualitative analysis is secondary, or even
tertiary
17. ď˝ âScientific replicationâ
⌠Goldthorpe and Lockwood (1962-3) tested the âaffluent workerâ
hypothesis by taking highly-paid car assembly workers at the
Vauxhall factory in Luton as a âcritical caseâ to investigate whether
everyone was becoming middle-class. They concluded that this
wasnât the case, albeit on the basis of a small quantitative sample
and (Savage argues) inadequate engagement with the data. Savage
(2005:39) suggests that the conclusion arose because they had
fitted their data into a particular typology which closed off
alternative interpretations.
ď˝ Methodological insights
⌠Gillies and Edwards (2011:23) describe how in Townsend and
Marsdenâs (1965) study of single mothers, their âphysical
attractiveness (or lack of it) is commented on [...and] perceived
intelligence and character was also subject to evaluationâ.
18. ď˝ Some examples from developing countries, e.g.
ESRC funded restudy of Indian villages studied in
the 1950s by Bailey, Mayer and Pocock; van
Schendel, 1981 (Bangladesh); Breman et al., 1997
(S. & S.E. Asia); Breman, 2007 (Gujarat, India),
ď˝ And some controversies, often due to âinterpretive
overreachâ (e.g. Freeman vs. Mead, Lewis vs.
Redfield, Tierney vs. Chagnon)
ď˝ Why restudy?
ď (Dis)confirming original findings
ď Epistemological/ methodological insights
ď Longitudinal perspective
ď Running out of field sites...
19. ď˝ Constructivist revisits (types 1 and 2) assume the
site being studied at two points in time does not
change so differences are due to the different
relation of the ethnographer to the site (type I) or
theory that the ethnographer brings to the site
(type 2)
ď e.g. Weiner's (1976) feminist reconstruction of Malinowskiâs
(1922) Argonauts of the Western Pacific highlighted the
importance of mortuary ceremonies in cementing womenâs
control over ancestral identity alongside Malinowskiâs
celebration of the âKula ringâ
ď˝ Positionality and assumptions also important in
evaluation
20. ď˝ Realist revisits (types 3 and 4) study
historical change. Type 3 revisits focus on
internal processes in accounting for
differences between authorsâ, while type 4
emphasise external forces
ď Hutchinsonâs (1996) revisit to Nuerland (now Southern
Sudan), which was studied by Evans-Pritchard (1940),
explicitly to explore the impact of decolonization, war,
Christianity, and transnational capital on the lives of the
Nuer
ď˝ Hard to reliably attribute and understand effects of
particular interventions in complex, changing
environments
21. ď˝ Would Bemba society (N. Zambia) break down when
men migrated to southern African mines because
the slash and burn agricultural system (citimene)
needed men to cut down the trees?
⌠Richards â yes (conclusions enthusiastically adopted by colonial
administrators/ chiefs for their own reasons)
⌠Moore and Vaughan (50 years later) â no (real threat was Zambian
governmentâs agrarian reforms which promoted the labour-
intensive cash crop maize)
ď˝ But Moore and Vaughanâs analysis may also have
been âone-sided, governed by specific feminist and
Foucauldian assumptionsâ (Burawoy, 2003:667) and
effects will be seen in future revisitsâŚ
22. ď˝ What lessons can be learned for
evaluation?
⌠Watch out for interest groups!
⌠Donât forget about the women
⌠Impacts of interventions are often unintended,
multi-dimensional, and affect every member of
the household, not just the target
ď maize income and womenâs labour controlled by men,
women diverted from subsistence farming, family
becomes food insecure and women not entitled to take
a share of the income to buy food, childrenâs health
suffers...
23. ď˝ Quality of research is an ethical issue
ď˝ Increasing role for qualitative research in impact
evaluation requires increased reflection on current
standards
ď˝ Epistemological and methodological problems
cannot easily be detected or influenced through
conventional processes (standards, committees....)
ď˝ But qualitative researchers have own strategies for
ensuring rigour and reliability
ď˝ Data archiving and restudies contribute to these