David Griffiths (University of Bolton) University of Salamanca, Spain November 29th, 2013

1. Methodology: a real problem
Professor David (Dai) Griffiths
The Institute for Educational Cybernetics
The University of Bolton
D.E.Griffiths@bolton.ac.uk

2. A methodological journey
● I am from the humanities, but working with
technology
● I will talk about
○ the methodological approaches that have
influenced me
○ some examples of problems that worry me
● I will not attempt to give a complete history of
methodology in 30 minutes
● I don’t want to upset anyone, but no doubt I will!

3. Caught between art and science
The arts focus on personal experience. I wanted
to understand how that related to an external
world
originating in or based on observation or
experience <empirical data>
capable of being verified or disproved by
observation or experiment <empirical laws>
But I wanted to avoid naive realism...
“Empirical” in the Merriam-Webster Dictionary

4. Some answers (or at least
questions) from the 1700’s

5. 1748. David Hume, An Enquiry
Concerning Human Understanding
● “What is the nature of all our reasonings concerning matter of
fact?”
–
“the relation of cause and effect”
● “What is the foundation of all our reasonings and conclusions
concerning that relation?”
–
“Experience“
● “What is the foundation of all conclusions from experience?”
–
“priority in time, proximity in space, and necessary
connection”
● How does necessary connexion arise
–
“from a number of similar instances which occur of the
constant conjunction of these events”

6. Empiricist methodology
● In the past 200 years methodologies based on
empiricism have triumphed
– Observe regularities (conceived in different
ways)
–
In the data are facts, or refutations, terlinked
facts about the world emerge from the data (or
disproven hypotheses)
● Empirical methodologies demonstrated their
success by manipulating the world

7. Empiricism demonstrated its
success with powerful technologies

8. Social and political questions...
● In empiricism (and its close cousin positivism),
there is no room for social or political
understanding
● Kuhn’s Structure of Scientific Revolutions
–
Showed that scientific understanding does not proceed
by gradual uncovering of truth
–
Large scale shifts of paradigm change the way that
scientists see the world

9. Critiques from a (broadly) Marxist
perspective
● If science is a social activity, how far do power and
influence determine what is true?
● The tobacco industry undermined my confidence
(echoed by climate science, the claims for
genetics)
● Two writers who impressed me
● Gould (1981) The Mismeasure of Man
(Excellent on statistics)
● Rose (2013) Cells and Brains: The Promethean
Promises of the New Biology

10. Paradox undermined my
confidence further
Russell: Does the baber
shave himself?
Answer: it is nonsense.

11. But Goedl showed...
● if you have a self-consistent recursive axiomatic
system powerful enough to describe the
arithmetic of the natural numbers
● Then there are true propositions about the
naturals that cannot be proved from the axioms.
This was deeply shocking.
It showed there were limits to what could be
achieved by empirical logic.

12. Recursion is a deep problem
● Hume
The mind has never anything present to it but the perceptions,
and cannot possibly reach any experience of their connexion
with objects. Hume (119)
● Empiricism has triumphed because as humans we
have very similar experiences of the physical world
● As social beings we have different experiences of
the world
● When we study ourselves, we do not find that
shared facts emerge from our observations

13. Cybernetics: an alternative set of
theories with a focus on recursion
● Alternative ways of conceptualising information.
(Bateson, Shannon, McCulloch)
● Modelling and conceptualising self reference
(Maturana, Luhmann, Leydesdorff, Beer)
● A performative (Pickering) method of
experimentation, modelling (e.g. Pask, Beer,
Checkland) and abduction (e.g. Bateson)
● Very helpful in understanding empiricism in the
context of lived experience and professional
practice
● I chose ‘Professor of Educational Cybernetics’ as
my title

14. I was also drawn to...
● Phenomenology: Denying the relevance of the
empirical to the human. Phenomenology “has, as
its exclusive concern, experiences intuitively
seizable and analysable in the pure generality of
their essence” (Hussrl 1913, in Moran 1990)
● Grounded theory: (Glaser & Strass) Accepting
the empirical, but ‘coding’ to emerging local
theoretical framework from the data
● Action Research (Lewin, Argyris, Heron).
Research applied to achieve a desired social
change, with a spiral of progressive improvement.

15. I have been involved in all three,
but...
● None of phenomenology, grounded theory or
action research have a strong method for
cumulation of results
● This makes it hard, or impossible, to make sense
of the technological or social interventions
● So maybe empirical, positivist approaches are the
right way to go in social science...

16. The Empirical strikes back
US Dept. of Education, project requirements, 2003
“… to determine whether the project produces
meaningful effects on student achievement or teacher
performance.”
“Evaluation methods using an experimental design are
best for determining project effectiveness. Thus, the
project should use an experimental design under which
participants--e.g., students, teachers, classrooms, or
schools--are randomly assigned to participate in the
project activities being evaluated or to a control group
that does not participate in the project activities being
evaluated.”

17. Evidence based policy
● Quantitative experimental methodologies with
control groups
○ confirm that managerial control is possible
○ provide defensible answers for managers
● But do they always provide the right answers?
● I suggest you read Seddon on this
Seddon (2008) Systems Thinking in the Public
Sector: The Failure of the Reform Regime.... and a
Manifesto for a Better Way

18. Big money is at stake. Is education
like the climate, and tobacco?
● McGraw Hill: “Using LearnSmart to study has
proven to lead to improved learning efficiency,
greater engagement, and better career
readiness” http://chronicle.com/items/biz/pdf/McGraw-Hill_LearnSmartEffectiveness-Study.pdf
● Pearson: “More than 6 million students around
the world are now using Pearson MyLab &
Mastering products. … Integrated usage of
these programs has shown to provide
measurable gains in student retention,
subsequent success, and overall achievement”
●
http://www.pearson.com.au/why-pearson/technology-learning/mylabsmastering/proven-results/

19. Gregory Bateson on the dormitive
principle
“ A common form of empty explanation is the
appeal to what I have called ‘dormitive principles’,
borrowing the word dormitive from Molière. There
is a coda in dog Latin to Molière’s Le Malade
Imaginaire, and in this coda, we see on the stage
a medieval oral doctoral examination. The
examiners ask the candidate why opium puts
people to sleep. The candidate triumphantly
answers, ‘Because, learned doctors, it contains a
dormitive principle’.”
Gregory Bateson, 2002. Mind and Nature: A Necessary Unity. Hampton Press, Cresskill NJ. p 80.

20. The dormitive principle in the
classroom
Mea culpa
Where is the pedagogic goodness of a good lesson located in a
lesson? If we can capture it, we can share it!
I looked for the teachers activity and tried to capture that with
IMS LD. Others look for it in content or curricula.
We can test for it, assess it, publish it, and average it. But it is
an idealisation, with no criteria for causal efficacy.
More generally
Why does that child learn more than the other?
Because, learned doctors, they contain more or less capacity to
learn. Now let’s go for our tribunal lunch.

21. Three domains of the real, according
to Bhaskar & Critical Realism
The EMPIRICAL: events that are
actually observed and experienced
The ACTUAL: events (and non-events)
that are generated by the mechanisms
The REAL: Mechanisms and structures
with enduring properties
Adapted from Mingers, J. & Brocklesby, J., 1997. Multimethodology: Towards a framework for mixing methodologies.
Omega, 25(5), pp.489–509.

22. Informed by Critical Realism...
● Take an unexplained phenomenon
● Propose hypothetical mechanisms that would, if
they existed, generate or cause the
phenomenon which we are trying to explain
● Not a traditional empirical approach because
○ The explanation does not emerge from the data
○ The data is not the phenomenon to be
explained
● Go from experiences in the empirical domain to
possible structures in the real domain.
● Competing explanations supported / eliminated
● Compatible with the performative cybernetic

23. Pawson and Tilley apply this to
social science evaluation
From Pawson, R. & Tilley, N., 1997. Realistic
Evaluation, Sage Publications Inc.
Page 72

24. By what method do we observe
outcomes?
● When trying to confirm or deny a mechanism, use
as many different methods as you like.
● Statistics are valuable, especially with big data and
analytics. They serve to confirm or rule out
mechanisms, but do not give direct access to them
● Multiple methodologies create epistemological
contradictions which will need to be handled with
care
● The mechanism provides a means for cumulation
● I recommend Mingers Realising Systems Thinking:
Knowledge and Action in Management Science

25. Market equilibrium theory: Goldman
Sachs report, 2004
“The development of the capital markets has
provided significant benefits to the average
citizen. Most importantly, it has led to more jobs
and higher wages… The capital markets have
also acted to reduce the volatility of the economy.
Recessions are less frequent and milder when
they occur. As a result upward spikes in the
unemployment rate have occurred less frequently
and have become less severe.”
W. Dudley, US Chief Economist Goldman Sachs.
R.G Hubbard, Dean, Columbia Business School

26. An epidemic

27. An epidemic
http://guardianlv.com/2013/11/adhd-becomes-an-epidemic/

28. Visser et al. Nov. 2013
provide figures for 2011
● Approx 2 million more U.S. 4 to 17 year olds
diagnosed with ADHD in 2011 than 2003.
● Taking medication for ADHD
○ 69% of children with current ADHD
○ 6.1%, of all children: 3.5 million children
● Medicated ADHD increased by 28% from 2007
to 2011.

29. Response to this data...
● Dr. John Walkup, Director Child & Adolescent
Psychiatry at Weill Cornell Medical College
● ... the data suggests the increasing diagnosis
rate of ADHD is getting closer to the true
prevalence of ADHD, which is even higher.
○ "We've been working so hard for so long to
improve treatment… If the prevalence rate is 9
to 11% and we're getting 8% currently
diagnosed, it suggests that the public advocacy
for treatment is paying off."
http://thechart.blogs.cnn.com/2013/11/22/adhd-diagnoses-rise-to-11-of-kids/comment-page-2/

30. Response to this data...
● Dr. John Walkup, Director Child & Adolescent
Psychiatry at Weill Cornell Medical College
● ... the data suggests the increasing diagnosis
rate of ADHD is getting closer to the true
prevalence of ADHD, which is even higher.
○ "We've been working so hard for so long to
improve treatment… If the prevalence rate is 9
to 11% and we're getting 8% currently
diagnosed, it suggests that the public advocacy
for treatment is paying off."
http://thechart.blogs.cnn.com/2013/11/22/adhd-diagnoses-rise-to-11-of-kids/comment-page-2/

31. Response to this data...
● Dr. Allen Frances, former chair of Psychiatry
Department at Duke University
○ Psychiatry... is a history of fads, and we are
now suffering from a fad of ADHD
○ the rates have tripled over the past 15 years
because of sales pressure from
pharmaceutical companies selling stimulants
to treat ADHD.
○ "We are medicalizing immaturity and turning
childhood into a disease," Frances said.
http://thechart.blogs.cnn.com/2013/11/22/adhd-diagnoses-rise-to-11-of-kids/comment-page-2/

32. Visser et al. conclude...
● “Efforts to further understand ADHD diagnostic
and treatment patterns are warranted”
● How could we resolve this debate?
● As researchers you need to be able to take a
position on this kind of debate, within your field
and beyond
● The implications cut deep into your work. I
suggest that you think hard about this, not just
apply a recipe.