Evidence-Based HR Management & Systematic Reviews PhD Consortium of the 7th International Conference of the Dutch HRM network, Rob Briner, Eric Barends

1. Evidence-Based Management
What is it?
Why do we need it?
How does it look like in practice?
PhD Consortium of the 7th International
Conference of the Dutch HRM network

2. Evidence-Based Management
Postgraduate Course
 What is it?
 Where does it come from?
 Why do we need it?
 What is stopping us?
 How does is look like in practice?

3. Postgraduate Course
1. Evidence based management:
What is it?

4. Four propositions
Postgraduate Course
 Research produced by management scholars could be useful to
organizations
 Drawing on available evidence (including research produced by
academics) is likely to improve decisions
 Managers and organizations do not appear to be strongly aware
of nor use research findings
 We need to increase awareness of and access to research
findings

5. What is EBMgt?
Postgraduate Course
Evidence-based management is about making
decisions through the conscientious, explicit,
and judicious use of four sources of information:
practitioner expertise and judgment, evidence
from the local context, a critical evaluation of
the best available research evidence, and the
perspectives of those people who might be
affected by the decision.
(Briner, Denyer, Rousseau, 2009)

6. What is EBMgt?
Postgraduate Course
The problem is that management is not as evidence-
based as it should be nor as it could be

7. What is EBMgt?
Postgraduate Course
 It is not a completely new idea – managers and
organizations use evidence all the time
 EBMgt is different because it‟s about:
 Increasing the types of evidence we use
 Using it more thoughtfully and carefully
(conscientious, judicious, explicit)
 Its only purpose is to help us make better
decisions through more and more systematic
use of evidence

8. Who says it’s a problem?
Postgraduate Course
 People in many other fields (medicine, social
work, criminology, politicians) say it‟s a
problem in their fields
 Not everyone in management but some of
people think it‟s a problem
 Some academics and researchers
 Some managers and organizations
 Some professional associations
 Some commentators and journalists

9. •9

10. Origins of EBMgt
Postgraduate Course
Management not the only field where there are these
concerns. What field is this?
 “a research-user gap”
 “practitioners do not read academic journals”
 “the findings of research into what is an effective intervention are
not being translated into actual practice”
 “academics not practitioners are driving the research agenda”
 “the relevance, quality and applicability of research is
questionable”
 “practice is being driven more by fads and fashions than
research”
 “many practices are doing more harm than good”
 “the collective wisdom from research is being lost”

11. Evidence-Based Practice
Postgraduate Course
1991 Medicine
1998 Education
1998 Probation service
1999 Housing policy
1999 Social care
2000 Nursing
2000 Criminal justice
???? Management?

12. Academic interest in EBMgt
Postgraduate Course
Similar ideas around for a long time
2003 – Systematic reviews of evidence
2006 – Rousseau EBMgt Presidential Address and
Pfeffer & Sutton book
2007-2009 – Rousseau EBMgt Collaborative
2008-2011 – Several conferences
2011 – Center for Evidence-Based Management
(CEBMa) in Amsterdam
2011 – EBMgt Handbook
But many researchers not interested at all in EBMgt

13. Manager & practitioner interest in EBMgt
 Some HRM and professional bodies express
interest (SHRM, SIOP, VOV Learning Network)
 Some Universities running courses for
practitioners (e.g., Amsterdam, Ghent)
 Some publications for practitioners
 But managers are:
 Used to working in a different way
 Under pressure to adopt fads and fashions
 Have high expectations of evidence

14. Managers used to working in a
different way:
 Need to act quickly: Speed more
important than accuracy
 Organizational politics
 Formal authority and hierarchies
 Over-emphasize experience
 Rewarded for getting things done not
doing what works

15. Sometimes we are evidence-based
 Try to gather data and information
 Invest time and effort in trying to
understand and apply it
 Question our and others‟ assumptions and
logic
 Are sceptical about what appear to be fads
 Resist the temptation to act quickly

16. Sometimes we are not so
evidence-based
 Act on gut feeling (though intuition can be
important for some decisions)
 Copy other people who appear successful
(benchmarking)
 Think there is one ideal way (best practice)
 Let the „solution‟ frame and define our „problem‟
and create need (kitchen gadgets)
 Want to fit in and be as cool as everyone else
(fashion)

17. Postgraduate Course
2. Evidence based management:
Where does it come from?

18. Medicine: Founding fathers
Postgraduate Course
David Sackett Gordon Guyatt
McMaster University Medical School, Canada

19. Problem I: too much information
Postgraduate Course
 More than 1 million articles in 40.000 medical journals per
year (= 1995; now probably more than 2 million). For a
specialist to keep up this means reading 25 articles every
day (for a GP more than 100!)
 Most of the new insights and treatment
methods don‟t reach the target group

20. Problem II: persistent convictions
Postgraduate Course
if you’re
breathe into a bag
hyperventilating

21. Problem III: jumping to conclusions
Postgraduate Course
people who have an give them a drug that
irregular heartbeat are reduces the
much more likely to die number of
of coronary disease irregular beats

22. Postgraduate Course
Examples of mechanistic reasoning
gone wrong:
 Oestrogen replacement therapy to reduce cardiac events and stroke
in post-menopausal women.
 Treatment of measles with antibiotics.
 Rest for recovery.
 Placing babies on their fronts to prevent Sudden Infant Death
Syndrome (SIDS).
 Debriefing after psychological trauma

23. Problem IV
Postgraduate Course
David Sackett
Half of what you learn in medical school will be
shown to be either dead wrong or out-of-date
within 5 years of your graduation; the trouble is that
nobody can tell you which half.
The most important thing to learn is how to learn
on your own.
(Remember that your teachers are as full of bullshit
as your parents)

24. Evidence based decision
Postgraduate Course
David Sackett
“Good doctors use both individual clinical expertise and
the best available external evidence, and neither alone is
enough. Without clinical expertise, practice risks
becoming tyrannized by evidence, for even excellent
external evidence may be inapplicable to or inappropriate
for an individual patient. Without current best
evidence, practice risks becoming rapidly out of date, to
the detriment of patients.”

25. Management: Founding Mother
Postgraduate Course

26. Management: Founding Fathers
Postgraduate Course
Jeffrey Pfeffer Robert Sutton

27. Postgraduate Course
2. Evidence-based management:
Why do we need it?

28. EBMgt: some basic assumptions
Postgraduate Course
 Research produced by management scholars could be useful to
organizations
 Drawing on available evidence (including research produced by
academics) is likely to improve decisions
 Organizations do not appear to be strongly aware of nor use
research findings
 EBMgt is a potentially useful way of thinking about how we can
incorporate research evidence into decision-making

29. Postgraduate Course
Reason 1:
Errors and Biases of Human
Judgment

30. Errors and Biases of Human Judgment
Postgraduate Course
 Seeing order in randomness
 Mental corner cutting
 Misinterpretation of incomplete data
 Halo effect
 False consensus effect  Confirmation bias
 Reinterpreting evidence  Authority bias
 Group think  In-group bias
 Self serving bias  Recall bias
 Sunk cost fallacy  Anchoring bias
 Cognitive dissonance reduction  Inaccurate covariation detection
 Distortions due to plausibility

31. Errors and Biases of Human Judgment
Postgraduate Course
 Seeing order in randomness
 Mental corner cutting
 Misinterpretation of incomplete data
 Halo effect
 Confirmation bias
 False consensus effect
 Authority bias
 Reinterpreting evidence
 In-group bias
 Group think
 Recall bias
 Self serving bias
 Anchoring bias
 Sunk cost fallacy
 Inaccurate covariation detection
 Cognitive dissonance reduction
 Distortions due to plausibility

32. Seeing order in randomness
Postgraduate Course
We are predisposed to see order, pattern and causal
relations in the world.
Patternicity: The tendency to find meaningful patterns in
both meaningful and meaningless noise.

33. Seeing order in randomness
Postgraduate Course
We are pattern seeking primates: association learning

43. Points of impact of V-1 bombs in London
Postgraduate Course

44. Points of impact of V-1 bombs in London
Postgraduate Course

45. Errors and Biases of Human Judgment
Postgraduate Course
 A Type I error or a false positive, is
believing a pattern is real when it is not
(finding a non existent pattern)
 A Type II error or a false negative, is
not believing a pattern is real when it is
(not recognizing a real pattern)
Dr. Michael Shermer
(Director of the Skeptics Society)

46. Errors and Biases of Human Judgment
Postgraduate Course
 A Type I error or a false positive: believe that the
rustle in the grass is a dangerous predator when it is
just the wind (low cost)

47. Errors and Biases of Human Judgment
Postgraduate Course
 A Type II error or a false negative: believe that the
rustle in the grass is just the wind when it is a
dangerous predator (high cost)

48. Errors and Biases of Human Judgment
Postgraduate Course
Pattern detection problem
Assessing the difference between a Type I and
Type II error is highly problematic (especially in
split second „life and death‟ situations), so the
default position is to assume
all patterns are real.

49. Errors and Biases of Human Judgment
Postgraduate Course
Jennifer Whitson, University of Texas Austin, corporate environments

50. Errors and Biases of Human Judgment
Postgraduate Course
Erroneous beliefs plaque both experienced
professionals and less informed laypeople alike.
stress peptic ulcer

51. Oct 2005
Peptic ulcer – an infectious disease!
This year's Nobel Prize in Physiology or Medicine goes to Barry Marshall and Robin
Warren, who with tenacity and a prepared mind challenged prevailing dogmas. By
using technologies generally available (fibre endoscopy, silver staining of
histological sections and culture techniques for microaerophilic bacteria), they
made an irrefutable case that the bacterium Helicobacter pylori is causing disease.
By culturing the bacteria they made them amenable to scientific study.
In 1982, when this bacterium was discovered by Marshall and Warren, stress and
lifestyle were considered the major causes of peptic ulcer disease. It is now
firmly established that Helicobacter pylori
causes more then 90% of duodenal ulcers.
The link between Helicobacter pylori
infection and peptic ulcer disease has been
established through studies of human
volunteers, antibiotic treatment studies and
epidemiological studies.

52. Errors and Biases of Human Judgment
Postgraduate Course
Doctors, teachers, lawyers and managers hold many
erroneous beliefs, not because they are ignorant or
stupid, but because they seem to be the most sensible
conclusion consistent with the available evidence.
They hold such beliefs because they seem to be the
irresistible products of their own professional experience.
They are the products, not of irrationality, but of flawed
rationality

53. Errors and Biases of Human Judgment
Postgraduate Course
 Managers seem to be extremely good at generating
ideas, theories, and explanations that have the ring of
plausibility. They may be relatively
deficient, however, in evaluating and testing those
ideas once they are formed.
 This requires that we think critically about
experience, question our assumptions, and challenge
what we think we know
 (Show me the evidence!)

54. Postgraduate Course
3. Evidence-based management:
What is stopping us?

55. Quick fixes (1)
Postgraduate Course
What is the quick fix? A „solution‟ which
 Focuses on style and presentation not content
 Is not evaluated
 Is always slower than we hoped
 Usually doesn‟t work
 Is followed by another quick fix
 Everybody forgets and becomes subject to
organizational amnesia

56. Quick fixes (2)
Postgraduate Course
So why do we do quick fixes?
 Can be career-enhancing for managers
(e.g., issue selling, kick-ass CEOs)
 Speed is often valued over accuracy
 Heavily sold and marketed
 Are we all looking for quick and easy
solutions?
So who needs or wants academic research?

57. •57

58. •58

59. •59

60. •60

61. •61

62. •62

63. •63

64. •64

65. Management Fads (1)
Postgraduate Course
The nearly-forgotten fads
 Scientific Management/Taylorism
 Business Process Reengineering
 Management by results
 Excellence
 Total Quality Management
 Learning Organizations
 Knowledge Management

66. Management Fads (2)
Postgraduate Course
The fads that haven‟t been forgotten (yet)
 Talent management
 Management development
 Executive coaching
 Emotional intelligence
 Employee engagement
 Myers Briggs Type Indicator
 Belbin Team Roles
General concern about the destructive impact of fads
from both practitioners and researchers

67. •67

68. •68

69. •69

70. •70

71. •71

72. •72

73. •73

74. •74

75. •75

76. •76

77. *
FADS SEEM TO BE ATTRACTIVE,
COMPELLING AND IRRESISTIBLE
 Promise to deliver a lot and fast
 Appear simple
 New and shiny
 Will make everything alright and help contain
anxieties around intractable problems
 Help user feel effective and cutting edge
 Bits of some fads may work in some contexts
So who needs or wants academic research?
*Evidence-based management not a fad!

78. •78

79. •79

80. dDwtNzI3
•80

81. •81

82. Errors and Biases of Human Judgment
Postgraduate Course
 Seeing order in randomness
 Mental corner cutting
 Misinterpretation of incomplete data
 Halo effect
 False consensus effect  Confirmation bias
 Reinterpreting evidence  Authority bias
 Group think  In-group bias
 Recall bias
 Anchoring bias
 Inaccurate covariation detection
 Distortions due to plausibility

83. Under pressure to adopt fads
Postgraduate Course
“And there we see the power of any big managerial
idea (or fad). It may be smart, like quality, or
stupid, like conglomeration. Either way, if everybody's
doing it, the pressure to do it too is immense. If it turns
out to be smart, great. If it turns out to be
stupid, well, you were in good company and most likely
ended up no worse off than your competitors. Your
company's board consists mostly of CEOs who were
probably doing it at their companies. How mad can they
get?

84. Under pressure to adopt fads
Postgraduate Course
The true value of conventional management wisdom is
not that it's wise or dumb, but that it's conventional. It
makes one of the hardest jobs in the world, managing
an organization, a little easier. By following it, managers
everywhere see a way to drag their sorry behinds
through another quarter without getting fired. And isn't
that, really, what it's all about?”
(Colvin, 2004, Fortune)

85. Postgraduate Course
4. Evidence based management:
How does it look like in practice?

86. Four sources
Postgraduate Course

87. Postgraduate Course
JAMA, 1992

88. Push vs Pull
Postgraduate Course
Push: teaching (management) principles
based upon a convergent body of
research and telling students what to do.
Pull: teaching (managers) how to
find, appraise and apply the outcome of
research (evidence) by themselves

89. The 5 steps of ‘pull’ EBP
Postgraduate Course
1. Formulate an answerable question
2. Search for the best available evidence
3. Critically appraise the evidence
4. Integrate the evidence with your managerial
expertise and organisational concerns and apply
5. Monitor the outcome

90. The 5 steps of ‘pull’ EBP
Postgraduate Course
1. Formulate an answerable question
2. Search for the best available evidence
3. Critically appraise the evidence
4. Integrate the evidence with your managerial
expertise and organisational concerns and apply
5. Monitor the outcome

91. Answerable question
Postgraduate Course
I am a consultant, my client a large health-care
organization. The board of directors has plans for a
merger with a smaller healthcare organization.
However, it‟s been said that the organizational culture
differs widely between the two organizations. The board
want‟s to know if this can impede a successful outcome.

92. Answerable question: PICO(C)
Postgraduate Course
P = Population or problem
I = Intervention or successfactor
C = Comparison
O = Outcome
C = Context

93. Answerable question: PICOC
Postgraduate Course
P: What kind of Population are we talking about? Middle managers,
back-office employees, medical staff, clerical staff?
O: What kind of Outcome are we aiming for? Employee productivity,
return on investment, profit margin, competitive position, innovation
power, market share, customer satisfaction?
P/C: And how is the assumed cultural difference assessed? Is it the
personal view of some managers or is it measured by a validated
instrument?

94. The 5 steps of ‘pull’ EBP
Postgraduate Course
1. Formulate an answerable question
2. Search for the best available evidence
3. Critically appraise the evidence
4. Integrate the evidence with your managerial
expertise and organisational concerns and apply
5. Monitor the outcome

95. Where do we search?
Postgraduate Course

96. Where do we search?
•Postgraduate School

97. The 5 steps of ‘pull’ EBP
Postgraduate Course
1. Formulate an answerable question
2. Search for the best available evidence
3. Critically appraise the evidence
4. Integrate the evidence with your managerial
expertise and organisational concerns and apply
5. Monitor the outcome

98. Critical appraisal
Postgraduate Course
How to read a research article?

99. Critical appraisal
Postgraduate Course
1. Study designs
2. Levels of evidence
3. Bias / confounding
4. Effect sizes
5. External validity

100. Research designs
Postgraduate Course
Which study for which question?
The “best” evidence depends on the question type !

101. Levels of evidence
Postgraduate Course

102. The 5 steps of ‘pull’ EBP
Postgraduate Course
1. Formulate an answerable question
2. Search for the best available evidence
3. Critically appraise the evidence
4. Integrate the evidence with your
managerial expertise and organisational
concerns and apply
5. Monitor the outcome

103. Organization concerns
Postgraduate Course
Always ask yourself to what extent the evidence
is applicable in your situation:
1. Is your organization / division / population so different from
those in the study that its results cannot apply?
2. How relevant is the study to what you are seeking to
understand or decide?
3. What are your organization‟s potential benefits and harms from
the intervention?
4. Is the intervention feasible in your setting?

104. The 5 steps of ‘pull’ EBP
Postgraduate Course
1. Formulate an answerable question
2. Search for the best available evidence
3. Critically appraise the evidence
4. Integrate the evidence with your managerial
expertise and organisational concerns and apply
5. Monitor the outcome

105. Monitor the outcome
Postgraduate Course
 posttest?
 pretest?
 control group?

106. Do a trial!
Postgraduate Course

107. Postgraduate Course
(*). Why should academics be
interested in EBMgt?

108. Postgraduate Course
(5). Evidence based management:
What can you do
as a PhD student?

109. Barriers
Postgraduate Course

110. Barriers
Postgraduate Course
Best available evidence
=
systematic reviews

111. Postgraduate Course

112. Postgraduate Course

113. Postgraduate Course

114. Postgraduate Course
Better than a single study:
a replication study
Better than a replication study:
a systematic review / meta analysis
If there were 100 RCT‟s, 99 of which gave a „negative‟
result (where, say, the new intervention appeared to be
harmful), while one had a „positive‟ result (were the
intervention appeared helpful), it would obviously be a
mistake to consider only the single positive RCT.

115. TYPES OF LITERATURE REVIEW
Explicit systematic: Explicit use of rigorous method - can vary as
least as much as the range of methods in primary research
Implicit systematic: rigorous method but not stated
False systematic: described as systematic but with little evidence of
explicit rigorous method
Argument/thematic: a review that aims to explore and usually
support a particular argument or theme with no pretension to use
an explicit rigorous method (though thematic reviews can be
systematic)
Expert or ad hoc review: informed by the skill and experience of the
reviewer but no clear method so open to hidden bias.
Rapid evidence assessment: a rapid review that may or may not be
rigorous and systematic. If it is systematic then in order to be
rapid it is likely to be limited in some explicit aspect of scope.
(Gough 2007)
•115

116. LITERATURE REVIEWS IN
MANAGEMENT
How many here have had training in reviewing literature?
Are we really “standing on the shoulders of giants”?
Do you recognize these sort of unqualified statements?
• “Previous studies have shown that…”
• “It has been demonstrated that…”
But how many studies? Demonstrated how? Did other
studies find something else?
Very few systematic reviews in management
•116

117. WHAT QUALITIES SHOULD
LITERATURE REVIEWS HAVE?

118. WHAT QUALITIES SHOULD
LITERATURE REVIEWS HAVE?
Comprehensive? Focused?
Reader-friendly? Exploratory?
Informative? Inclusive? (of different types
Balanced? of evidence)
Insightful? Transparent?
Critical? Accurately referenced?
Rigorous? Objective?
Accessible? Replicable?
User led? Interesting?
Up-to-date? Standardized?
•118

119. WHAT IS A SYSTEMATIC
REVIEW?
It‟s research on existing research
With a clear, explicit and replicable methodology
• Clear review question
• Search strategy
• Quality criteria
Allows us to draw reliable conclusions about what we know
and do not know about a given question or problem
•119

121. Systematic review
Postgraduate Course
The intention behind a systematic review is to identify as fully
as possible all the scientific studies of relevance to a particular
subject and to assess the validity and authority of the evidence
of each study separately. As the name indicates, a systematic
review takes a systematic approach to identifying studies and
has the methodological quality critically appraised by multiple
researchers independently of each other, as a consequence of
which the review is transparent and reproducible and can be
monitored. The use of statistical analysis techniques in a
systematic review to pool the results of the individual studies
numerically in order to achieve a more accurate estimate of
the effect is termed a “meta-analysis”.

122. THE QUESTIONS SRs ANSWER
For any given specific problem:
What do we know?
What do we not know?
What are we not sure about?
How do we know we know or don‟t know or are not sure
that…?
What is the basis for our claims? (e.g., How much evidence?
What quality?)
•122

123. Stages of a SR SYSTEMATIC
REVIEW
1. problem
formulation;
2. locating
Practice- studies; What do Informs
Existing research we practice
relevant studies 3. study know?
selection Informs
question What we future
and
evaluation; do not research
know? questions
4. analysis
and
synthesis;
5. reporting of
the results

124. Stages of a systematic review
Postgraduate Course
1. Formulate a focussed review question
2. Search for the best available evidence
3. Select relevant studies
4. Critically appraise the evidence
5. Synthesise the findings
6. Report what is known

125. WHAT SORT OF QUESTIONS
CAN BE ADDRESSED IN A SR?
Each would require much more specificity
Does team-building work?
Can you improve emotional intelligence?
Do increases in EI lead to performance improvements?
Does management development improve the performance of managers?
Does employee engagement predict organizational performance?
Is 360 degree feedback effective?
Can potentially great leaders be identified?
Is coaching effective?
•125

126. Does team-building work?
How would you make this question more
specific

127. PICOC & CIMO
Postgraduate Course
P = Population C = Context
I = Intervention or factor I = Intervention or factor
C = Comparison M = Mechanism
O = Outcome O = Outcome
C = Context

128. WHAT SORT OF QUESTIONS
CAN BE ADDRESSED IN A SR?
Does team-building work?
• What is meant by „team‟? And what is not included as a „team‟?
• What kind of teams?
• In which particular contexts or settings?
• What is „team building‟? And what is not „team building‟?
• What does „work‟ mean?
• „Work‟ compared to any other team intervention? No intervention?
• What outcomes are relevant?
• What are the mechanisms, processes and theory which might
account for possible effects of team building on outcomes?
• What time periods are relevant for observing any possible effects?
• What about possible negative effects or harm?
• What types of data from what sorts of designs would in principle
provide good quality, medium quality and poor quality evidence?
•128

129. Which study for which question?
Postgraduate School
Qualitative Observational Controlled
Research question Surveys
studies studies studies
Effectiveness: does it work?, does A work
better than B? + ++
Process: how does it work, why does it work? ++ +
Context: in what circumstances does it
work, for whom? ++ + +
Safety: will it do more good than harm? + + ++
Acceptability: will the target group accept the
intervention / new method of working? ++ + +
Cost effectiveness: does it reduce costs? is A
cheaper than B? ++
Appropriateness: is this the right intervention /
method for this target group? ++ ++
Satisfaction: is the target group satisfied with
the new method of working? ++ ++ +

130. Stages of a systematic review
Postgraduate Course
1. Formulate a focussed review question
2. Search for the best available evidence
3. Select relevant studies
4. Critically appraise the evidence
5. Synthesise the findings
6. Report what is known

131. MAKING DECISIONS ABOUT
SEARCH STRATEGY
What sources of evidence?
What sources will you include or exclude and why?
How iterative can you be?
• Test the question doing some simple searches
• Does the question work?
• Does the search strategy work?
•131

132. Search strategy
Postgraduate School
Two types of search strategies
Snowball method Building blocks method

133. Snowball method
Postgraduate School
Starting from one book or article, you search
for other literature on the same topic.
Snowballing to older publications by finding out which
publications were used by the author (see bibliography of
book or article).
Snowballing to more recent publications by finding out
how often that book or article has been cited by other
authors (see Web of Knowledge).

134. Snowball method
Postgraduate School
ISI Web of Knowledge

135. Building blocks method
Postgraduate School
Keyword 1 Keyword 2 Keyword 3 Keyword 4
Synonyms or Synonyms or Synonyms or Synonyms or
related terms related terms related terms related terms
• …. • …. • …. • ….
• …. OR • …. OR • …. OR • ….
• …. • …. • …. • ….
• …. • …. • …. • ….
AND AND AND

136. Postgraduate School

137. Building blocks method
Postgraduate School

138. Building blocks method
Postgraduate School

139. Stages of a systematic review
Postgraduate Course
1. Formulate a focussed review question
2. Search for the best available evidence
3. Select relevant studies
4. Critically appraise the evidence
5. Synthesise the findings
6. Report what is known

140. Stages of a systematic review
Postgraduate Course
1. Formulate a focussed review question
2. Search for the best available evidence
3. Select relevant studies
4. Critically appraise the evidence
5. Synthesise the findings
6. Report what is known

141. Standard appraisal questions
•Postgraduate School
1. Did the study adress a clearly focused issue?
2. Is the sample size justified?
3. Is the design appropriate to the stated aims?
4. Are te measurements likely to be valid and reliable?
5. Are the statistical methods described?
6. Did untoward events occur during the study?
7. Were the basic data adequately described?
8. Do the numbers add up?
9. Was the statistical significance assessed?
10. What do the findings mean?
11. Are important effects overlooked?
12. What implications does the study have for your practice?

142. Stages of a systematic review
Postgraduate Course
1. Formulate a focussed review question
2. Search for the best available evidence
3. Select relevant studies
4. Critically appraise the evidence
5. Synthesise the findings
6. Report what is known

143. Once a body of evidence has been
collated….
How relevant is this to what we are seeking to
understand or decide?
How representative is this of the population that
concerns us?
How reliable, how well-founded theoretically,
empirically is it?
„These are tough but necessary tests for evidence based policy and
practice‟ Solesbury 2004

144. Stages of a systematic review
Postgraduate Course
1. Formulate a focussed review question
2. Search for the best available evidence
3. Select relevant studies
4. Critically appraise the evidence
5. Synthesise the findings
6. Report what is known

145. What does synthesis mean in MOS?
Analysis,
• “…is the job of systematically breaking down something into its
constituent parts and describing how they relate to each other – it is
not random dissection but a methodological examination”
• Is the aim is to extract key data, ideas, theories, concepts [arguments]
and methodological assumptions from the literature?
Synthesis,
• “…is the act of making connections between the parts identified in
analysis. It is about recasting the information into a new or different
arrangement. That arrangement should show connections and
patterns that have not been produced previously”
(Hart, 1998: p.110)

146. Summarizing evidence
What does the evidence say?
Consistency of evidence?
Quality of evidence
Quantity of evidence (avoiding double
counting)
•146

147. The example of management and
leadership development
Does leadership development work?
What do we know?
What is the „best‟ research evidence available?
How can this evidence be „put together‟?
What are the strengths and weaknesses of different
approaches to synthesis?

148. What methods of synthesis are available? (1/2)
Aggregated synthesis Meta analysis
Analytic induction Meta ethnography
Bayesian meta analysis Meta narrative mapping
Meta needs assessment
Case Survey
Meta synthesis
Comparative case study Metaphorical analysis
Constant targeted comparison Mixed method synthesis
Content analysis Narrative synthesis
Critical interpretive synthesis Quasi statistics
Realist synthesis
Cross design synthesis
Reciprocal analysis
Framework analysis Taxonomic analysis
Grounded theory Thematic synthesis
Hermeneutical analysis Theory driven synthesis
Logical analysis

149. What methods of synthesis are available? (2/2)
Synthesis by aggregation
• extract and combine data from separate studies to increase the
effective sample size.
Synthesis by integration
• collect and compare evidence from primary studies employing
two or more data collection methods.
Synthesis by interpretation
• translate key interpretations / meanings from one study to
another.
Synthesis by explanation
• identify causal mechanisms and how they operate.
•(Rousseau, Manning, Denyer, 2008)

150. Van Buren & Erskine, 2002 (building on
Kirkpatrick)
Organizations reported collecting data on:
78% reaction (how participants have reacted to the
programme)
32% learning (what participants have learnt from the
programme)
9% behaviour (whether what was learnt is being applied
on the job)
7% results (whether that application is achieving results)

151. Stages of a systematic review
Postgraduate Course
1. Formulate a focussed review question
2. Search for the best available evidence
3. Select relevant studies
4. Critically appraise the evidence
5. Synthesise the findings
6. Report what is known

152. Postgraduate School
What do you do with this?

153. Does management and leadership
development work
Overall, the results suggest a medium to
large effect size for learning and
behaviour (largely based on self
report)
Absence of evidence of impact of
business impact

154. Do the results of synthesis create clarity -
or confusion, conflict and controversy?
“A wide variety of program outcomes are
reported in the literature – some that are
effective, but others that are failing. In some
respects the lessons for practice can be found
in the wide variance reported in these studies.
The range of effect sizes clearly shows that it
is possible to have very large positive
outcomes, or no outcomes at all” (p. 240/241)
(Collins and Holton, 1996: 240/241)

155. Do the results of synthesis create clarity -
or confusion, conflict and controversy?
“Organizations should feel comfortable that their
managerial leadership development programmes will
produce substantial results, especially if they offer the
right development programs for the right people at the
right time. For example, it is important to know
whether a six-week training session is enough or the
right approach to develop new competencies that
change managerial behaviours, or it is individual
feedback from a supervisor on a weekly basis
regarding job performance that is most effective?”
(Collins and Holton, 1996: 240/241)

156. systematic review
Postgraduate Course
Example

157. Flexible working conditions and their effects on
employee health and wellbeing (Joyce et al, 2010)
Background: Flexible working conditions are increasingly popular in
developed countries but the effects on employee health and
wellbeing are largely unknown.
Objectives: To evaluate the effects (benefits and harms) of flexible
working interventions on the physical, mental and general health
and wellbeing of employees and their families.
Search strategy: Our searches (July 2009) covered 12 databases
including the Cochrane Public Health Group Specialized Register,
CENTRAL; MEDLINE; EMBASE; CINAHL; PsycINFO; Social
Science Citation Index; ASSIA; IBSS; Sociological Abstracts; and
ABI/Inform. We also searched relevant websites, hand searched
key journals, searched bibliographies and contacted study
authors and key experts.
•157

158. Flexible working conditions and their effects on
employee health and wellbeing (Joyce et al, 2010)
Selection criteria: Randomized controlled trials (RCT), interrupted
time series and controlled before and after studies (CBA), which
examined the effects of flexible working interventions on
employee health and wellbeing. We excluded studies assessing
outcomes for less than six months and extracted outcomes
relating to physical, mental and general health/ill health measured
using a validated instrument. We also extracted secondary
outcomes (including sickness absence, health service usage,
behavioral changes, accidents, work-life balance, quality of life,
health and wellbeing of children, family members and co-workers)
if reported alongside at least one primary outcome.
Data collection and analysis: Two experienced review authors
conducted data extraction and quality appraisal. We undertook a
narrative synthesis as there was substantial heterogeneity
between studies.
•158

159. Flexible working conditions and their effects on
employee health and wellbeing (Joyce et al, 2010)
Main results: Ten studies fulfilled the inclusion criteria. Six CBA studies reported on
interventions relating to temporal flexibility: self-scheduling of shift work (n = 4),
flexitime (n = 1) and overtime (n = 1). The remaining four CBA studies evaluated a
form of contractual flexibility: partial/gradual retirement (n = 2), involuntary part-time
work (n = 1) and fixed-term contract (n = 1). The studies retrieved had a number of
methodological limitations including short follow-up periods, risk of selection bias
and reliance on largely self-reported outcome data. Four CBA studies on self-
scheduling of shifts and one CBA study on gradual/partial retirement reported
statistically significant improvements in either primary outcomes (including systolic
blood pressure and heart rate; tiredness; mental health, sleep duration, sleep quality
and alertness; self-rated health status) or secondary health outcomes (co-workers
social support and sense of community) and no ill health effects were reported.
Flexitime was shown not to have significant effects on self-reported physiological
and psychological health outcomes. Similarly, when comparing individuals working
overtime with those who did not the odds of ill health effects were not significantly
higher in the intervention group at follow up. The effects of contractual flexibility on
self-reported health (with the exception of gradual/partial retirement, which when
controlled by employees improved health outcomes) were either equivocal or
negative. No studies differentiated results by socio-economic status, although one
study did compare findings by gender but found no differential effect on self-
reported health outcomes.
•159

160. Flexible working conditions and their effects on
employee health and wellbeing (Joyce et al, 2010)
Authors’ conclusions: The findings of this review
tentatively suggest that flexible working interventions
that increase worker control and choice (such as self
scheduling or gradual/partial retirement) are likely to
have a positive effect on health outcomes. In contrast,
interventions that were motivated or dictated by
organizational interests, such as fixed-term contract
and involuntary part-time employment, found equivocal
or negative health effects. Given the partial and
methodologically limited evidence base these findings
should be interpreted with caution. Moreover, well-
designed intervention studies are needed to delineate
the impact of flexible working conditions on health,
wellbeing and health inequalities.
•160