VOV HR Network Belgium Workshop Evidence-Based HR Management: Searching & Finding The best Available Scientific Evidence January 11, 2013, Anterp, Belgium

1. Pit stop EBMgt
Evidence-Based Management
(Searching & Finding Scientific Evidence)
Pitstop Evidence-Based HR, VOV lerend netwerk,
Antwerpen, 11 januari 2013

2. Pit stop EBMgt
1. Evidence-Based Management
A (very) Short Introduction

3. Medicine: Founding fathers
Pit stop EBMgt
David Sackett Gordon Guyatt
McMaster University Medical School, Canada

4. How it all started
Pit stop EBMgt

5. Problem I: too much information
Pit stop EBMgt
 More than 1 million articles in 40,000 medical journals per
year (= 1995; now probably more than 2 million). For a
specialist to keep up this means reading 25 articles every
day (for a GP more than 100!)
 Most of the new insights and treatment
methods don‟t reach the target group

6. Problem I: too much information
Pit stop EBMgt
 HRM: 1,350 articles in 2010 (ABI/INFORM). For an HR
manager to keep up this means reading 3 to 4 articles
every day (for a „general‟ manager more than 50!)

7. Problem II: persistent convictions
Pit stop EBMgt
if you’re
breathe into a bag
hyperventilating

8. Problem III: jumping to conclusions
Pit stop EBMgt
elderly people who have give them a drug that
an irregular heartbeat are reduces the
much more likely to die of number of
coronary disease irregular beats

9. How 40,000 cardiologists can be wrong
Pit stop EBMgt
In the early 1980s newly introduced anti-
arrhythmic drugs were found to be highly
successful at suppressing arrhythmias.
Not until a RCT was performed was it realized
that, although these drugs suppressed
arrhythmias, they actually increased mortality.
The CAST trial revealed Excess mortality of
56/1000.
By the time the results of this trial were
published, at least 100,000 such patients had
been taking these drugs.

10. Pit stop EBMgt
David Sackett
 Half of what you learn in medical school will be
shown to be either dead wrong or out-of-date
within 5 years of your graduation; the trouble is that
nobody can tell you which half.
 The most important thing to learn is how to learn
on your own.
 (Remember that your teachers are as full of bullshit
as your parents)

11. Evidence-Based Practice
Pit stop EBMgt
1991 Medicine
1998 Education
1999 Social care
2000 Nursing
2000 Criminal justice
???? Management?

12. Pit stop EBMgt
2. Evidence based management:
What is it?

13. Definition
Pit stop EBMgt
Evidence based practice:
Improve information to support decision making

14. Definition
Pit stop EBMgt
Evidence-based management means making decisions
about the management of employees, teams or
organizations through the conscientious, explicit and
judicious use of four sources of information:
1. The best available scientific evidence
2. Organizational facts, metrics and characteristics
3. Stakeholders’ values and concerns
4. Practitioner expertise and judgment

15. Four sources
Pit stop EBMgt

16. Pit stop EBMgt
Evidence-based practice:
 Focuses on the decision making process
 Thinks in terms of probability (instead of
golden bullets).

17. Pit stop EBMgt
3. Evidence-based management:
Why do we need it?

18. Four sources
Pit stop EBMgt

19. Pit stop EBMgt
Trust me: 20 years of management experience!

20. Bounded rationality
Pit stop EBMgt

21. Bounded rationality
Pit stop EBMgt
System 1
 Fast
 Intuitive, associative
 heuristics & biases
System 2
 Slow (lazy)
 Deliberate, „reasoning‟
 Rational

22. System 1: prone to biases
Pit stop EBMgt
 Seeing order in randomness
 Mental corner cutting
 Misinterpretation of incomplete data
 Halo effect
 False consensus effect
 Group think  Confirmation bias
 Self serving bias  Authority bias
 Sunk cost fallacy  Small numbers fallacy
 Cognitive dissonance reduction  In-group bias
 Recall bias
 Anchoring bias
 Inaccurate covariation detection
 Distortions due to plausibility

23. Pit stop EBMgt
2. Searching & Finding
The Best Available Scientific Evidence

24. Pit stop EBMgt
Step 1: Formulate a focused question

25. 5-step approach
Pit stop EBMgt
EBMgt is a 5-step approach
1. Formulate an answerable question (PICOC)
2. Search for the best available evidence
3. Critical appraise the quality of the found evidence
4. Integrate the evidence with managerial expertise
and organizational concerns and apply
5. Monitor and evaluate the results

26. Asking the right question?
Pit stop EBMgt
Effect vs Non-effect

27. Types of questions: effect
Pit stop EBMgt
Does it work?
Does it work better than ....?
Does it have an effect on ....?
Effect
What is the success factor for ....?
What is required to make it work ...?
Will it do more good than harm?

28. Types of questions: non-effect
Pit stop EBMgt
Needs: What do people want or need?
Attitude: What do people think or feel?
Experience: What are peoples’ experiences?
Prevalence: How many / often do people / organizations ...?
Procedure: How can we implement ...?
Process: How does it work?
Explanation: Why / how does it work?
Economics: How much does it cost?

29. Focused question?
Pit stop EBMgt
 Does team-building work?
 What are the costs and benefits of self-steering teams?
 What are the success factors for culture change?
 Does management development improve the
performance of managers?
 Does employee participation prevent resistance to
change?
 How do employees feel about 360 degree feedback?

30. Foreground question?
Pit stop EBMgt
 Does team-building work?
 What is a „team‟?
 What kind of teams?
 In what contexts/settings?
 What counts as „team-building‟?
 What does „work‟ mean?
 What outcomes are relevant?
 Over what time periods?

31. Answerable question: PICOC
Pit stop EBMgt
P = Population
I = Intervention (or success factor)
C = Comparison
O = Outcome
C = Context

32. Example: merger
Pit stop EBMgt

33. Answerable question: PICOC
Pit stop EBMgt
Imagine you are a consultant, your client is the board of
directors of a large Canadian health-care organization.
The board of directors has plans for a merger with a
smaller healthcare organization in a nearby town.
However, it‟s been said that the organizational culture
differs widely between the two organizations. The board
of directors asks you if this culture-difference can impede
a successful outcome of a merger. Most of them
intuitively sense that cultural differences matter, but they
want an evidence-based advice.

34. Answerable question: PICOC
Pit stop EBMgt
What else would you like to know?

35. Answerable question: PICOC
Pit stop EBMgt
P: What kind of Population are we talking about? Middle managers,
back-office employees, medical staff, clerical staff?
O: What kind of Outcome are we aiming for? Employee productivity,
return on investment, profit margin, competitive position, innovation
power, market share, customer satisfaction?
P/C: And how is the assumed cultural difference assessed? Is it the
personal view of some managers or is it measured by a validated
instrument?

36. Answerable question: PICOC
Pit stop EBMgt
According to the board the objective of the merger is to
integrate the back-office of the two organizations (ICT,
finance, purchasing, facilities, personnel administration,
etc.) in order to create economy of scale. The front
offices and primary process of the two organizations will
remain separate.
The cultural difference is not objectively assessed (it is
the perception of the senior managers of both
organizations).

37. Answerable question: PICOC
Pit stop EBMgt
P = back office employees in a healthcare organisation
I = merger, integration back office
C = status quo
O = economy of scale
C = different organizational culture, unequal

38. Exercise
Pit stop EBMgt
1. Read the following five scenario‟s
2. Formulate on the basis of each scenario a focused
question (use the PICOC format).

39. Scenario 2
Pit stop EBMgt
“Ik werk als manager bij een grote financiële dienstverlener. Het
concern is een holdingmaatschappij met daaronder een aantal divisies
met een eigen P&L verantwoordelijkheid. Het accountmanagement van
de divisies wordt aangestuurd op basis van een eigen set kritische
prestatie indicatoren (kpi‟s) die niet onderling is afgestemd.
Het gevolg is dat de accountmanagers elkaar regelmatig in de wielen
rijden bij gezamenlijke klanten en de omzet wordt „weggekaapt‟ bij de
ander. Kortom, er wordt onvoldoende samengewerkt. Nu wordt door de
Executive Board van de holding een beroep gedaan op de
divisiedirecties om meer met elkaar te gaan samenwerken.
Mijn vraag is: staat het feit dat de divisies geen gedeelde kpi‟s hebben
een goede interne samenwerking in de weg of hoeft dit geen probleem
te zijn?”

40. Scenario 3
Pit stop EBMgt
“I am a Dutch quality manager at a large international brewery. In
the past 4 years I facilitated a lean management program that is
running for more than 10 years in this organization. Since a
couple of years, however, we see a decline in results, such as:
 The number of problem analysis and improvement teams has
been increased, but the amount of breakdowns and short stops
in the production lines has not decreased.
 There is slow progress for the total program for the
Netherlands at this moment.
According to the middle managers the improvement program
does not address really important topics. I would therefore like to
know how our employees feel about lean management.”

41. Pit stop EBMgt
1. Neem een actueel probleem / vraag / issue dat speelt in de
organisatie waar je stage hebt gelopen (of een vraagstuk
dat je persoonlijke interesse heeft).
2. Formuleer op basis daarvan een onderzoeksvraag
3. Geef aan wat voor type vraag het is (effect of non-effect)
4. Benoem de PICOC elementen.

42. Pit stop EBMgt
Step 2: Search for the best available
evidence

43. 5-step approach
Pit stop EBMgt
EBMgt is a 5-step approach
1. Formulate an answerable question (PICOC)
2. Search for the best available evidence
3. Critical appraise the quality of the found evidence
4. Integrate the evidence with managerial expertise and
organizational concerns and apply
5. Monitor and evaluate the results

44. Searching evidence
Pit stop EBMgt
What do we search?

45. Information sources
Pit stop EBMgt
Type of Information Source
Wall street Journal, Financial Times,
Current Information Business week, Financieel Dagblad
Overview of a subject Textbooks and popular books
General background Encyclopedias, yearbooks & book reviews
ABI/INFORM, Business Source Premier,
Academic Information Emerald, PsychInfo, Science Direct
Statistical Information CBS Statline, Eurostat
Theories about a subject Textbooks and encyclopedias
Company Annual Reports, Datastream,
Company information Factiva.com, Amadeus

46. Articles in peer reviewed journals
Pit stop EBMgt

47. Searching for evidence
Postgraduate Course

48. Databases
Postgraduate Course
 ABI/INFORM
 Business Source Elite
 PsycINFO
 Web of Knowledge
 ERIC
 Google Scholar

49. Searching for evidence
Postgraduate Course

50. Searching evidence
Postgraduate Course
How do we search?
Search Strategy

51. Search strategy
Pit stop EBMgt
Two types of search strategies
Snowball method Building blocks method

52. Snowball method
Pit stop EBMgt
Starting from one book or article, you search
for other literature on the same topic.
 Snowballing to older publications by finding out which
publications were used by the author (see bibliography of
book or article).
 Snowballing to more recent publications by finding out
how often that book or article has been
cited by other authors (see Web of
Knowledge or Google Scholar).

53. Snowball method
Pit stop EBMgt

54. Building blocks method
Pit stop EBMgt
Keyword 1 Keyword 2 Keyword 3 Keyword 4
Synonyms or Synonyms or Synonyms or Synonyms or
related terms related terms related terms related terms
• …. • …. • …. • ….
• …. OR • …. OR • …. OR • ….
• …. • …. • …. • ….
• …. • …. • …. • ….
AND AND AND

55. Answerable question: PICOC
Pit stop EBMgt
P = back office employees, health care
I = merger, integration, back office
C = status quo
O = economy of scale
C = different organizational culture, unequal
1. Underline the keywords
2. Number the order of importance from 1-4

56. Answerable question: PICOC
Pit stop EBMgt
P = back office employees 5. healthcare
I = 1. merger, 3. integration, back office
C = status quo
O = 4. economy of scale
C = different 2. organizational culture, unequal
1. Underline the keywords
2. Number the order of importance

57. Search terms
Pit stop EBMgt
Operationalise your Pico elements!
O = long term profitability?

58. Construct validity
Pit stop EBMgt
The Ambidextrous Organization (??)

59. Select keywords
Pit stop EBMgt
The keywords of your PICOC may be enough.
If not, select more words by using:
 synonyms
 alternate spelling, translations
 related terms / words / subjects
 narrower or broader terms
corporate culture: organizational behavior/character, corporate identity
merger: acquisition, take-over, fusion, combination, unification
profitability: profit, advantage, return on investment, shareholder value

60. Search in the following order
Pit stop EBMgt
1. Thesaurus (SU)
2. Title (TI)
3. Abstract (AB)
4. Alternate spellings, synonyms

61. Search Query: an example
Pit stop EBMgt
I I O C
4. Health care 2. Corporate
1. Merger 3. Integration
organization culture
• Merger • Integration • Healthcare • Corporate culture
• Fusion organization
• Organizational
• Non profit
• Combination behavior
• Not for profit
• Take over OR OR OR • Organizational
• Acquisition character
• Unification • Corporate
•… identity
AND AND • Core beliefs
• Shared values

62. Boolean operators
Pit stop EBMgt
apples AND oranges apples OR oranges
apples oranges apples oranges

63. Start up
Pit stop EBMgt

64. Always search for indexed topics
or thesaurus terms
Pit stop EBMgt

65. thesaurus terms
Pit stop EBMgt

66. Combining results
Pit stop EBMgt
Use the history function to combine results

67. Combining results
Pit stop EBMgt
Use the history function to combine results

68. Narrow your results
Pit stop EBMgt

69. Narrow your results
Pit stop EBMgt

70. Save your results
Pit stop EBMgt

71. If an article is not available in ABI/ Inform or
BSE: try Google (advanced search)
Pit stop EBMgt

72. Summary
Pit stop EBMgt
1. Formulate a focused question and break it down into keywords (PICOC),
2. Underline the 2 to 4 most relevant keywords,
3. Start with the most relevant keyword: is it listed in the thesaurus? If not: search in
the title or abstract.
4. Repeat your search with alternative terms, synonyms or alternate spellings.
5. Combine the outcome of step 4 (thesaurus), 5 (title or abstract) and 6 (alternative
terms) with OR (use the history function!)
6. Repeat above steps for keyword 2
7. Combine the outcome of your search with keyword 1 and 2 with AND (use the
history function!)
8. Filter the articles with “studies” (subject > studies)
9. Read the titles and abstracts of your results, select the relevant articles and safe
them.
10. No full text available in ABI/Inform or BS Elite? Try Google Advanced.

73. Pit stop EBMgt
Learning through play !
 Try all buttons
 Make lots of mistakes
 Have fun !

74. Assignment: search strategy
Pit stop EBMgt
Search for research articles on teambuilding

75. Exercise: Search for evidence
Pit stop EBMgt
2. Search in ABI/Inform:
1. How many articles has Stephen Covey published in peer reviewed journals?
2. How many of these articles are based on scientific research?
3. Are there articles (by other authors) that are critical of Covey’s 7 Habits?
4. How many of these critical articles are based on scientific research?

76. Pit stop EBMgt
Evidence-based management
Step 3: Critical appraisal of studies

77. 5-step approach
Pit stop EBMgt
EBMgt is a 5-step approach
1. Formulate an answerable question (PICOC)
2. Search for the best available evidence
3. Critically appraise the quality of the found evidence
4. Integrate the evidence with managerial expertise
and organizational concerns and apply
5. Monitor and evaluate the results

78. Pit stop EBMgt
Effect vs Non-effect

79. Pit stop EBMgt
Internal validity

80. Internal validity
Pit stop EBMgt
internal validity = indicates to what extent the
results of the research may be biased and is thus
a comment on the degree to which alternative
explanations for the outcome found are possible.

81. Levels of internal validity
Pit stop EBMgt

82. Which design for which
Pit stop EBMgt
question?
Explanation

83. Pit stop EBMgt
Best research design?

84. Critical appraisal
Pit stop EBMgt

85. Critical appraisal
Postgraduate EBMgt
Pit stop Course
1. Is the study design appropriate to the stated aims?
2. Was a control group used?
3. Was a pretest used?
4. Are the measurements likely to be valid and reliable?
5. Could bias or confounding have occurred?
6. How large was the effect size?

86. Levels of internal validity
Pit stop EBMgt
1. Were there enough subjects in the study?
2. Was a control group used?
3. Were the subjects randomly assigned?
4. Was a pretest used?
5. Was the study started prior to the intervention or event?
6. Was the outcome measured in an objective and reliable way?
6x yes = very high (A)
5x yes = high (A)
4-3x yes = limited (B)
2x yes = low (C)
1-0x yes = very low (D)