Presentatie over Evidence-Based Practice, Lunchbijeenkomst NVAO, 14 maart 2013

1. Evidence-Based Practice
Lunchbijeenkomst
14 maart 2013

3. Post Grad
HBO Universiteit (E)MBA
Business
Schools

4. Aanleiding
1. Wat moet een student in de praktijk straks kunnen:
- zelf onderzoek doen?
- onderzoek kunnen beoordelen + toepassen?
2. Wat betekent dat voor het curriculum?
3. Wat betekent dat voor de accreditatie?

5. HBO standaard (2009)
“In onze moderne samenleving is het cruciaal dat hbo-
bachelors over een onderzoekend vermogen beschikken
dat leidt tot reflectie, tot evidence-based practice, en tot
innovatie.”

6. Evidence-Based
Practice ?

7.  Wat is het?
 Waar komt het vandaan?
 Hoe ziet het er uit in een opleiding?
 Hoe zit dat bij 4e jaars studenten?

8. Evidence based practice:
Wat is het?

9. Evidence-based practice
Uitgangspunt bij evidence-based practice is
dat beslissingen gebaseerd dienen te zijn
op een combinatie van wetenschappelijk
denken en de best beschikbare 'evidence'.

10. Evidence based practice
Met het begrip 'evidence' wordt niet
meer bedoeld dan 'informatie'.
Dit kan informatie zijn afkomstig uit
wetenschappelijk onderzoek, maar ook
interne bedrijfsinformatie en zelfs
persoonlijke ervaring geldt als
'evidence’.

11. Evidence based practice
In principe neemt iedere manager dus
beslissingen op basis van 'evidence'.
De meeste managers besteden echter
nauwelijks aandacht aan de kwaliteit
van de 'evidence' waarop ze hun
beslissingen baseren.

12. Evidence-based practice:
 kritisch en wetenschappelijk denken
 van verschillende informatiebronnen
gebruik maken
 de beschikbare evidence kritisch tegen het
licht houden
 denken in termen van waarschijnlijkheid
in plaats van 'golden bullets'.

13. Evidence based practice
Best available Professional
scientific evidence expertise and
judgement
Evidence-based
decision
Best available Stakeholders’
organizational
values and concerns
evidence

14. Evidence based practice:
Waar komt het vandaan?

15. What field is this?
 “there is a large research-user gap”
 “practitioners do not read academic journals”
 “the findings of research into what is an effective intervention
are not being translated into actual practice”
 “academics not practitioners are driving the research agenda”
 “the relevance, quality and applicability of research is
questionable”
 “practice is being driven more by fads and fashions than
research”
 “many practices are doing more harm than good”

16. Medicine: Founding fathers
David Sackett Gordon Guyatt
McMaster University Medical School, Canada

17. How it all started

18. Problem I: persistent convictions
if you’re
breathe into a bag
hyperventilating

19. Problem I: persistent convictions
elderly people who have give them a drug that
an irregular heartbeat are reduces the
much more likely to die of number of
coronary disease irregular beats

20. How 40,000 cardiologists can be wrong
In the early 1980s newly introduced
anti-arrhythmic drugs were found to be
highly successful at suppressing
arrhythmias.
Not until a RCT was performed was it
realized that, although these drugs
suppressed arrhythmias, they actually
increased mortality.
By the time the results of this trial were
published, at least 100,000 such
patients had been taking these drugs.

21. David Sackett
 Half of what you learn in medical school will be
shown to be either dead wrong or out-of-date
within 5 years of your graduation; the trouble is that
nobody can tell you which half.
 The most important thing to learn is how to learn
on your own: search for the evidence!
 (Remember that your teachers are as full of bullshit
as your parents)

22. Problem II: too much information
 More than 1 million articles in 40,000 medical journals per
year (= 1995; now probably more than 2 million). For a
specialist to keep up this means reading 25 articles every
day (for a GP more than 100!)

23. Problem II: too much information
 HRM: 1,350 articles in 2010 (ABI/INFORM). For an HR
manager to keep up this means reading 3 to 4 articles
every day (for a „general‟ manager more than 50!)
BTW: most of the research is seriously
flawed or irrelevant for practice

24. The 5 steps EBP
1. Formulate a focused question (Ask)
2. Search for the best available evidence (Acquire)
3. Critically appraise the evidence (Appraise)
4. Integrate the evidence with your professional
expertise and apply (Apply)
5. Monitor the outcome (Assess)

25. Evidence-Based Practice
1991 Medicine
1998 Education
1999 Social care, public policy
2000 Nursing
2000 Criminal justice
???? Management?

26. Evidence based practice:
Hoe ziet het er uit in de opleiding?

27. Evidence-based practice
Fase 1: Het ontwikkelen van kritisch en wetenschappelijk
denken dat leidt tot een professioneel-kritische attitude met
betrekking tot organisatievraagstukken
Fase 2: Het kunnen formuleren van een expliciete vraag en
op basis van deze vraag kunnen zoeken in online databases
naar uitkomst van relevant wetenschappelijk onderzoek.
Fase 3: Het kritisch kunnen beoordelen van
wetenschappelijke en organizational evidence (critical
appraisal)
Fase 4: Uitkomst van wetenschappelijk onderzoek kunnen
toepassen in de eigen beroepspraktijk (application of science)

28. Evidence based practice
Best available Professional
scientific evidence expertise and
judgment
Evidence-based
decision
Best available Stakeholders’
organizational
values and concerns
evidence

29. Waarom (wetenschappelijk) onderzoek?

30. Trust me, 20 years of experience

31. Bounded rationality

32. Richard Feynman
“The first principle is that you must not fool
yourself - and you are the easiest person to
fool”.

33. Bounded rationality

34. Het feilbare brein
System 1
 Snel, actie
 Intuitief, associatief
 shortcuts & biasses
System 2
 Langzaam (lui!)
 Rationeel
 Nadenken

35. Systeem 1

36. Systeem 1: het feilbare brein
 Seeing order in randomness
 Mental corner cutting
 Misinterpretation of incomplete data
 Halo effect
 False consensus effect
 Group think  Confirmation bias
 Self serving bias  Authority bias
 Sunk cost fallacy  Small numbers fallacy
 Cognitive dissonance reduction  In-group bias
 Recall bias
 Anchoring bias
 Inaccurate covariation detection
 Distortions due to plausibility

37. Waarom onderzoek?
Het feilbare brein
- Meningen
- Assumpties (aannames)
- Overtuigingen
BIAS
- Persoonlijke ervaringen
Wetenschappelijk onderzoek
- Feiten
- Evidence: bewijs / aanwijzingen

38. Fase 1: Kritisch & wetenschappelijk denken
1. Denkfouten
2. Informatiebronnen
3. Mythbusting
4. Assumpties

39. 7 Denkfouten (die u beter aan anderen kunt overlaten)
1. Seeing order in randomness
2. Confirmation bias
3. Small numbers fallacy
4. Outcome bias
5. Halo effect
6. Authority bias
7. Group think

40. Seeing order in randomness
 Een Type I fout of een vals positief:
denken dat er een patroon / verband is
terwijl dat er in het echt niet is.
 Een Type II fout of een vals negatief:
denken dat er geen patroon / verband is
terwijl dat er in het echt wel is
Dr. Michael Shermer
(Director of the Skeptics Society)

41. Het feilbare brein: patern recognition
 Een Type I fout of een vals positief: denken dat het
geritsel in de bosjes een gevaarlijk roofdier is, terwijl
het gewoon de wind is (goedkoop foutje)

42. Het feilbare brein: patern recognition
 Een Type II fout of een vals negatief: denken dat het
geritsel in de bosjes gewoon de wind is, terwijl het een
gevaarlijk roofdier is (duur foutje)

43. Het feilbare brein: patern recognition
Het probleem met patroon herkenning:
Het kritisch beoordelen of er sprake is van een
Type I of een Type II fout is best moeilijk, (vooral
in „split second life and death‟ situaties), dus de
default positie is om aan te nemen dat alle
patronen echt zijn.

44. Het feilbare brein: patern recognition
 Een Type I fout of een vals positief: denken dat het
geritsel in de bosjes een gevaarlijk roofdier is, terwijl
het gewoon de wind is (goedkoop foutje)
 Een Type II fout of een vals negatief: denken dat het
geritsel in de bosjes gewoon de wind is, terwijl het een
gevaarlijk roofdier is (duur foutje)

45. Het feilbare brein: patern recognition
Ook ervaren mensen en experts zien patronen
en verbanden waar ze niet zijn.
stress & lifestyle peptic ulcer

46. Oct 2005
Peptic ulcer – an infectious disease!
This year's Nobel Prize in Physiology or Medicine goes to Barry Marshall and Robin
Warren, who with tenacity and a prepared mind challenged prevailing dogmas. By
using technologies generally available (fibre endoscopy, silver staining of
histological sections and culture techniques for microaerophilic bacteria), they
made an irrefutable case that the bacterium Helicobacter pylori is causing disease.
By culturing the bacteria they made them amenable to scientific study.
In 1982, when this bacterium was discovered by Marshall and Warren, stress and
lifestyle were considered the major causes of peptic ulcer disease. It is now
firmly established that Helicobacter pylori
causes more then 90% of duodenal ulcers.
The link between Helicobacter pylori
infection and peptic ulcer disease has been
established through studies of human
volunteers, antibiotic treatment studies and
epidemiological studies.

47. Het feilbare brein: patern recognition
Ook slimme mensen houden er verkeerde ideeën op
na, niet omdat ze dom of eigenwijs zijn, maar omdat het
de meest logische conclusie is op basis van hun eigen
ervaringen.
(systeem 1 doet altijd mee!)

48. Evidence-based practice
Fase 1: Het ontwikkelen van kritisch en wetenschappelijk
denken dat leidt tot een professioneel-kritische attitude met
betrekking tot organisatievraagstukken
Fase 2: Het kunnen formuleren van een expliciete vraag en
op basis van deze vraag kunnen zoeken in online databases
naar uitkomst van relevant wetenschappelijk onderzoek.
Fase 3: Het kritisch kunnen beoordelen van
wetenschappelijke en organizational evidence (critical
appraisal)
Fase 4: Uitkomst van wetenschappelijk onderzoek kunnen
toepassen in de eigen beroepspraktijk (application of science)

50. 5-step approach
EBMgt is a 5-step approach
1. Formulate an answerable question (PICOC)
2. Search for the best available evidence
3. Critically appraise the quality of the found evidence
4. Integrate the evidence with managerial expertise
and organizational concerns and apply
5. Monitor and evaluate the results

51. Formulate a focused question

52. Focused question?
 Does team-building work?
 What are the costs and benefits of self-steering teams?
 What are the success factors for culture change?
 Does management development improve the
performance of managers?
 Does employee participation prevent resistance to
change?
 How do employees feel about 360 degree feedback?

53. Foreground question?
 Does team-building work?
 What is a „team‟?
 What kind of teams?
 In what contexts/settings?
 What counts as „team-building‟?
 What does „work‟ mean?
 What outcomes are relevant?
 Over what time periods?

54. Answerable question: PICOC
P = Population
I = Intervention or success factor
C = Comparison
O = Outcome
C = Context

55. 2. Finding the best available evidence

56. Searching evidence
What do we search?

57. What do we search?
Current Information
Overview of a subject
General background
Academic Information
Statistical Information
Theories about a subject
Company information

58. Peer reviewed journals

59. Searching evidence
Where do we search?

60. Searching for evidence

61. Databases
 ABI/INFORM
 Business Source Elite
 PsycINFO
 Web of Knowledge
 ERIC
 Google Scholar

62. Searching for evidence

63. Searching evidence
How do we search?
Search Strategy

64. Search strategy
Two types of search strategies
Snowball method Building blocks method

65. Search strategy

66. Exercise: Search for evidence
Search in ABI/Inform:
1. How many articles has Stephen Covey published in peer reviewed journals?
2. How many of these articles are based on scientific research?
3. Are there articles (by other authors) that are critical of Covey’s 7 Habits?
4. How many of these critical articles are based on scientific research?

67. Exercise: Search for evidence
Search in ABI/Inform or BSE:
 Search for peer reviewed research articles to answer the following
question: What is the long term effect of a hostile take-over on the
financial performance of the acquired organization? Use the following
search terms:
“hostile takeovers”, “financial performance”, “long term”
How many studies did you find?

68. Evidence-based practice
Fase 1: Het ontwikkelen van kritisch en wetenschappelijk
denken dat leidt tot een professioneel-kritische attitude met
betrekking tot organisatievraagstukken
Fase 2: Het kunnen formuleren van een expliciete vraag en
op basis van deze vraag kunnen zoeken in online databases
naar uitkomst van relevant wetenschappelijk onderzoek.
Fase 3: Het kritisch kunnen beoordelen van
wetenschappelijke en organizational evidence (critical
appraisal)
Fase 4: Uitkomst van wetenschappelijk onderzoek kunnen
toepassen in de eigen beroepspraktijk (application of science)

69. Research designs
Which design for which question?

70. What is the best design?
 Randomized controlled study?
 Grounded theory approach?
 Cohort / panel study?
 Qualitative field research?
 Longitudinal study?
 Post-test only study?
 Survey?
 Action research?
 Case study?

71. What is the BEST car?

72. What is the best design?
quants vs quallies, positivists vs post structuralist, etc

74. Effect vs Non-effect

75. Types of questions
Does it work?
Does it work better than ....?
Does it have an effect on ....?
Effect
What is the success factor for ....?
What is required to make it work ...?
Will it do more good than harm?

76. Types of questions: non-effect
Needs: What do people want or need?
Attitude: What do people think or feel?
Experience: What are peoples’ experiences?
Prevalence: How many / often do people / organizations ...?
Procedure: How can we implement ...?
Process: How does it work?
Explanation: Why does it work?
Economics: How much does it cost?

77. Internal validity

78. Internal validity
internal validity = indicates to what extent the
results of the research may be biased and is thus
a comment on the degree to which alternative
explanations for the outcome found are possible.

80. Causal relations
We are pattern seeking primates:
we are predisposed to see order
and causal relations in the world

81. Causality
Considerations for research:
1. Are the "cause" and the "effect” related?
effect size
1. Does the "cause" precede the "effect" in time?
before and after measurement
2. Are there no plausible alternative explanations for
the observed effect?
randomization, blinding, control group, measurements

82. Bias & Confounding
Research shows:
Shoe size > quality of handwriting
Smoking youngsters > better lung function

83. Levels of internal validity

84. Which design for which question?
Explanation

85. Different types of research questions
require different types of research designs,
but ...

86. But 1: feasibility
Best research design?

87. But 1: feasibility

88. Step 3: Critical appraisal of studies

89. Intermezzo
How to read a research article?

90. Critical appraisal: quick and dirty
Is the study design appropriate to the stated
aims?
Are the measurements likely to be valid and
reliable?
Was there a relevant effect size?
Is the outcome (population, type of organization)
generalizable to your situation?

91. Levels of internal validity
1. Were there enough subjects in the study?
2. Was a control group used?
3. Were the subjects randomly assigned?
4. Was a pretest used?
5. Was the study started prior to the intervention or event?
6. Was the outcome measured in an objective and reliable way?
6x yes = very high (A)
5x yes = high (A)
4-3x yes = limited (B)
2x yes = low (C)
1-0x yes = very low (D)

92. Appraisal
Critical appraisal questionnaires
www.cebma.org/ebp-tools

93. CAT: Critically Appraised Topic

94. CAT: Critically Appraised Topic
A critically appraised topic (or CAT) is a structured, short (3
pages max) summary of evidence on a topic of interest,
usually focused around a practical problem or question. A
CAT is like a “quick and dirty” version of a systematic review,
summarizing the best available research evidence on a topic.
Usually more than one study is included in a CAT.

95. CAT: structure
1) Background / context
2) Question (PICOC)
3) Search strategy
4) Results / evidence summary
5) Findings
6) Limitations
7) Recommendation

96. CAT-walk