Implementation of an Evaluation Model for Evaluating Complex Health Research Outcomes
1. Implementation of an Evaluation Model for Evaluating Complex
Health Research Outcomes
American Evaluation Association November 1-5 2011
Kathryn Graham (PhD), Heidi Chorzempa (MSc)
3. Our Organization and Mandate
•Alberta Innovates Health Solutions has a
three decades long history of supporting
medical/health research
•“Support, for the economic and social
well-being of Albertans, health research
and innovation activities aligned to meet
Government of Alberta priorities,
including, without limitation, activities
directed at the development and growth of
the health sectors, the discovery of new
knowledge and the application of that
knowledge”.
4. Why We Evaluate Health Research?
“…..economic and social well-being….
Source: Canadian Academy of Health Sciences 2009
5. Implementation Approach and Considerations
Inputs Results
TIMING – SHORT TO LONG TERM
TIMING – SHORT TO LONG TERM
1. Evidence-Based Practice
• Environmental scans Organizational
• Theoretical & Conceptual Models (CAHS) and Health Socio-
• Retrospective Data Mine Studies Research
• Prospective reporting Economic
Inputs,
• Psychometric Testing
Processes, Impact
Reach, Outputs Categories
2. Practice-Based Evidence
• Integated into the business model & Outcomes
Missi • Stakeholder/Actors need, values, use
Organiz • Evaluation tools e.g. collaborative logic models
3. Top Down and or Bottom Up Indicators & Indicators &
• Policy/Priority Driven and or Discovery Driven
Metrics Metrics
4. Context
• Strategy, External Factors, Clusters, Domains
5. Levels of Aggregation
Macro, Meso, Micro
Mixed Methodology Approach
6. Target Audience
• Tailor results to stakeholder needs & values
6. Drivers for Retrospective Study
• Verification: Using CAHS - does it make sense? Were
the impact categories, sub categories and measures
meaningful?
• Applying Theory to Practice “in Context” - Alberta’s
story
• Evidence Informed Decision Making: Lessons learned
informing design of new AIHS programs
7. Retrospective Data Mine Study
242 health researchers supported
Pillar I: Pillar II: Pillar III: Pillar IV:
Biomedical Clinical Health Population
Research Research Services Health
• Scholar • Clinical • Scholar • Population
• Scientist Investigator • Scientist Health
• Sr. Scholar • Sr. Scholar Investigator
Program Duration 5 3+3 5 3+3
years years years years
8. How we did it….
• A retrospective review of finalized grants from 2004-2008
• Population of a database
• Review 855 annual reports across 8 types of personnel grants
• Recognizing project limitations:
-Attribution and contribution
-Self-report
-Data variability
-Outcome underestimation
-Timing of measurement
-Retrofitting
9. Retrofitting the data to the model
Annual Report Template
Advancing Capacity Informing Health Socioeconomic
Knowledge Building Decision Impact Impact
Making
# trainees # of collaborations # of diagnostic/
# publications # patents
- Grad students prognosis
# of influences in advancements
# book - Post Docs # spin-off
- Summer students guidelines
# of treatment companies
chapters - Other trainees # of influences in advancements
policies # products in
# abstracts * Infrastructure # of access development
# of media citations improvements
# of invited * External PI funding # of influences in # of safety
amount public education
talks improvements
10. Impact Summary
Research Diffusion
100%
Research That produces That influences That affects That contribute
activity results decision making healthcare and to changing
in…. health risk factors health, social
90%
21% National/ Global Research 11 collaborations and economic
with Health wellbeing
15% Authorities
Health Care
Knowledge
Research Results
100% Alberta •42 improvements
1196 Collaborations with other Researchers
•5 guidelines
Pool to health care
Innovates
effectiveness
Health Improvements
•3833 25 collaborations •8 improvements to
Solutions health care in health and
publications with Industries
efficiency & wellbeing
•Across 4 •4398 •17 products accessibility •Qualitative
Research Pillars abstracts developed Results
Advancing Improvements in:
Health •386 book •30 therapeutics
Knowledge
Status and chapters •12 diagnosis/
17 collaborations •prognosis
Capacity function, •2789
with Government techniques
Building well-being, invited talks
economic
•4 policies Economic &
conditions •3 data services
Informing Research Capacity Social Impact
Decision •$217,381 in attracted Determinants of •107 patents
Making funding Health •10 products in
18 collaborations •Personal development
•990 trainees on with Non-profits behaviour •5 spin-off companies
Health average per award year •Social/cultural
Impact •8 guidelines determinants
•14 new laboratories •Environmental
Socioeconomic The Public
Impact •16 media events
Impacts feed back into inputs for future
research
11. Timeline of Innovation 2001-2008
“We were the first lab in the world
to describe the functional “In addition, these studies showed for
consequences of mutations in T-type the first time, that chimerism
channels linked to childhood absence “The second paper in J Neurophysiol was cited by the induces a dominant form of humoral
epilepsy in humans” Faculty of 1000 Biology as highly innovative. This tolerance, allowing B cell tolerance to
Sr. Scholar, year 3 paper explained why thalamic stimulation is not multiple donors.” Bio. Scholar, year 5
“I have designed a new approach where a target gene can
broadcast widely to the motor cortex and why DBS does
be cloned into a large number of expression vectors by Significance: II found differences in the
not disrupt motor control. It also helped explain the Significance: II showed that a combined
combinatorial assembly of the target gene with a limited calcium channels in epileptic patients
mechanism of action of subcortical DBS for severe protein can improve tolerance of donated
number of pre-made DNA fragments.” Scholar, year 2 compared to non-epileptics. This could
refractory depression. “ pancreatic tissue, which could improve
elucidate the pathophysiology of epilepsy
CI, year 4 long-term success rates of islet
Significance: II created a procedure that can disperse as well as potential therapeutic venues.
many copies of one gene with limited resources. This transplantation in diabetics.
Significance: Helps explain how deep brain
could be used to increase research productivity (for
stimulation works, which could lead to
example, placing the gene responsible for insulin in
many different cell types to try to determine the cell that improvements on this treatment modality.
produces insulin the most efficiently.
2001 2002 2003 2004 2005 2006 2007 2008
“This lab contains state-of-the-art computing “The data in this manuscript were the first to show that
infrastructure… and likely the first in a medical isoprostanes have the ability to modulate inflammatory
imaging lab in Canada)” Bio.Scholar, year 2. processes with relevance to airway respiratory diseases. “
Sr. Scholar, year 3
Significance: II has cutting edge technology which
could be used to improve diagnostic/prognostic Significance: II found evidence for the use of isoprostanes
modalities or be used for research (higher (a form of anti-inflammatories) in the treatment of
resolution images may display more information conditions such as asthma and COPD, where inflammation
that is needed for progression). is the main source of disease progression.
“In 2007, we published the first major report
“The former was developed in collaboration with _____& _____and
represents the first imaging tool to assess disease activity in both from the Alberta Diabetes Surveillance System
spine and sacroiliac joints of patients with AS.” Sr. Scholar, year 2 (ADSS), in collaboration with the Health
Significance: II developed a tool that can be used to assess Strategies and Surveillance Branches of Alberta
progression of anklosying spondylitis (a chronic and painful Health and Wellness (AHW).” H.Scholar, year 4
condition). This can help determine guidelines for treatment
(ex. When to use a specific therapy)
Significance: The increased surveillance
abilities resulting from ADSS will help inform
health authorities’ efforts to reduce the
*Please note that this timeline only represents a small proportion of the 320 innovation statements captured during the review. burden of diabetes on Albertans.
13. Example of the Balanced Scorecard
Scorecard Information –
Strategy Map Indicators
Scorecard Data –
Dash board
Courtesy PM2 Linking and Rating Processes Linking and Ranking Projects
14. Lessons Learnt
• Verification – “YES WE COULD DO IT”
• Theory to Practice Lessons – Implementation
- Complemented CAHS model with other models
- Imbed knowledge translation – decision making rubrics – progress markers
- Link collaborative activity to outcomes
- Tell the story through qualitative and quantitative indicators
• Context Lessons
- Alberta Innovates System – innovation
- Link organizational and program goals and objectives to CAHS categories
- Tease out Innovation
• Challenges
• Change Management – theory of change
• Utilization – is this a useful model for evaluating health research funding
15. Values and Valuing of Measurement
“We are noticing a shift around the
measurement of scientific value
versus social/public value as well
as the concept of collective action
and shared value. Using a staged
implementation approach, the
CAHS model provided us with a
best practice framework to measure
benefits across the spectrum.
16. Contact Information
Kathryn Graham (PhD)
Director, Performance Management
Alberta Innovates Health Solutions
1500 – 10104 103 Avenue N.W.
Edmonton, Alberta, Canada T5J 4A7
Email: kathryn.graham@albertainnovates.ca
Heidi Chorzempa (MSc)
Manager, Performance Management
Email: heidi.chorzemp@albertainnovates.ca
THANK YOU
Hinweis der Redaktion
Acknowledge, the Performance Management Team: Heidi Chorzempa, May Kharaghani, Liza Chan, Daniel Zhang and Andrew LeJeune I am going to present on implementating the CAHS model, results of a retrospective study, lessons learnt from implementing overall and next steps.
The Edmonton based Alberta Innovates Health Solutions is a not for profit provincial health research funding agency. In early 2010 AHFMR was incorporated under the new realigned research and innovation system designed to strengthen the provinces role as a world leader in using science to seek solutions. Alberta Innovates Health Solutions was established in 1980 to support medical/health research AIHS has provided unprecedented opportunities in health research and innovation Contributed > $1 billion to the scientific community Supported > 8,500 researchers approximately Our mandate is to support the economic and social well-being of Albertans,
Definition CAHS identified three main objectives for conducting evaluation in the context of health research: Evaluation helps account for money that is spent Evaluation helps advocate for more funding Evaluation helps optimize research outcomes Attribution and Contribution Time lags Counterfactual Hard to Measure
When measuring outcomes and impacts, a number of considerations need to be taken into account and choice made, all of which influence the measures Organization’s mission Evidence Base – conceptual models e.g. CAHS, other frameworks in existence Stakeholder needs and values Evaluation tools such as logic models and or Balanced Score cards Whether the approach is top down i.e. priority and or policy driven and or bottom up i.e. curiosity driven research Context, Alberta Health Research and Innovation Strategy (AHRIS looked at 12 strategic focus areas, and 3 foundational areas, Highly Skilled People, Knowledge Translation, Innovation Platforms) Multi-faceted Approach: Mixed Methodology – Qualitative and Quantitative Indicators, narrative Multi-Method – case studies to bibliometrics Multi-Dimensional – spectrum of outputs, outcomes and impact Level of aggregation, need to determine the unit of analysis and appropriate level of measurement, e.g. individual versus system level Target audience – again to need.
Verification: Retrofit to Personnel Support Grant - Ability to retrofit and operationalize the CAHS model to an existing grant program based on an open discovery model Theory to Practice: Investigate contextual issues in operationalizing the CAHS model to an Alberta context, cross the 4 CIHR pillars, Alberta context, feasibility and usability Evidence Informed Decision Making: Test the model retrospectively for feasibility, usability, issues etc. prior to piloting it prospectively with two other grant programs.
The study was based on personnel support to fund the career and livelihoods of hundreds of remarkable researchers each year. Resulting in an approximate 40 million dollar investment each year. Across CIHR research pillars as per Gretchen’s presentation
A retrospective review of annual progress reports finalized from 2004-2008 Population of a database containing 584 variables Recognizing project limitations: Attribution and contribution Self-report Data variability Outcome underestimation Retrofitting
Documents Vision, mission, goals, values, business planning documents, corporate and operational, individual performance management plans 08/01/12
Dangling Indicators/Measures must be linked to something e.g. mission, objective, question etc. Metrics only approaches are behind the times, “Best Practice combines narratives with relevant qualitative and quantitative indicators to gauge broader social, environmental, cultural and economic public value. Limited consultation between policy-makers and the research evaluation community has led to a lack of policy-learning from international developments” (Claire Donovan 2011 Research Evaluation) Verification – “YES WE COULD DO IT” Theory to Practice Lessons – Implementation Complimented model with process measures for the linkage and exchange e.g. collaborations, networks, etc. Need to both qualititative and quantitative indicators and narratives to tell the story Context Lessons Change in Research Landscape and Strategy Organizational Excellence – Balanced Score Cards KT model to enhance Redesign – Goals and objectives Innovation Collective action Evidence Informed Decision Making Utilization – user perspective – data capture rules Change Management - awareness, education, systems
