The ISOR Project No health without research. And no research without evaluation Panel Session - HTAi 2012, Bilbao

1. The ISOR Project
No health without research.
And no research without evaluation
Panel Session - HTAi 2012, Bilbao
Maite Solans-Domènech
Catalan Agency for Health Information, Assessment and Quality
(CAHIAQ)

2. Outline of the presentation
1. CAHIAQ call in clinical and health
services research
2. The ISOR project
3. Impact on informing decision-making

3. Outline of the presentation
1. CAHIAQ call in clinical and health
services research
2. The ISOR project
3. Impact on informing decision-making

4. Call in clinical and health services
research
• Mission: to fill local knowledge gaps and inform clinical and
healthcare decision-makers
• Research topic identification and priority-setting designed
and implemented to ensure that the knowledge gaps of
decision makers are addressed
• Ex-ante, ongoing and ex-post evaluations (13 years
experience)
• There is a scientific committee that monitors the whole
process.
• Predominance of oriented research with local relevance and
informing local decision-making
• 7 calls from 1996 to 2008

5. Call in clinical and health services
research
Prioritised Funded projects Amount granted Average Euros/
Call
topics N (€) project
1996 15 18 510,860.0 28,381.1
1998 20 19 540,911.0 28,469.0
2000 30 25 841,417.0 33,656.7
2002 35 25 841,417.0 33,656.7
2004 30 22 1.021,721.5 46,441.9
2006 29 18 1.021,721.5 56,762.3
2008 20 14 1,021,214.0 72,943.9
TOTAL 179 141 5,779,261.0 37,622.4

6. Outline of the presentation
1. CAHIAQ call in clinical and health
services research
2. The ISOR project
3. Impact on informing decision-making

7. Closing the assessment cycle:
The ISOR Project
Objectives:
 To carry on the evaluation of the impact of research
funded by CAHIAQ (accountability)
 To develop a methodology and/or tool for the
assessment of research projects (transfer methods and
capacity-building)
 To contribute to the spreading of these assessment
practices beyond CAHIAQ (advocacy)

8. Closing the assessment cycle:
The ISOR Project
Adoption of the Canadian CAHS-ROI model:
 Useful common framework for any different type of
health research (from basic research to health services)
 Useful for any stakeholder’s view (from funder to
evaluator)
 Provides a set of generic indicators in the five impact
levels:
− Advancing knowledge
− Capacity building
− Informing decision-making
− Health benefits
− Broad economic and social impacts

9. Closing the assessment cycle:
The ISOR Project
Initiation and Diffusion of Health Research Impact
Health industry
Global Research
Healthcare
Appropriateness, Improvements in Health
Other industries Access, etc. and Well-being
Interactions/ ollaborations
Research Results
Knowledge Pool
Health Catalan Health Prevention and
Status, Research Treatment
Government
Function, /Feedbac k
Well-being, • Clinical C
• Health Services Research Agenda Economic and Social
Economic
• Population and Prosperity
conditions
Public Health Int erac cions
Determinants of Health
Public Information,
Groups
Research capacity
Impacts feed back into inputs for future research
PAYBACK Topic identification Primmary
Secondary
Inputs Outputs Adoption Final Outcomes
FRAMEWORK Outputs
Process Dissemination
Advancing Knowledge Informing Decision Making Economic Benefits
Capacity Building Health Benefits Canadian Academy of Health Science.
Return on Investment framework 2009

10. Outline of the presentation
1. CAHIAQ call in clinical and health
services research
2. The ISOR project
3. Impact on informing decision-making

11. Impact on informing decision-making
Bottom-up assessment of CAHIAQ Calls
 Descriptive
 [based on questionnaires to 70 PI]
 Case study (respiratory disease research)
 [based on semi-structured interviews to 8 decision-
makers and 15 researchers related to 6 projects]

12. Impact on informing decision-making
Questions to PI (n=70) n (%)
Types of changes carried out in practice based 60% PI: (n=40)
on the research outcomes
• Clinical 29 (58.0) Real changes
• Organizational - 12 (24.0) Clinical and/or
centre/institution organizational
• Patient´s behaviour 5 (10.0) setting.
• Public Health management 2 (4.0)
• Legal/regulations 2 (4.0)
Unknown / No changes in practice. 28 (40.0)
Stakeholders or recipients who have used or 70% (n=46)
taken into account the research outcomes
• Managers and planners 24 (41.4) • Stakeholders in
• Clinical professionals 17 (29.3) clinical and health
• Investigators/researchers 12 (20.7) services
• Scientific Societies 5 (8.6)
Not taken into account/ unknown 13 (18.6) • Scientific community
• Scientific societies.

13. Impact on informing decision-making
… some “good stories”:
“Yes, it has all generated a “After two years ... it [home-based
culture... We are essentially a hospitalization] grew as a real
clean unit…” service and an integrated care cross-
sectional unit was created within the
hospital”
 Almost all projects have induced or may have induced changes in
clinical practice or in the organization of health services
 Changes
 Direct / indirect
 Contribute to a complex change in clinical practice or
healthcare organization

14. Impact on informing decision-making
 Scientific publications: scientific articles, communications in
congresses
 Direct transmission to potential users: clinical sessions, …
 Unconventional or poorly studied ways for the knowledge transfer
 Scientific societies - as authoritative sources
 Connection between managers and research teams –
direct/indirect participation and collaboration
 The channels used to transfer new knowledge into clinical
practice are complex
… dissemination…. “... the participation of hospital and
primary care specialists in charge was
beneficial [to change]”

15. Impact on informing decision-making
 Structural barriers
 Lack of channels for the translation of research
 Organizational dynamics
 Individual barriers
 Personal factors: reluctance to change
 Nature of research: gap between researchers and
practitioners
...and translation barriers:
“Overall, I would say that researchers are
forced to take on many roles. I am ready “Because hospital directors change
to be a researcher, but not to make an frequently, or because this issue is
electoral pamphlet. In this regard, I think not deemed to be as important as
the relationship with healthcare policies it should be...”
and health plans is not as it should be.”

16. Impact on informing decision-making
IMPACT
Capacity
Relevant
contribution (few Dissemination in
Impact found mainly in
intermediate levels
prior studies) Scientific Societies
Building
Contribution to the design
Informed care to chronic patients and
Forecast of
health care
hospital management*
decision- services to
reduce
making Identification of modificable
admissions
risk factors
New lines of research
Gaps or distances
Difficulty to cultural
change due to the
between potential and
patients (learning
with the disease)l real impacts
Health Prevention in
benefits excerbations
Overall
economic Cost reduction
benefit
Input Primary Secondary Adoption Final
Other impacts
PHASE outcome outcome outcome emerged implicitly and
not intentionally
Project: Risk factors for relapse in COPD

17. ISOR project: Challenges
 ‘Ongoing project’
 SIRECS
 Comparability (across countries, research programmes,
disciplines, …)
 Attribution
 Time-lag
 Interviewees (researchers, health policy or healthcare
decision-makers) interpret narrowly the terms (‘impact’)
 General context: incipient culture of assessment

18. Thanks to Paula Adam, Marta Aymerich, Silvina Berra, Imma Guillamón, Gaietà Permanyer-
Miralda, Joan MV Pons and Emilia Sánchez for their contribution to CAHIAQ strategy and the
ISOR project