Prof Nancy Devlin, presentation at HTAi, Tokyo - May 2016

1. Professor Nancy J. Devlin
Director of Research, OHE
HTAi
Tokyo • May 2016
Structured decision making approaches to the
inclusion of multiple criteria in HTA

2. Contents
1. Multiple criteria in HTA decision making
2. ISPOR Best Practice Taskforces on MCDA
3. Experience from pilots of MCDA in HTA globally
4. Why has MCDA not been implemented in HTA?
5. Specific issues to be addressed re HTA
6. Next steps
7. Concluding remarks

3. 1. Multiple criteria in HTA
• Improving health is generally the primary goal of health care
• But health care systems face multiple objectives that go beyond beyond
improvements in population health eg. not just how much health, but how it
is distributed; other equity considerations; societal preferences for prioritising
certain problems or patients (eg children).
• e.g. QALY maximisation is a pragmatic way of assessing value for money – but
clearly does not capture everything that matters to society.
• HTA systems vary in the extent to which they are explicit and consistent in
identifying what these other criteria are, and how they are reflected in their
decision making processes.
• But HTA generally relies on deliberative processes to weigh up different kinds
of evidence.

4. • The question isn’t ‘should multiple criteria be taken into account in
HTA?’ – because they already are.
• The question is: how best should multiple criteria taken into account?
Fully quantified
Purely
qualitative/
deliberative
Criteria decided ex ante Core criteria – but with potential to
make exceptions
Criteria decided on an ad hoc basis
Fixed criteria Criteria vary with explicit justification Criteria may vary
Fixed weights Trade-offs/weights normally the
same – but may vary, and are always
explicit
Trade- offs between criteria vary
Explicit and fully transparent process Process explicit and full transpareny Process not pre-agreed; basis for
decisions not explicit or transparent
‘Structuring judgments’

5. The potential merits of structured decision making
Desirable features of HTA process: Consistency and replicability; Transparency (to
taxpayers; stakeholders; signals to industry about what innovations are of value);
Accountability .
• Weighing up complex information is cognitively demanding
• Literature shows that individuals are subject to “cognitive bias”
• Deliberative processes are influenced by group dynamics and factors including
chairing styles and dominant personalities
• “The preferred options identified by MCDA are likely to out-perform the use of
intuitive judgement alone” (Devlin and Sussex, 2011)
• Policy initiatives reflect the need to take into account multiple criteria simultaneously
and in a systematic way (e.g. VBA in the UK; GPS-Health in global context - Norheim et
al, 2014)
• Increasingly a wide range of stakeholders, including patients and clinicians, have been
involved in HTA (e.g. PACE in Scotland) - but unclear how stakeholders’ views influence
final decisions

6. 2. Growing interest in MCDA
But also some reservations about
this set of techniques; and some
misunderstandings about MCDA

7. Growing interest in use of MCDA in HTA globally
• NICE:
• ‘Structured decision making’ included for the first time in the 2013 methods
review
• Explicit criteria for new highly specialised technologies process (currently under
consultation).
• Examples of one-off pilots in the literature :
• Israeli Health Basket Committee (Golan and Hansen, 2012)
• Colombia
• Belgium
• Italy (Radaelli et al., 2014)
• Thailand (Youngkong et al., 2012)
• Germany (IQWiG, 2013)

8. Israel’s Health Basket
Committee pilot
• Bubble size =
quality of
evidence
• Used to inform
a deliberative
process
Source: adaptation of Golan
and Hansen (2012)

9. MCDA in Lombardia, Italy
• For the implementation of new health technologies, the Lombardia region in
Italy has introduced a system combining elements of the EUnetHTA Core model
(for the assessment) and an MCDA approach (EVIDEM) as a decision-making aid
• The MCDA framework includes 9 broad dimensions and 20 criteria, including
disease-, treatment-, financial- and social-related aspects
• This approach has been deemed successful and used for 26 technologies
(Radaelli et al., 2014)

10. Thai pilot
Performance matrix for the selection of interventions for assessment
Source: Youngkong
et al., 2012

11. Thai pilot (2)
• Assessment and appraisal of selected interventions based on
• Value for money (incremental cost effectiveness ratio
against a threshold)
• Budget impact
Source: Youngkong et al., 2012

12. IQWiG pilots
• Aims were:
• identifying patient-relevant outcomes in depression and hepatitis C
• eliciting patient preferences on the selected outcomes using two approaches
(Analytic Hierarchy Process (AHP) and Discrete Choice Experiment (DCE))
• enabling aggregation of outcome-specific efficiency frontiers based on
obtained weights

13. IQWiG pilots
• Both pilots concluded that MCDA approaches can be used to support the HTA
process to incorporate patient preferences (Thokala et al., 2016)
• Some challenges remain
• E.g. “Which persons should be interviewed and how transferable are these
results to the entire patient population?” (IQWiG, 2015)

14. 4. Why hasn’t MCDA been implemented in HTA?
• HTA organisations may have some discomfort with a requirement to be
fully explicit about the basis for its decisions.
• Fundamental misunderstandings about the use of MCDA to structure
deliberation: eg: the view that ‘If we use MCDA there wont be any point
in having HTA committees – decisions would be made by machines and
algorithms, and there wont be any role for judgement’
• Conservatism with a small ‘c’ (the cost per QALY system is simple,
practical, and well accepted; moving away from it causes nervousness)
• Important methodological challenges to work through specifically in
relation to use of MCDA in HTA.

15. 5. Issues still to be addressed in the context of HTA
Criteria Weights Uncertainty Budget constraint
Established in advance;
the same across all
decisions? Or varying
case-by-case?
Whose preferences
should be used to
establish the
weights/trade-offs?
Uncertainty a separate
criterion – or uncertainy
handled via sensitivity
analysis?
How to reflect cost? (eg
separate criterion, as in
EVIDEM? Or MCDA to
create a composite
measure of value, to be
weighed up against cost?)
How to establish a
legitimate set of criteria?
(current practice in HTA?
Stated objectives of the
health care system?
Consultation with the
general public?)
Weights to reflect the
views of those in HTA
committees – or
supplement these with
quantitative evidence
(stated preferences) of
patients, the general
public?
How is opportunity cost
to be addressed in
decision making involving
multiple criteria? What
are the implications for
the cost effectiveness
threshold?

16. 6. Next steps
• Further work required to establish appropriate methods and to
address methodological questions relating specifically to the
development and use of MCDA in HTA
• Plans to build on existing ISPOR taskforce, by a further taskforce
specifically to address the issues identified above.

17. 7. Concluding remarks
• Structured decision making approaches can offer considerable advantages in HTA
by providing a coherent/unifying framework for healthcare decision making
• They do not aim to replace the judgement of HTA committees – but to help
committees ensure they exercise judgements in a consistent, explicit way.
• What specific approaches are best will depend on the specific socio-political
characteristics of the health care system – ‘one size does not fit all’.
• Consideration of cost and opportunity cost in a systematic way remains a
methodological challenge (not only from an MCDA perspective)
• Do costs of implementation justify benefits of improved decision making? Formal
evaluation required.
• Need to move beyond tendency of current pilots to focus on feasibility (‘can we
do it?’) to wider questions (‘do decision makers find it acceptable? What would
‘success’ look like?)

18. References
Devlin and Sussex (2011). Incorporating Multiple Criteria in HTA. Methods and processes. OHE research
https://www.ohe.org/publications/incorporating-multiple-criteria-hta-methods-and-processes
Golan and Hansen (2012). Which health technologies should be funded? A prioritization framework
based explicitly on value for money. Israel Journal of Health Policy Research 1:44
Norheim O. F., Baltussen R., Johri M., et al., 2014. Guidance on priority setting in health care (GPS-
Health): the inclusion of equity criteria not captured by cost-effectiveness analysis. Cost Effectiveness
and Resource Allocation 12:18
Radaelli et al., (2014). Implementation of EUnetHTA core Model® in Lombardia: the VTS framework.
International Journal of Technology Assessment in Health Care 30:1; 105-12
Thokala et al, for the ISPOR MCDA Task Force (2016) MCDA for Health Care Decision Making – An
Introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value in Health (in
press)
Walker, A., (2016). Challenges in using MCDA for reimbursement decisions on new medicines? Value in
Health 19: 123-124
Youngkong S, Baltussen R, Tantivess S, Mohara A, Teerawattanon Y (2012). Multicriteria decision analysis
for including health interventions in the Universal Health Coverage Benefit package in Thailand. Value in
Health 15; 961-970
IQWiG (2013). Analytic Hierarchy Process (AHP) – pilot project to elicit patient preferences in the
indication “depression”. https://www.iqwig.de/download/Executive-summary-of-working-
paper_Analytic-Hierarchy-Process-pilot-project.pdf