The long-term goal of this project is to identify critical social, communication and cognitive factors that can inform a fundamental rethinking of effective Drug-Drug Interaction alerts (DDI alerts) for physicians. Specifically, our objective is to uncover, demonstrate and evaluate novel principles for effective and novel alert design that are based on what physicians consider important when sharing advice from peers in the context of their daily clinical activities.

1. From Critique to
Collaboration: Rethinking
Computerized Clinical Alerts
Debaleena Chattopadhyay, Romisa Rohani Ghahari,
Jon D. Duke (Co-PI), Davide Bolchini (PI)
Presented by
Debaleena Chattopadhyay & Davide Bolchini
soic.iupui.edu
NSF Award #1343973

2. Drug-Drug Interaction (DDI) Alerts
© New Yorker

3. Drug-Drug Interaction (DDI) Alerts

4. Efficacy of DDI Alerts
How much did the efficacy of clinical
alerts—especially DDI alerts—change
in the last decade?

5. Reviewed 42,641 orders
11% (4690) produced alerts
DDI alert override rate = 88%
Allergy override rate = 69%
Reviewed 18,354 orders
13% (2455) produced alerts
DDI alert override rate = 95%
Allergy override rate = 91%
2002
2014

6. Efficacy of DDI Alerts
How much did the efficacy of clinical
alerts—especially DDI alerts—change
in the last decade?
It did NOT.

7. The Problem
How to improve the design of Drug-
Drug Interaction Alerts (DDI Alerts)
to improve physician’s adherence?

8. Motivation
 Drug safety alerts are critical for patient safety but
largely ignored by doctors during medication
prescribing
 Despite efforts to improve design and reduce alert
fatigue, physicians continue to distrust
computerized recommendations

9. Transforming the Outlook
•To improve alerts, we must first look at how to
improve the trust between physician and
computerized advice.
•We explore the foundational principles of what
physicians consider important when taking advice
from peers
•We use this knowledge to create novel designs for
drug safety guidance that elicit physician trust and
a sense of collaboration

10. So what? The Broader Impact
•Potentially reduce the over 2M adverse drug events
per year by improving the safety of drug prescribing
•Translating findings into real-world EMR systems
through the Regenstrief Institute
•Spread adoption to industry with EHR vendors and
NIST

11. Our Approach
User Experience
requirements
design
prototyping
evaluating
1

12. Mining Requirements
Formative studies in clinical settings to unearth key
factors in sharing trusted advice among doctors
when making prescribing decisions
1

13. Our Approach
User Experience
requirements
design
prototyping
evaluating
2

14. Design and Prototyping
Design and deploy novel drug safety alert interfaces
to convey drug safety information to providers in a
more trusted manner.
2

15. Our Approach
User Experience
requirements
design
prototyping
evaluating
3

16. Evaluation
Evaluation studies in the lab and in central Indiana
hospital to assess the effect of the proposed designs
on alert compliance
3

17. Our Approach
User Experience
requirements
design
prototyping
evaluating

18. Requirement Analysis
Understanding Requirements for DDI Alerts

19. Requirements for DDI Alerts
• Before looking at how to improve alerts, we look at
how to improve the trust between physicians and
computerized advice.
• A starting point to address this issue is looking at
whom physicians do trust: their medical colleagues
and mentors.
TRUST

20. Understanding Advice Sharing
among Physicians
• In 3 contextual inquires, we examined why clinical
advice is trusted among physicians.
• 255 minutes – 22 health care professionals during
three inpatient team meetings

22. The Flow Model
• Two primary functional roles emerged from the
model—decision makers and decision implementers.
• Attending and residents primarily served as decision
makers (e.g., “I think we should go ahead and do it.”).
• But when specialists or pharmacists provided the
necessary advice, they acted as decision implementers
(e.g., “Has she received IV iron? Renal [says] they
recommend IV iron.”).
• Medical students (and sometimes pharmacist students)
primarily acted as decision implementers. Although
they actively took part in the decision making process,
their decision was almost always endorsed or corrected
by the supervisors (attending or residents).

24. The Cultural Model
• The cultural model identified a pervasive mindset of an
inpatient team: a strict adherence to medical hierarchy
and a strong preference for experts’ opinions.
• Inter-departmental relations influenced the team’s
decisions to engage in soliciting consults from domain
experts (e.g., “N-surg [neurosurgeons] didn’t put in
clear recs [recommendations] for what they wanted to
do. [..] Maybe rad-onc [radiology oncology], and n-surg
have a better way to talk to each other because we
don’t get calls from them.”).

25. The Cultural Model (cont.…)
• Attending and residents (supervisors in the flow
model) completely influenced the activity of the
medical students and interns (primary decision
implementers in the flow model). But…
• Whereas the flow model uncovered a supervisor-
supervisee relationship evident in an inpatient
meeting, the cultural model surfaced an
undertone—identifying it more closely to a
mentor-mentee relationship.
• We observed a strong influence of expert opinions
on the team’s decisions

26. The Emerging Themes
Informed by our consolidated work models, we identified
eight themes driving trusted advice among physicians in
clinical settings
1. Specialization
2. Role in the Medical Hierarchy
3. Demonstrated Experience
4. Evidence of Understanding the Patient’s Situation
5. Empathy
6. Demonstrated Knowledge of Evidence from the Literature
7. Collaborative and Inclusive Language
8. Timeliness of the Advice

27. Validating Themes with Survey
• To validate with a larger sample of physicians the
crucial themes emerging from our formative study, we
designed and administered an online survey.
• Demographics: Of the 87 questionnaires sent, 37 were
returned (22 females). Respondents were mostly less
than 30 (17) or less than 40 (11) years old, and were
mostly either resident (19) or attending physicians (17).
• 17 physicians worked less than five years while four
worked for more than 25 years. Except four
respondents, all physicians currently worked in an
inpatient environment, and 20 physicians spent more
than 50% of their time in an inpatient environment.

29. Connecting Themes to Survey
Questions

30. Survey Results
Recommending colleagues’ hierarchical role significantly
affected how much a second-year resident would trust or
follow their recommendation. ID (infectious disease)
consultant would be significantly more trusted than the
primary intern.

31. Survey Results (cont.…)
When faced with conflicting recommendations
coming from peers, a second-year resident would
trust and follow a curbside consult from Hospitalist
than one from the new ID fellow.

32. Survey Results (cont.…)
Themes emerging from our contextual inquiry
significantly affected the likeliness of a second-year
resident to trust or follow a recommendation.
Specialization would be trusted and followed
significantly more than role in medical hierarchy.

33. Survey Results (cont.…)
A treemap visualization showing the emerging nodes by number of coding
references. The size of the rectangles represents the number of coding references.
Self-reference was the most commonly coded frame of reference and reflective
was the most commonly coded tone of communication.

34. Design Directions for Trust-Based
Alerts
• Endorsed alerts
• Transparent alerts
• Team-sensing alerts
• Collaborative alerts
• Empathy-driven alerts
• Conflict-mitigating alerts
• Agency-laden alerts
Chattopadhyay, D., Rohani Ghahari, R., Duke, J., D., & Bolchini, D. (2015).
Understanding Advice Sharing among Physicians: Towards Trust-Based Clinical Alerts.
Interacting with Computers (In review).

35. Designing Clinical Alerts
Toward a Model for Trusted Clinical Alerts
Work-in-Progress…

36. Endorsed Alerts
• Endorsed alerts
Steve Nissen, MD
Chair of Cardiology
Approved
Alert

37. Transparent Alerts
Addendum: AMOXICILLIN 500MG. Allergy Alert Override by Smith, JD. 11/14/2014 at 8:31am.
Embed in Chart

38. Empathy-Driven Alerts
Allergy
Allergy Warning

39. Next Steps
• Detailed Design of trusted DDI alerts.
• Low-fidelity, interactive prototyping of DDI alerts.
• Feedback from physicians on the DDI alerts to
understand their responses.

40. Thank you!
bit.do/trusted-alerts
http://mypage.iu.edu/~dbolchin/trusted-
advice.html
{debchatt | dbolchin} @ iupui.edu
Debaleena Chattopadhyay
Romisa Rohani Ghahari
Jon D. Duke (Co-PI)
Davide Bolchini (PI)
This research is based upon work supported by the National Science Foundation under Grant Number
IIS-1343973. Any opinions, findings, and conclusions or recommendations expressed in this material are
those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
soic.iupui.edu