1. Lone Inventor
Sarah Lebovitz, University of Virginia
With Natalia Levina, NYU Stern School of Business
And Hila Lifshitz-Assaf, Warwick Business School
To Engage or Not to Engage with AI
for Critical Judgments?

2. Surge of AI “outperformance” claims across professional fields

4. promising
Higher quality, better accuracy, faster
speed, greater efficiency,…..

5. 1.
How are experts using
these tools in practice?
2.
How do we know the
quality of AI tools?

6. Current Debate in the Literature
Vs.
Future of work with AI is debated through the automation vs. augmentation discourse
(Brynjolfsson and McAfee 2014, Daugherty and Wilson 2018, Davenport and Kirby 2016, Raisch and Krakowski 2021;
Puranam 2021, Felton, Raj, & Seamens 2021)
AI AS AUGMENTING AI AS AUTOMATING

7. What is the goal of “augmenting” humans in knowledge work?
Human AI Augmentation is explained as humans and machines coming together to
“multiply their capabilities” and “combine their complementary strengths” (Raisch and
Krakowski 2021, p. 6).
• Is the machine really adding a complementary and valuable expertise?
• Are knowledge workers really “collaborating” with machines?

8. 5/31/23 8
How do you collaborate with a
“black box” machine?

9. Collaborating with a “black box” machine
• Collaboration = synergistic combination of diverse expertise (Maguire et al. 2004)
• Collaboration involves agents understanding the other party’s input and relating it to
their own expertise through reflection (but not necessarily agreement) (Hardy et al. 2005,
Levina 2005, Maguire et al. 2004)
• How do you collaborate with a machine that also happens to be black-boxed?

10. II. Opacity and “Black Box” AI
• Opacity refers to the difficulty to follow the reasoning behind a given outcome
when such reasoning is obscured or hidden from view.
• Deep-learning-based algorithms are commonly referred to as “black boxes”, as they
are difficult to understand or explain their outputs
• Yet, they are used for critical decisions, and we know little on how professionals
deal with the opacity (Christin 2020, Diakopoulos 2020).

11. How do professionals deal with opacity when
using AI for critical decisions?
Research Question

12. Data Collection
• Observation: 11-month ethnographic
observation of radiologist work practices
across the three departments
500+ hrs observation
40 radiologists
1000+ diagnoses
• Interviews: 33 semi-structured interviews
• Documentations and Artifacts: (1)
diagnostic data, medical notes, training
materials, (2) published regulatory
documentation, technical research papers,
and vendor documentation

13. Research Design
This study focuses on 3 different diagnostic AI tools used in practice:
Breast department Chest department Pediatric department

14. Professional
Makes the Initial
Diagnosis
Professional
Makes the Final
Diagnosis
AI
generated
diagnosis
Sequence of Augmentation
Human &
AI
diagnoses
diverge
Human &
AI’s
diagnoses
align

15. Professional
Makes the Initial
Diagnosis
Professional
Makes the Final
Diagnosis
AI
generated
diagnosis
Sequence of Augmentation
Human &
AI
diagnoses
diverge
Human &
AI’s
diagnoses
align
?
In all three cases, AI output regularly
conflicted with radiologists’ opinion

16. Breast Cancer Diagnosis

17. Professional
Initial
diagnosis
Making final
diagnosis
AI
generated
diagnosis
Using AI During Decision Making
Human
and AI
disagree
Human
and AI
agree

18. Breast Cancer Diagnosis
‘‘
Complex Task
• Visually overwhelming
• 90-100 cases per radiologist per day
• ~ 2-3 min per case
Critical Decision
• False negative: missed cancer
• False positive: physical, financial and
emotional costs of ‘unnecessary’ biopsy
Modality: Mammogram
The [abnormalities] we worry about are really faint and
tiny ones: those are signs of early cancer. They’re the
ones you can barely see…It’s like looking for a snowball
in a snowstorm (Dr. Vernier)

19. Professional
Initial
diagnosis
Making final
diagnosis
AI
generated
diagnosis
Using AI During Decision Making
Human
and AI
disagree
Human
and AI
agree

20. Breast Cancer Diagnosis: Using AI
Tool Design
• Neural network that segments abnormal regions
and classifies their likelihood to be malignant
• Updated version in use since 2005
Tool Use
• Single mouse click, Stars, triangles, and plus signs
displayed as an overlay on images
AI for Breast Cancer

21. Professional
Initial
diagnosis
Making final
diagnosis
AI
generated
diagnosis
Using AI During Decision Making
Human
and AI
disagree
Human
and AI
agree
The vast majority of cases we
observed involved conflicting views

22. Potentially
cancerous
Not
cancerous
Potentially
cancerous
A C
Not
cancerous
B D
AI result
AI in the Final Diagnosis
Professional
Initial
diagnosis

23. Potentially
cancerous
Not
cancerous
Potentially
cancerous
A C
Not
cancerous
B D
AI result
AI in the Final Diagnosis
Professional
Initial
diagnosis
When the AI was in agreement with the professional diagnosis, it was perceived as unnecessary
’’
It isn’t helping anybody. It's actually
just another step for me to do (Dr. Zhao)
‘‘

24. Potentially
cancerous
Not
cancerous
Potentially
cancerous
A C
Not
cancerous
B D
AI result
Professional
Initial
diagnosis
If there’s something that’s concerning to
you, based on your initial interpretation,
that the [AI tool] is saying, ‘Oh, this looks
normal,’ you couldn’t use that
information and say, ‘We’re not going to
biopsy it’ (Dr. Gosh)
‘‘
’’
AI in the Final Diagnosis
ààSpending time “using AI” but
ignoring knowledge claims

25. Potentially
cancerous
Not
cancerous
Potentially
cancerous
A C
Not
cancerous B D
AI result
Professional
Initial
diagnosis
AI in the Final Diagnosis

26. ’’
‘‘
Opaque AI Increases Uncertainty
What is it telling me to look at? At
this tissue? It looks just like the
tissue over here, which is perfectly
normal…I have no idea what it's
thinking (Dr. Kanopi).

27. AI result
Professional
Initial
diagnosis I blow so many things [AI results] off. Like
if there’s normal scarring or stable calcs
[benign tissue], it’s [AI tool] going to pick
up everything (Dr. Gibbs)
‘‘
’’
AI in the Final Diagnosis
àSpending time “using AI” but
ignoring knowledge claims
Potentially
cancerous
Not
cancerous
Potentially
cancerous
A C
Not
cancerous B D

28. Un-engaged “Augmentation”
Not relating AI knowledge claim to expert’s claim
Engaged Augmentation
Relating AI knowledge claim to expert’s claim
Considering AI
knowledge claim
Increasing epistemic
uncertainty
Experiencing AI-in-use
opacity
- No ability to view
reasoning behind a given
AI knowledge claim
Blindly accepting AI
claim
Enacting AI
interrogation
practices
Overruling AI claim
Synthesizing AI and
expert’s claims to form
a new claim
Ignoring AI claim
Forming initial
knowledge claim
Reflectively agreeing
with AI claim
Process of Unengaged Augmentation

29. Lung Cancer Diagnosis

30. Lung Cancer Diagnosis
Modality: CT Scan
Complex Task
• Visually overwhelming
• ~ 3-4 minutes per case
• 20-30 cases per radiologist per day
Critical Decision
• False negative: missed cancer
• False positive: physical, financial and
emotional costs of biopsy
Am I hallucinating a nodule?…I think it’s there, but it’s hard
to see. It’s in a bad location…It’s behind two ribs, so it’s
impossible to get a good look there. (Dr. Ziv)
‘‘

31. Lung Cancer Diagnosis: Using AI
AI for Lung Cancer
Tool Design
• Neural network that segments abnormal regions
and classifies their likelihood to be malignant
• Updated version in use since 2007
Tool Use
• Single mouse click, Circles displayed as an
overlay on images

32. Professional
Initial
diagnosis
Making final
diagnosis
AI
generated
diagnosis
Using AI During Decision Making
Human
and AI
disagree
Human
and AI
agree

33. Potentially
cancerous
Not
cancerous
Potentially
cancerous
A C
Not
cancerous
B D
AI result
If I don't see any lung nodules,
and [CT AI] doesn't see any lung
nodules, then okay, we're good! I feel
very comfortable saying there's no
lung nodules. (Dr. Galas)
’’
‘‘
AI in the Final Diagnosis
Professional
Initial
diagnosis
This time, [CT AI] found nothing.
Any time that happens, it puts a
big smile on my face (Dr. Farber)
’’
‘‘
When in agreement then it only increased confidence and reduces uncertainty

34. Potentially
cancerous
Not
cancerous
Potentially
cancerous
A C
Not
cancerous
B D
AI result
AI in the Final Diagnosis
Professional
Initial
diagnosis
In the majority of cases we observed, the AI results conflicted with
radiologists’ initial judgments.

35. ’’
‘‘
Opaque AI Increases Uncertainty
How does [the AI tool]
know that this is a nodule,
but this isn’t? (Dr. Vreicht)
‘‘ ’’
There must be someone inputting the
data that says, ‘This is a nodule, this is a
nodule,’ but I have no idea what that
data looks like (Dr. Levi)

36. Dealing with AI-in-use Opacity:
Using “AI Interrogation Practices”
Sociomaterial knowledge work practices that
enable professionals to analyze an AI output and
hence incorporate it in their final judgment.

37. Engaged Augmentation :
Using AI Interrogation Practices
When the initial diagnosis and AI results conflicted, the radiologists built
understanding of the AI result by interrogating and examining the evidence
around the suspected area and relating it to their own claim:
• They zoomed in, scrolled forward and backward on the CT repeatedly,
• Assessing “flows” were in a continuous path vs disappeared abruptly
(nodules)
• Compared prior years’ exams to study change over time
• Changed the CT settings and levels of contrast (”windowing”)
• Etc.
I know my limitations and I know this [CT AI] is
going to help them [nodules] stand out a little
better. It’s worth the extra time in my mind
(Dr. Vreicht)
‘‘

38. Engaged Augmentation
By enacting AI interrogation practices, experts were able to:
• Build an understand of the AI result, albeit their experienced opacity
• Demonstrate their willingness to change their initial judgment
• Integrate the knowledge claims:
Reflectively agreeing, Overruling, or Synthesizing the claims

39. Opacity of AI-in-use for experts’ critical decision making
Un-engaged “Augmentation”
Not relating AI knowledge claim to expert’s claim
Engaged Augmentation
Relating AI knowledge claim to expert’s claim
Considering AI
knowledge claim
Increasing epistemic
uncertainty
Experiencing AI-in-use
opacity
- No ability to view
reasoning behind a given
AI knowledge claim
Blindly accepting AI
claim
Enacting AI
interrogation
practices
Overruling AI claim
Synthesizing AI and
expert’s claims to form
a new claim
Ignoring AI claim
Forming initial
knowledge claim
Reflectively agreeing
with AI claim

41. Contributions &
Implications:
I. Problematizing
Augmentation

42. The concept of augmentation assumes that organizations will achieve
positive outcomes, usually by improving quality of knowledge,
efficiency, or both
Do AI augmentation tools actually augment experts?
Who or What Is Being Augmented?

43. The concept of augmentation assumes that organizations will achieve
positive outcomes, usually by improving quality of knowledge,
efficiency, or both
Do AI augmentation tools actually augment experts?
• From an efficiency perspective - currently not, a risky goal
Who or What Is Being Augmented?
?

44. The concept of augmentation assumes that organizations will achieve
positive outcomes, usually by improving quality of knowledge,
efficiency, or both
Do AI augmentation tools actually augment experts?
• From an efficiency perspective - currently not, a risky goal
• From a quality/accuracy perspective- unclear
Who or What Is Being Augmented?
?

45. Contributions & Implications:
II. Black Box AI and Opacity-in-use

46. • Opacity associated with AI tools for critical decisions has become a fiercely
debated topic in both academic and societal conversations (Christin 2020,
Diakopoulos 2020, von Krogh 2018, Pasquale 2015).
• There is a recent focus on developing “explainable AI” or
“interpretable AI” (Barredo Arrieta et al. 2020, Bauer et al. 2021, Hooker et al. 2019, Rudin
2019, Samek et al. 2019),
• We illustrate opacity is not simply a technical characteristic of the tool, it is
experienced in use, it requires new work processes to help experts relate to
AI knowledge claims
Theoretical Contributions

47. Labs
Lone Inventor
Sarah.Lebovitz@Virginia.edu
@SarahLebovitz
Thank you