Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Nächste SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Wird geladen in …3
×
1 von 31

WiMLDS Paris O Vereschak Trust AI-assisted decisions

0

Teilen

Herunterladen, um offline zu lesen

Presentation of Oleksandra Vereschak, WiMLDS Paris and Poznan, May 2021

WiMLDS Paris O Vereschak Trust AI-assisted decisions

  1. 1. How to Evaluate Users’ Trust in AI-Assisted Decision Making? Selected results from CSCW2021 paper submission – online soon Oleksandra Vereschak Gilles Bailly Baptiste Caramiaux @WiMLDS_Paris x @WiMLDS_Poznan - May 19, 2021
  2. 2. Context: AI-Assisted Decision Making B A My recommendation is B. 1
  3. 3. Context: AI-Assisted Decision Making My recommendation is B. B it is! 1
  4. 4. Context: AI-Assisted Decision Making Cai et al., “Hello AI”, CSCW’19 The final overall grade is ____ You should not hire this candidate Image: CVVIZ 2
  5. 5. Context: Evaluating System with Users 3 Easy to use? Useful? Fun? Trustworthy?
  6. 6. Context: Evaluating System with Users 3 Easy to use? Useful? Fun? Annoyed? Satisfied? Relaxed? Trusted the system? Trustworthy?
  7. 7. Context: Evaluating System with Users 3
  8. 8. Context: Evaluating System with Users 3
  9. 9. Context: Evaluating Users’ Trust in AI-Assisted Decision Making Cai et al., “Hello AI”, CSCW’19 The final overall grade is ____ 4 Trust in AI Transparency IBM
  10. 10. Context: Evaluating Users’ Trust in AI-Assisted Decision Making Cai et al., “Hello AI”, CSCW’19 The final overall grade is ____ 4 Image: Dreamstime
  11. 11. Context: Evaluating Users’ Trust in AI-Assisted Decision Making Cai et al., “Hello AI”, CSCW’19 The final overall grade is ____ 4 Image: Dreamstime ?
  12. 12. Context: Evaluating Users’ Trust in AI-Assisted Decision Making Cai et al., “Hello AI”, CSCW’19 The final overall grade is ____ 4 Image: Dreamstime ? What is trust? What is important to evaluate trust? How to measure trust?
  13. 13. ? What is trust? What is important to evaluate trust? How to measure trust? 5 Human-AI Trust (decision making) Human-Human Trust
  14. 14. Takeaway Messages 1. Three key elements of trust (what makes trust different from other theoretical constructs) 2. Why and how to incorporate the key elements in the trust study environment 3. How and how not to choose trust measures 6 These results are a part of CSCW2021 paper, online soon.
  15. 15. What is trust? An attitude that an agent will achieve an individual’s goal in a situation characterized by uncertainty and vulnerability (Lee and See, 2004) 7
  16. 16. What is trust? An attitude that an agent will achieve an individual’s goal in a situation characterized by uncertainty and vulnerability (Lee and See, 2004) 7
  17. 17. What is trust? An attitude that an agent will achieve an individual’s goal in a situation characterized by uncertainty and vulnerability (Lee and See, 2004) 7 Attitude = evaluations you make about objects, ideas, events, or other people.
  18. 18. What is trust? An attitude that an agent will achieve an individual’s goal in a situation characterized by uncertainty and vulnerability (Lee and See, 2004) 7 Attitude = evaluations you make about objects, ideas, events, or other people. Sara finds spiders cute. David has mistrust and annoyance towards telemarketers.
  19. 19. What is trust? An attitude that an agent will achieve an individual’s goal in a situation characterized by uncertainty and vulnerability (Lee and See, 2004) 7 Attitude = evaluations you make about objects, ideas, events, or other people. Sara finds spiders cute. David has mistrust and annoyance towards telemarketers. Behavior Sara would never kill a spider. David always listens to telemarketers.
  20. 20. What is trust? An attitude that an agent will achieve an individual’s goal in a situation characterized by uncertainty and vulnerability (Lee and See, 2004) 8 That is there is an undesirable outcome associated with a wrong decision. If I assign a wrong treatment, this will have negative consequences on the patient’s health.
  21. 21. What is trust? An attitude that an agent will achieve an individual’s goal in a situation characterized by uncertainty and vulnerability (Lee and See, 2004) 8 That is there is an undesirable outcome associated with a wrong decision. If I assign a wrong treatment, this will have negative consequences on the patient’s health. If my estimate of the number of jelly beans in a jar is off, I am wrong.
  22. 22. What is trust? An attitude that an agent will achieve an individual’s goal in a situation characterized by uncertainty and vulnerability (Lee and See, 2004) 8 That is there is an undesirable outcome associated with a wrong decision. If I assign a wrong treatment, this will have negative consequences on the patient’s health. If my estimate of the number of jelly beans in a jar is off, I am wrong. Trust Confidence
  23. 23. What is trust? An attitude that an agent will achieve an individual’s goal in a situation characterized by uncertainty and vulnerability (Lee and See, 2004) 9 Positive expectations. I think the system is good enough to help me to do this task!
  24. 24. What is trust? An attitude that an agent will achieve an individual’s goal in a situation characterized by uncertainty and vulnerability (Lee and See, 2004) 10
  25. 25. Vulnerability and Study Environment While interacting with the system, participants should encounter outcomes related to their decisions and they should care about them. Virtual Real • Virtual penalties • Negative consequences for participants • Negative consequences for other stakeholders • Monetary • Cognitive effort • Damage • Temporal 11
  26. 26. Positive Expectations and Study Environment The introduction and the first interaction are important : ● Instructions (e.g., stated accuracy, relevant experience) ● Initial experience (first trials error-free) If this can lead to deception or unrealistic expectations or if participants are skeptical at the beginning, you might want to omit the first data points. 12
  27. 27. Measures of Trust Trust =/= Behavior Questionnaires: ● Single-item From 1 to 7, how much do you trust this system? further research ● Multi-item A list of carefully selected and validated questions: If another serious issues arises, I would feel comfortable using this system. This system is effective in helping me with the task. Examples: Mayer and Davis (1999); McKnight, Choudhury, Kacmar (2002). 13
  28. 28. Measures of Trust Trust =/= Behavior Questionnaires: ● Single-item From 1 to 7, how much do you trust this system? further research ● Multi-item A list of carefully selected and validated questions: If another serious issues arises, I would feel comfortable using this system. This system is effective in helping me with the task. Examples: Mayer and Davis (1999); McKnight, Choudhury, Kacmar (2002). 13
  29. 29. Measures of Trust Trust =/= Behavior Questionnaires: ● Single-item From 1 to 7, how much do you trust this system? further research ● Multi-item A list of carefully selected and validated questions: If another serious issues arises, I would feel comfortable using this system. This system is effective in helping me with the task. Do not forget to report psychometric statistics = indicator of “quality” of the questionnaire, especially if you modified it. Ex: Cronbach’s alpha, omega coefficient of McDonald 13
  30. 30. Takeaway Messages 1. Three key elements of trust (what makes trust different from other theoretical constructs): vulnerability, positive expectations, attitude 2. Why and how to incorporate the key elements in the trust study environment: task outcomes and first interactions are important 3. How and how not to choose trust measures: trust =/= behavior, triangulate 6 These results are a part of CSCW2021 paper, online soon.
  31. 31. How to Evaluate Users’ Trust in AI-Assisted Decision Making? Selected results from CSCW2021 paper submission – online soon Oleksandra Vereschak vereschak@isir.upmc.fr Gilles Bailly Baptiste Caramiaux @WiMLDS_Paris x @WiMLDS_Poznan - May 19, 2021

×