SlideShare a Scribd company logo

Explainability for Natural Language Processing

•
•
Yunyao Li
Yunyao Li

Final deck for our popular tutorial on "Explainability for Natural Language Processing" at KDD'2021. See links below for downloadable version (with higher resolution) and recording of the live tutorial. Title: Explainability for Natural Language Processing Presenter: Marina Danilevsky, Shipi Dhanorkar, Yunyao Li and Lucian Popa and Kun Qian and Anbang Xu Website: http://xainlp.github.io/ Recording: https://www.youtube.com/watch?v=PvKOSYGclPk&t=2s Downloadable version with higher resolution: https://drive.google.com/file/d/1_gt_cS9nP9rcZOn4dcmxc2CErxrHW9CU/view?usp=sharing @article{kdd2021xaitutorial, title={Explainability for Natural Language Processing}, author= {Marina Danilevsky, Shipi Dhanorkar and Yunyao Li and Lucian Popa and Kun Qian and Anbang Xu}, journal={KDD}, year={2021} } Abstract: This lecture-style tutorial, which mixes in an interactive literature browsing component, is intended for the many researchers and practitioners working with text data and on applications of natural language processing (NLP) in data science and knowledge discovery. The focus of the tutorial is on the issues of transparency and interpretability as they relate to building models for text and their applications to knowledge discovery. As black-box models have gained popularity for a broad range of tasks in recent years, both the research and industry communities have begun developing new techniques to render them more transparent and interpretable.Reporting from an interdisciplinary team of social science, human-computer interaction (HCI), and NLP/knowledge management researchers, our tutorial has two components: an introduction to explainable AI (XAI) in the NLP domain and a review of the state-of-the-art research; and findings from a qualitative interview study of individuals working on real-world NLP projects as they are applied to various knowledge extraction and discovery at a large, multinational technology and consulting corporation. The first component will introduce core concepts related to explainability inNLP. Then, we will discuss explainability for NLP tasks and reporton a systematic literature review of the state-of-the-art literaturein AI, NLP and HCI conferences. The second component reports on our qualitative interview study, which identifies practical challenges and concerns that arise in real-world development projects that require the modeling and understanding of text data.

Technology
1 of 174
Download to read offline
1 Explainability for Natural Language Processing 1 IBM Research – Almaden 2 Pennnsylvania State University. 3 Amazon 4 IBM Watson Lecture-style tutorial@KDD'2021 https://xainlp.github.io/ 1 Marina Danilevsky 2 Shipi Dhanorkar 1 Yunyao Li 1 Lucian Popa 3 Kun Qian 4 Anbang Xu
Outline of this Tutorial • PART I – Introduction • PART II - Current State of XAI Research for NLP • PART III – Explainability and Case Study • PART IV - Open Challenges & Concluding Remarks 2
3 PART I - Introduction Yunyao Li
4 AI vs. NLP source: https://becominghuman.ai/alternative-nlp-method-9f94165802ed
5 AI is Becoming Increasing Widely Adopted … …
6 AI Life Cycle Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize
7 AN ENGINEERING PERSPECTIVE Why Explainablity?
8 Why Explainability: Model Development Feature Engineering (e.g. POS, LM) Model Selection/Refine ment Model Training Model Evaluation Error Analysis Error Analysis e.g. “$100 is due at signing” is not classified as <Obligation> Root Cause Identification e.g. <Currency> is not considrerd in the model AI engineers/ data scientists Model Improvement e.g. Include <Currency> as a feature / or introduce entity-aware attention Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Source: ModelLens: An Interactive System to Support the Model Improvement Practices of Data Science Teams. CSCW’2019
9 Why Explainability: Monitor & Optimize Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize User Feedback (implicit/explicit) Feedback Analysis Model Improvement User Feedback e.g. Clauses related to trademark should not classified as <IP> Feedback Analysis e.g. Trademark-related clauses are considered part of <IP> category by The out-of-box model AI engineers Model Improvement e.g. Create a customized model to distinguish trademark-related clauses from other IP-related clauses. Source: Role of AI in Enterprise Applications. https://wp.sigmod.org/?p=2577
10 A PRACTICAL PERSPECTIVE Why Explainablity?
11 https://www.technologyreview.com/2020/09/23/1008757/interview-winner-million-dollar-ai-prize-cancer-healthcare-regulation
12 Needs for Trust in AI à Needs for Explainability What does it take? - Being able to say with certainty how AI reaches decisions - Building mindful of how it is brought into the workplace.
Why Explainability: Business Understanding Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize When Explainability is not Needed • No significant consequences for unacceptable results à E.g., ads, search, movie recommendations • Sufficiently well-studied and validated in real applications à we trust the system’s decision, even if it is not perfect E.g. postal code sorting, airborne collision avoidance systems Source: Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang. EMNLP’2017
14 Regulations Credit: Gade, Geyik, Kenthapadi, Mithal, Taly. Explainable AI in Industry, WWW’2020 General Data Protection Regulation (GDPR): Article 22 empowers individuals with the right to demand an explanation of how an automated system made a decision that affects them. Algorithmic Accountability Act 2019: Requires companies to provide an assessment of the risks posed by the automated decision system to the privacy or security and the risks that contribute to inaccurate, unfair, biased, or discriminatory decisions impacting consumers California Consumer Privacy Act: Requires companies to rethink their approach to capturing, storing, and sharing personal data to align with the new requirements by January 1, 2020. Washington Bill 1655: Establishes guidelines for the use of automated decision systems to protect consumers, improve transparency, and create more market predictability. Massachusetts Bill H.2701: Establishes a commission on automated decision-making, transparency, fairness, and individual rights. Illinois House Bill 3415: States predictive data analytics determining creditworthiness or hiring decisions may not include information that correlates with the applicant race or zip code.
15 … … Source: https://www.ftc.gov/news-events/blogs/business-blog/2020/04/using-artificial-intelligence-algorithms
16 Why Explainability: Data Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize • Datasets may contain significant bias • Models can further amplify existing bias Source: Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang. EMNLP’2017 man woman Training Data 33% 67% Model Prediction 16% (↓17%) 84% (↑17%) agent role in cooking images
17 Why Explainability: Data Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize • Explicitly capture measures to drive the creation of better, more inclusive algorithms. Source: https://datanutrition.org More: Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Emily M. Bender, Batya Friedman TACL’2018
A core element of model risk management (MRM) • Verify models are performing as intended • Ensure models are used as intended 19 Why Explainability: Model Validation Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Source: https://dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/ https://www.mckinsey.com/business-functions/risk/our-insights/the-evolution-of-model-risk-management Defective Model
20 Why Explainability: Model Validation Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize A core element of model risk management (MRM) • Verify models are performing as intended • Ensure models are used as intended Source: https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon- scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G Defective Model
21 Why Explainability: Model Validation Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize A core element of model risk management (MRM) • Verify models are performing as intended • Ensure models are used as intended Source: https://www.theguardian.com/business/2013/sep/19/jp-morgan-920m-fine-london-whale https://financetrainingcourse.com/education/2014/04/london-whale-casestudy-timeline/ https://www.reuters.com/article/us-jpmorgan-iksil/london-whale-took-big-bets-below-the-surface- idUSBRE84A12620120511 Incorrect Use of Model
22 Why Explainability: Model Validation Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize A core element of model risk management (MRM) • Verify models are performing as intended • Ensure models are used as intended Source: https://www.digitaltrends.com/cars/tesla-autopilot-in-hot-seat-again-driver-misuse/ Incorrect Use of Model
23 Why Explainability: Model Deployment Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize • Compliant to regulatory and legal requirements • Foster trust with key stakeholders Source: https://algoritmeregister.amsterdam.nl/en/ai-register/
24 Why Explainability: Model Deployment Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize • Compliant to regulatory and legal requirements • Foster trust with key stakeholders Source: https://algoritmeregister.amsterdam.nl/en/ai-register/
26 Why Explainability: Monitor & Optimize Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize • Carry out regular checks over real data to make sure that the systems are working and used as intended. • Establish KPIs and a quality assurance program to measure the continued effectiveness Source: https://www.huffpost.com/entry/microsoft-tay-racist-tweets_n_56f3e678e4b04c4c37615502
27 Why Explainability: Monitor & Optimize Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize • Carry out regular checks over real data to make sure that the systems are working and used as intended. • Establish KPIs and a quality assurance program to measure the continued effectiveness Source: https://www.technologyreview.com/2020/10/08/1009845/a-gpt-3-bot-posted-comments-on-reddit-for- a-week-and-no-one-noticed/
28 EXPLAIN EXPLANATIONS
29 Explainable AI Credit: Gade, Geyik, Kenthapadi, Mithal, Taly. Explainable AI in Industry, WWW’2020
Input A test example for which two different classifiers predict the same class A test example which one classifier predicts with high confidence A test example which the classifier predicts with low confidence Information Predicted class & top-m evidences of each model Predicted class & top-m evidences Predicted class & top-m evidences (& possibly top-n counter-evidences) What is the role of an explanation? Introduction Table adapted from [Lertvittayakumjorn and Toni, 2019] Reveal model behavior Justify model predictions Investigate uncertain predictions example task: text classification This tutorial’s focus: Outcome explanation problem
Explainability vs. Interpretability 31 Interpretability The degree to which a human can consistently predict the model's result à May not know why Explainability The degree to which a human can understand the cause of the model’s result. à Know why Source: Examples are not enough, learn to criticize! Criticism for interpretability. NIPS’2016 Explaining Explanations: An Overview of Interpretability of Machine Learning. DASS. 2018 explainable à interpretable interpretable !à explainable Explainability Interpretability
32 WHAT MAKES EXPLAINABILITY FOR NLP DIFFERENT?
Source: “Why Should I Trust You?” Explaining the Predictions of Any Classifier. KDD’2016 Example: Explanability for Computer Vision
Example: Explainability for NLP Source: Outpatient Text Classification Using Attention-Based Bidirectional LSTM for Robot-Assisted Servicing in Hospital. Information 2020
35 Which One Takes You Longer to Understand?
36 Which One Takes You Longer to Understand? Understanding explanation for imagine classification à Perceptual Task: a task we are hardwired to do Source: Why a diagram is (sometimes) worth ten thousand words. Jill H. Larkin Herbert A. Simon. Cognitive Science. 1987 Graphs as Aids to Knowledge Construction: Signaling Techniques for Guiding the Process of Graph Comprehension November 1999. Journal of Educational Psychology Understanding explanation for text classification à Graph Comprehension Task - encode the visual array - identify the important visual features - identify the quantitative facts or relations that those features represent - relate those quantitative relations to the graphic variables depicted
37 Which One Takes You Longer to Understand? Understanding explanation for imagine classification à Perceptual Task: a task we are hardwired to do Source: Why a diagram is (sometimes) worth ten thousand words. Jill H. Larkin Herbert A. Simon. Cognitive Science. 1987 Graphs as Aids to Knowledge Construction: Signaling Techniques for Guiding the Process of Graph Comprehension November 1999. Journal of Educational Psychology Understanding explanation for text classification à Graph Comprehension Task - encode the visual array - identify the important visual features - identify the quantitative facts or relations that those features represent - relate those quantitative relations to the graphic variables depicted
38 Challenges in Explainability for NLP Model Explanation Generation Explanation Interface Text modality à Additional challenges Source: SoS TextVis: An Extended Survey of Surveys on Text Visualization. February 2019. Computers 8(1):17 Generate explanation Display explanation Comprehend explanation
39 ONE-SIZE DOES NOT FIT ALL
40 Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Business Users Product Managers Data Scientists Domain Experts Business Users Data Providers AI Engineers Data Scientists Product managers AI engineers Data scientists Product managers Model validators Business Users Designers Software Engineers Product Managers AI Engineers Business Users AI Operations Product Managers Model Validators Appropriate Explanations for Different Stakeholders Whether and how explanations should be offered to stakeholders with differing levels of expertise and interests.
41 Are Explanations Helpful? Credit: Dan Weld. Keynote talk “Optimizing Human-AI Teams”. DaSH-LA @NAACL’2021 Source: Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. CHI’2021 AI outperforming human AI performing comparably as human
AI Correct AI Incorrect Explanations ARE Convincing Team (R+Human Expl.) Credit: Dan Weld. Keynote talk “Optimizing Human-AI Teams”. DaSH-LA @NAACL’2021 Source: “Does the whole exceed its parts? The effect of AI explanations on complementary team performance. CHI’2021c Regardless of whether AI is right or mistaken ”Better” explanation increases trust but NOT appropriate trust
Appropriate Explanations • Need to engender appropriate trust?! • Nature of task may determine affect of explanation on appropriate trust 43 Credit: Dan Weld. Keynote talk “Optimizing Human-AI Teams”. DaSH-LA @NAACL’2021 Source: Does the whole exceed its parts? The effect of AI explanations on complementary team performance. CHI’2021 Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA. ACL’2021 Yet another reason for focusing on explainability for NLP Open-domain QA
PART II – Current State of Explainable AI for NLP Marina Danilevsky Lucian Popa Kun Qian
Outline – Part II • Literature review methodology • Categorization of different types of explanation – Local vs. Global – Self-explaining vs. Post-hoc processing • Generating and presenting explanations – Explainability techniques • Common operations to enable explainability – Visualization techniques – Communication Paradigm • Other insights (XNLP website) – Relationships among explainability and visualization techniques • Evaluation of Explanations 45
Outline of Part II • Literature review methodology • Categorization of different types of explanation – Local vs. Global – Self-explaining vs. Post-hoc processing • Generating and presenting explanations – Explainability techniques • Common operations to enable explainability – Visualization techniques – Communication Paradigm • Other insights (XNLP website) – Relationships among explainability and visualization techniques • Evaluation of Explanations 46
Literature Review Methodology: • The purpose is NOT to provide an exhaustive list of papers • Major NLP/Data Science conferences: – KDD, ACL, EMNLP, NAACL, COLING, AAAI, IJCAI, WWW – Years: 2013 – 2021 – > 240 papers screened • What makes a candidate paper: – contain XAI keywords (lemmatized) in their title (e.g., explainable, interpretable, transparent, etc) • Reviewing process – A couple of NLP researchers first go through all the candidate papers to make sure they are truly about XAI for NLP – Every paper is then thoroughly reviewed by at least 2 NLP researchers • Category of the explanation • Explainability & visualization techniques • Evaluation methodology – Consult additional reviewers in the case of disagreement 47
Outline of Part II • Literature review methodology • Categorization of different types of explanation – Local vs. Global – Self-explaining vs. Post-hoc processing • Generating and presenting explanations – Explainability techniques • Common operations to enable explainability – Visualization techniques – Communicating Explanations • Other insights (XNLP website) – Relationships among explainability and visualization techniques • Evaluation of Explanations 48
How should we differentiate explanations? Multiple different taxonomies exist… [Arya et al., 2019] [Arrietaa et al, 2020] 49 Madsen et al., 2021
How should we differentiate explanations? Two fundamental aspects apply to any XAI problem Is the explanation for an individual instance or for an AI model ? Is the explanation obtained directly from the prediction or requiring post-processing ? Local explanation: individual instance Global explanation: internal mechanism of a model Self-explaining: directly interpretable Post-hoc: a second step is needed to get explanation 50
Local Explanation • We understand only the reasons for a specific decision made by an AI model • Only the single prediction/decision is explained (Guidotti et al. 2018) Machine translation Input-output alignment matrix changes based on instances [NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE. Bahdanau et al., ICRL 2015] 51
Global explanation • We understand the whole logic of a model • We can follow the entire reasoning leading to all different possible outcomes Img source: https://medium.com/@fenjiro/data-mining-for-banking-loan-approval-use-case-e7c2bc3ece3 Decision tree built for credit card application decision making 52
Self-explaining vs. Post-hoc • Self-explaining: we can directly get the explanation with the prediction • Post-hoc: explanation does not come directly with the prediction Only one model Model 1 Model 2 53
Task - how to differentiate different explanations Is the explanation for an individual instance or for an AI model ? Is the explanation obtained directly from the prediction or requiring post-processing ? Local explanation: individual instance Global explanation: internal mechanism of a model Self-explaining: directly interpretable Post-hoc: a second step is needed to get explanation orthogonal aspects 54 Two fundamental aspects apply to any XAI problem
Categorization of Explanations 55
Outline of Part II • Literature review methodology • Categorization of different types of explanation – Local vs. Global – Self-explaining vs. Post-hoc processing • Generating and presenting explanations – Explainability techniques • Common operations to enable explainability – Visualization techniques – Communication Paradigm • Other insights (XNLP website) – Relationships among explainability and visualization techniques • Evaluation of Explanations 56
From model prediction to user understanding XAI model Explained predictions End user Are explanations directly sent from XAI models to users? 57
Two Aspects: Generation & Presentation XAI model Mathematical justifications End user Visualizing justifications 1. Explanation Generation 2. Explanation Presentation UX engineers AI engineers 58
What’s next XAI model Mathematical justifications End user Visualizing justifications 1. Explanation Generation 2. Explanation Presentation UX engineers AI engineers Explainability techniques 59
What is Explainability? • The techniques used to generate raw explanations – Ideally, created by AI engineers • Representative techniques: – Feature importance – Surrogate model – Example-driven – Provenance-based – Declarative induction • Common operations to enable explainability – Attention – First-derivative saliency – Layer-wise Relevance Propagation – Integrated Gradients – Explainability-aware architecture design 60
Explainability - Feature Importance The main idea of Feature Importance is to derive explanation by investigating the importance scores of different features used to output the final prediction. Can be built on different types of features Manual features from feature engineering [Voskarideset al., 2015] Lexical features including words/tokens and N-gram [Godin et al., 2018] [Mullenbachet al., 2018] Latent features learned by neural nets [Xie et al., 2017] [Bahdanau et al., 2015] [Li et al., 2015] Definition Usually used in parallel [Ghaeini et al., 2018] 61 [Ramnath et al., 2020]
Feature Importance – Machine Translation [Bahdanau et al., 2015] [NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE. Bahdanau et al., ICRL 2015] A well-known example of feature importance is the Attention Mechanism first used in machine translation Image courtesy: https://smerity.com/articles/2016/google_nmt_arch.html Traditional encoder-decoder architecture for machine translation Only the last hidden state from encoder is used 62
[NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE. Bahdanau et al., ICRL 2015] Image courtesy: https://smerity.com/articles/2016/google_nmt_arch.html Attention mechanism uses the latent features learned by the encoder • a weighted combination of the latent features Weights become the importance scores of these latent features (also corresponds to the lexical features) for the decoded word/token. Similar approach in [Mullenbachet al., 2018] Local & Self-explaining Input-output alignment 63 Feature Importance – Machine Translation [Bahdanau et al., 2015]
Feature Importance – token-level explanations [Thorne et al] Another example that uses attention to enable feature importance NLP Task: Natural Language Inference (determining whether a “hypothesis” is true (entailment), false (contradiction), or underdetermined (neutral) given a “premise” Premise: Children Smiling and Waving at a camera Hypothesis: The kids are frowning GloVe LSTM ℎ! ℎ" Attention c h i l d r e n S m i l i n g a n d W a v i n g a t a c a m e r a The kid are frowning MULTIPLE INSTANCE LEARNING to get appropriate thresholds for attention matrix 64
65 Interpreting BERT for Reading Comprehension Based QA • RCQA: given a document and a question, find the answer in the document. Towards Interpreting BERT for Reading Comprehension Based QA. Ramnath et. al. EMNLP 2020 What are the roles of each BERT layer plays?
66 Interpreting BERT for Reading Comprehension Based QA • Use Integrated Gradients (IG) to compute the importance of different features in different BERT layers – IG is one of the most popular techniques to interpret DL models – Will be discussed in later operation section in details. • Observations: – Initial layers: query-passage interaction – Later layers: contextual understanding and enhancing the answer prediction. Towards Interpreting BERT for Reading Comprehension Based QA. Ramnath et. al. EMNLP 2020
Explainability – Surrogate Model Model predictions are explained by learning a second, usually more explainable model, as a proxy. Definition A black-box model or a hard-to-explain model Input data Predictions without explanations or hard to explain. An interpretable model that approximates the black-box model Interpretable predictions [Liu et al. 2018] [Ribeiro et al. KDD] [Alvarez-melis and Jaakkola] [Sydorova et al, 2019] 67
Surrogate Model – LIME [Ribeiro et al. KDD] Blue/Pink area are the decision boundary of a complex model 𝒇, which is a black-box model Cannot be well approximated by a linear function Instance to be explained Sampled instances, weighted by the proximity to the target instance An Interpretable classifier with local fidelity Local & Post-hoc LIME: Local Interpretable Model-agnostic Explanations 68
Surrogate Model – LIME [Ribeiro et al. KDD] Local & Post-hoc High flexibility – free to choose any interpretable surrogate models 69
Surrogate Model - Explaining Network Embedding [Liu et al. 2018] Global & Post-hoc A learned network embedding Induce a taxonomy from the network embedding 70
Surrogate Model - Explaining Network Embedding [Liu et al. 2018] Induced from 20NewsGroups Network Bold text are attributes of the cluster, followed by keywords from documents belong to the cluster. 71
Explainability – Example-driven • Similar to the idea of nearest neighbors [Dudani, 1976] • Have been applied to solve problems including – Text classification [Croce et al., 2019] – Question Answering [Abujabal et al., 2017] Such approaches explain the prediction of an input instance by identifying and presenting other instances, usually from available labeled data, that are semantically similar to the input instance. Definition 72
Explainability – Text Classification [Croce et al., 2019] “What is the capital of Germany?” refers to a Location. WHY? “What is the capital of California?” which also refers to a Location in the training data Because 73
Explainability – Text Classification [Croce et al., 2019] Layer-wise Relevance Propagation in Kernel-based Deep Architecture 74
Explainability – Text Classification [Croce et al., 2019] Need to understand the work Thoroughly 75
Explainability – Provenance-based • An intuitive and effective explainability technique when the final prediction is the result of a series of reasoning steps. • Several question answering papers adopt this approach. – [Abujabal et al., 2017] – Quint system for KBQA (knowledge-base question answering) – [Zhou et al., 2018] – Multi-relation KBQA – [Amini et al., 2019] – MathQA, will be discussed in the declarative induction part (as a representative example of program generation) Explanations are provided by illustrating (some of) the prediction derivation process Definition 76
Provenance-Based: Quint System A. Abujabal et al. QUINT: Interpretable Question Answering over Knowledge Bases. EMNLP System Demonstration, 2017. • When QUINT answers a question, it shows the complete derivation sequence: how it understood the question: o Entity linking: entity mentions in text ⟼ actual KB entities o Relation linking: relations in text ⟼ KB predicates the SPARQL query used to retrieve the answer: “Eisleben” “Where was Martin Luther raised?” place_of_birth Martin_Luther “ … a German professor of theology, composer, priest, monk and …” desc Martin_Luther_King_Jr. desc Learned from query templates based on structurally similar instances (“Where was Obama educated?”) Eisleben Derivation provides insights towards reformulating the question (in case of errors) 77
Provenance-Based: Multi-Relation Question Answering M. Zhou et al. An Interpretable Reasoning Network for Multi- Relation Question Answering. COLING, 2018. IRN (Interpretable Reasoning Network): • Breaks down the KBQA problem into multi-hop reasoning steps. • Intermediate entities and relations are predicted in each step: Reasoning path: John_Hays_Hammond ⟶ e ⟶ a Child Profession Facilitates correction by human (e.g., disambiguation) • multiple answers in this example (3 different children) 78
Explainability – Declarative Induction The main idea of Declarative Induction is to construct human-readable representations such as trees, programs, and rules, which can provide local or global explanations. Explanations are produced in a declarative specification language, as humans often would do. Trees [Jiang et al., ACL 2019] - express alternative ways of reasoning in multi-hop reading comprehension [Wu et al., ACL 2020] – Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT Rules: [A. Ebaid et al., ICDE Demo, 2019] - Bayesian rule lists to explain entity matching decisions [Pezeshkpour et al, NAACL 2019] - rules for interpretability of link prediction [Sen et al, EMNLP 2020] - linguistic expressions as sentence classification model [Qian et al, CIKM 2017] - rules as entity matching model Definition Examples of global explainability (although rules can also be used for local explanations) (Specialized) Programs: [A. Amini et al., NAACL, 2019] – MathQA, math programs for explaining math word problems Examples of local explainability (per instance) 79 79
Declarative Induction: Reasoning Trees for Multi-Hop Reading Comprehension Task: Explore and connect relevant information from multiple sentences/documents to answer a question. Y. Jiang et al. Explore, Propose, and Assemble: An Interpretable Model for Multi-Hop Reading Comprehension. ACL, 2019. Question subject: “Sulphur Spring” Question body: located in administrative territorial entity Reasoning trees play dual role: • Accumulate the information needed to produce the answer • Each root-to-leaf path represents a possible answer and its derivation • Final answer is aggregated across paths • Explain the answer via the tree 80
Declarative Induction: (Specialized) Program Generation Task: Understanding and solving math word problems A. Amini et al. MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. NAACL-HLT, 2019. • Seq-2-Program translation produces a human- interpretable representation, for each math problem • Does not yet handle problems that need complicated or long chains of mathematical reasoning, or more powerful languages (logic, factorization, etc). • Also fits under provenance-based explainability: • provides answer, and how it was derived. Domain-specific program based on math operations • Representative example of a system that provides: • Local (“input/output”) explainability • But no global explanations or insights into how the model operates (still “black-box”) 81
Declarative Induction: Post-Hoc Explanation for Entity Resolution Black-box model ExplainER • Local explanations (e.g., how different features contribute to a single match prediction – based on LIME) • Global explanations Approximate the model with an interpretable representation (Bayesian Rule List -- BRL) • Representative examples Small, diverse set of <input, output> tuples to illustrate where the model is correct or wrong • Differential analysis Compare with other ER model (disagreements, compare performance) A. Ebaid et al. ExplainER: Entity Resolution Explanations. ICDE Demo, 2019. M. T. Ribeiro et al. Why should I trust you: Explaining predictions of any classifier. In SIGKDD, pp. 1135—1144, 2016. B. Letham et al. Interpretable classifiers using rules and Bayesian analysis. Annals of Applied Stat., 9(3), pp. 1350-1371, 2015. Can also be seen as a tree (of probabilistic IF-THEN rules) Entity resolution (aka ER, entity disambiguation, record linking or matching) 82
Declarative Induction: Rules for Link Prediction Task: Link prediction in a knowledge graph, with focus on robustness and explainability P. Pezeshkpour et al. Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications. NAACL 2019. Uses adversarial modifications (e.g., removing facts): 1. Identify the KG nodes/facts most likely to influence a link 2. Aggregate the individual explanations into an extracted set of rules operating at the level of entire KG: • Each rule represents a frequent pattern for predicting the target link Works on top of existing, black-box models for link prediction: DistMult: B. Yang et al. Embedding entities and relations for learning and inference in knowledge bases. ICLR 2015 ConvE: T. Dettmers et al. Convolutional 2d knowledge graph embeddings. AAAI 2018 Global explainability able to pinpoint common mistakes in the underlying model (wrong inference in ConvE) 83
84 Declarative Induction – Interpreting BERT with Perturbed Masking Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT. Wu et. al,. ACL 2020 • BERT is acclaimed for learning useful representations for different NLP tasks • Probing classifiers are used for measure the quality of representations learned by BERT – a small NN that uses BERT embeddings as input for a specific task – Problem: additional NN with its own parameters. • Can we draw insights directly from BERT’s parameters, that is, in a probe-free fashion? • This paper introduces a parameter-free probing technique called Perturbed Masking.
85 Declarative Induction – Interpreting BERT with Perturbed Masking Perturb input sentence and extract an impact matrix Use graph-based dependency parsing algorithm to extract dependency trees out of impact matrix Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT. Wu et. al,. ACL 2020
• Previous approach for rule induction: – The result is a set of logical rules (Horn rules of length 2) that are built on existing predicates from the KG (e.g., Yago). • Other approaches also learn logical rules, but they are based on different vocabularies: – Next: o Logical rules built on linguistic predicates for Sentence Classification – [SystemER – see later] o Logical rules built on similarity features for Entity Resolution 86
Declarative Induction: Linguistic Rules for Sentence Classification Notices will be transmitted electronically, by registered or certified mail, or courier Syntactic and Semantic NLP Operators (Explainable) Deep Learning ∃ verb ∈ 𝑆: verb.theme ∈ ThemeDict and verb.voice = passive and verb.tense = future ThemeDict communication notice Dictionaries Rules (predicates based on syntactic/semantic parsing) “Communication” Sen et al, Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification. EMNLP 2020. • Representative example of system that uses a neural model to learn another model (the rules) • The rules are used for classification but also act as global explanation • Joint learning of both rules and domain-specific dictionaries • Explainable rules generalize better and are more conducive for interaction with the expert (see later – Visualization) Label … “Terms and Termination” … Task: Sentence classification in a domain (e.g., legal/contracts domain) 87
Common Operations to enable explainability • A few commonly used operations that enable different explainability – Attention – First-derivative saliency – Layer-wise relevance propagation – Integrated Gradients – Input perturbations – Explainability-aware architecture design 88
Operations – Attention [NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE. Bahdanau et al., ICRL 2015] Image courtesy: https://smerity.com/articles/2016/google_nmt_arch.html Attention mechanism uses the latent features learned by the encoder • a weighted combination of the latent features Weights become the importance scores of these latent features (also corresponds to the lexical features) for the decoded word/token. Naturally, can be used to enable feature importance [Ghaeini et al., 2018] 89
90 Operations – Attention – Self-attention [images from Stanford CS224] Transformer-based models such as BERT use a multi-head self-attention mechanism Attention weights are still used as importance scores
91 Operations – Attention – Self-attention • Attention score (left) and attribution score (right) of a single head in BERT. • The color is darker for larger values. • Model prediction for the sentence is contradiction. • Attribution tends to identify more sparse word interactions that contribute to the final model decision [Hao et al., 2021]
Operations – First-derivative saliency • Mainly used for enabling Feature Importance – [Li et al., 2016 NAACL], [Aubakirova and Bansal, 2016] • Inspired by the NN visualization method proposed for computer vision – Originally proposed in (Simonyan et al. 2014) – How much each input unit contributes to the final decision of the classifier. Input embedding layer Output of neural network Gradients of the output wrt the input 92
Operations – First-derivative saliency Sentiment Analysis Naturally, it can be used to generate explanations presented by saliency-based visualization techniques [Li et al., 2016 NAACL] 93
Operations – Layer-wise relevance propagation • For a good overview, refer to • An operation that is specific to neural networks – Input can be images, videos, or text • Main idea is very similar to first-derivative saliency – Decomposing the prediction of a deep neural network computed over a sample, down to relevance scores for the single input dimensions of the sample. [Montavon et al., 2019] [Montavon et al., 2019] 94
Operations – Layer-wise relevance propagation • Key difference to first-derivative saliency: – Assigns to each dimension (or feature), 𝑥#, a relevance score 𝑅# (%) such that • Propagate the relevance scores using purposely designed local propagation rules – Basic Rule (LRP-0) – Epsilon Rule (LRP-ϵ) – Gamma Rule (LRP-γ) 𝑓 𝑥 ≈ ( # 𝑅# (%) [Croce et al] 𝑅# (%) > 0 – the feature in favor the prediction 𝑅# (%) < 0 – the feature against the prediction [Montavon et al., 2019] 95
96 Integrated Gradients (IG) • An attribution-based approach that enables Feature Importance over DL models. – Hands-on tutorial on IG using TensorFlow: https://www.tensorflow.org/tutorials/interpretability/inte grated_gradients • Introduced two axioms that every attribution method should satisfy – Sensitivity: for every input and baseline that differ in one feature but have different predictions then the differing feature should be given a non-zero attribution – Implementation invariance: . Two networks are functionally equivalent if their outputs are equal for all inputs, despite having very different implementations. Axiomatic Attribution for Deep Networks. Sundararajan, Taly, Yan. ICML 2017 Image source: https://www.tensorflow.org/tutorials/interpretability/integrated_gradients
97 Integrated Gradients (IG) • Start with a baseline input (e.g., empty input text or all zero embeddings) • Iteratively “move” the baseline input to the final input in the high-dimensional space – cumulate the gradients along the (straight) path from baseline to the final input – The integrated gradient along the i-th dimension for an input x and baseline x’ is defined as • Nice properties of IG: – Satisfy the two aforementioned properties – No additional parameters needed – No training required – Simple to implement Axiomatic Attribution for Deep Networks. Sundararajan, Taly, Yan. ICML 2017 Integrated Gradients is included in Captum: an XAI package for pytorch approximation Image source: https://www.tensorflow.org/tutorials/interpretability/integrated_gradients
Operations – Input perturbation • Usually used to enable local explanation for a particular instance – Often combined with surrogate models 𝑋 𝑓 𝑓(𝑋) An input instance A classifier Usually not interpretable Prediction made by the classifier 𝑋! , 𝑋" , … 𝑋# 𝑓(𝑋! ), 𝑓(𝑋" ), … 𝑓(𝑋# ) Perturbed instances Prediction made by the original classifier 𝒈 A surrogate classifier usually interpretable 𝒈(𝑋) Explanation to humans 98
Operations – Input perturbation • Different ways to generate perturbations – Sampling method (e.g., [Ribeiro et al. KDD] - LIME ) – Perturbating intermediate vector representation (e.g., [Alvarez-Melis and Jaakkola, 2017]) • Sampling method – Directly use examples within labeled data that are semantically similar • Perturbating intermediate vector representation – Specific to neural networks – Introduce minor perturbations to the vector representation of input instance 99
Input perturbation – Sampling method Find examples that are both semantically “close” and “far away” from the instance in question. - Need to define a proximity metric - Will not generate instance [Ribeiro et al. KDD] 100
Perturbating intermediate Vector Representation [Alvarez-Melis and Jaakkola, 2017]) • Explain predictions of any black-box structured input – structured output model • Around a specific input-output pair • An explanation consists of groups of input-output tokens that are causally related. • The dependencies (input-output) are inferred by querying the blackbox model with perturbed inputs 101
Perturbating intermediate Vector Representation [Alvarez-Melis and Jaakkola, 2017]) 102
Explainability-Aware: Rule Induction Architecture Works on top of existing, black-box models for link prediction, via a decoder architecture (i.e., from embeddings back to graph) P. Pezeshkpour et al. Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications. NAACL 2019. Knowledge graph facts: wasBornIn (s, o) Optimization problem in Rd: find the most influential vector zs’r’ whose removal would lead to removal of wasBornIn (s, o) Decode back the vector zs’r’ to actual KG: isLocatedIn (s’, o) Rules of length 2 are then extracted by identifying frequent patterns isAffiliatedTo (a, c) ∧ isLocatedIn (c, b) → wasBornIn (a,b) 103
Part II – (b) Visualization Techniques 104
What’s next XAI model Mathematical justifications End user Visualizing justifications 1. Explanation Generation 2. Explanation Presentation UX engineers AI engineers Visualization Techniques 105
Visualization Techniques • Presenting raw explanations (mathematical justifications) to target user • Ideally, done by UX/UI engineers • Main visualization techniques: – Saliency – Raw declarative representations – Natural language explanation – Raw examples 106
Visualization – Saliency has been primarily used to visualize the importance scores of different types of elements in XAI learning systems Definition • One of the most widely used visualization techniques • Not just in NLP, but also highly used in computer vision • Saliency is popular because it presents visually perceptive explanations • Can be easily understood by different types of users A computer vision example 107
Visualization – Saliency Example Input-output alignment [Bahdanau et al., 2015] 108
Visualization – Saliency Example [Schwarzenberg et al., 2019] [Wallace et al., 2018] [Harbecke et al, 2018] 109 [Antognini et al, 2021]
Visualization – Raw Examples • Use existing knowledge to explain a new instance Explaining by presenting one or more instances (usually from the training data, sometimes generated) that are semantically similar to the instance that needs to be explained Definition Jaguar is a large spotted predator of tropical America similar to the leopard. [Panchenko et al., 2016] Word sense disambiguation Raw examples 110
Visualization – Raw Examples [Croce et al, 2018] Classification Sentences taken from the labeled data Explain with one instance (basic model) Explain with > 1 instance (multiplicative model) Explain with both positive and negative instance (contrastive model) 111
Visualization – Natural language explanation The explanation is verbalized in human-comprehensible language Definition • Can be generated by using sophisticated deep learning approaches – [Rajani et al., 2019] • Can be generated by simple template-based approaches – [Croce et al, 2018] – raw examples and natural language explanations – Many declarative representations can be extended to natural language explanation 112
Visualization – Natural language explanation [Rajani et al., 2019] A language model trained with human generated explanations for commonsense reasoning Open-ended human explanations Explanation generated by CAGE 113
Visualization – Natural language explanation [Croce et al, 2018] Classification Sentences taken from the labeled data Explain with one instance (basic model) Explain with > 1 instance (multiplicative model) Explain with both positive and negative instance (contrastive model) 114
Visualization – Natural language explanation [Croce et al, 2018] Classification Template-based explanatory model 115
Visualization – Declarative Representations • Trees: [Jiang et al., ACL 2019] – Different paths in the tree explain the possible reasoning paths that lead to the answer • Rules: [Pezeshkpour et al, NAACL 2019] – Logic rules to explain link prediction (e.g., isConnected is often predicted because of transitivity rule, while hasChild is often predicted because of spouse rule) [Sen et al, EMNLP 2020] – Linguistic rules for sentence classification: • More than just visualization/explanation • Advanced HCI system for human-machine co- creation of the final model, built on top of rules (Next) isConnected(a,c) ∧ isConnected (c,b) ⟹ isConnected(a,b) isMarriedTo (a,c) ∧ hasChild (c,b) ⟹ hasChild(a,b) ∃ verb ∈ 𝑆: verb.theme ∈ ThemeDict and verb.voice = passive and verb.tense = future Machine-generated rules Human feedback 116
Linguistic Rules: Visualization and Customization by the Expert Passive-Voice Possibility-ModalClass Passive-Voice VerbBase-MatchesDictionary-SectionTitleClues VerbBase-MatchesDictionary-SectionTitleClues Theme-MatchesDictionary-NonAgentSectionTitleClues Theme-MatchesDictionary-NonAgentSectionTitleClues Manner-MatchesDictionary-MannerContextClues Manner-MatchesDictionary-MannerContextClues VerbBase-MatchesDictionary-SectionTitleClues NounPhrase-MatchesDictionary-SectionTitleClues NounPhrase-MatchesDictionary-SectionTitleClues Label being assigned Various ways of selecting/ranking rules Rule-specific performance metrics Y. Yang, E. Kandogan, Y. Li, W. S. Lasecki, P. Sen. HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop. ACL, System Demonstration, 2019. Sentence classification rules learned by [Sen et al, Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification. EMNLP 2020] 117
Visualization of examples per rule Passive-Voice Present-Tense Negative-Polarity Possibility-ModalClass Imperative-Mood Imperative-Mood VerbBase-MatchesDictionary-SectionTitleClues Manner-MatchesDictionary-MannerContextClues Manner-MatchesDictionary-MannerContextClues VerbBase-MatchesDictionary-VerbBases VerbBase-MatchesDictionary-VerbBases Linguistic Rules: Visualization and Customization by the Expert Y. Yang, E. Kandogan, Y. Li, W. S. Lasecki, P. Sen. HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop. ACL, System Demonstration, 2019. 118
Playground mode allows to add/drop predicates Future-Tense Passive-Voice Future-Tense Passive-Voice NounPhrase-MatchesDictionary-SectionTitleClues NounPhrase-MatchesDictionary-SectionTitleClues Linguistic Rules: Visualization and Customization by the Expert Human-machine co-created models generalize better to unseen data • humans instill their expertise by extrapolating from what has been learned by automated algorithms Y. Yang, E. Kandogan, Y. Li, W. S. Lasecki, P. Sen. HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop. ACL, System Demonstration, 2019. 119
120 Paradigms for Communicating Explanations • Static vs. interactive – In an interactive system, explanations can naturally be incorporated as part of the human-AI feedback loop • Examples of systems with interactive explanations – HEIDL (seen before): human-machine co-creation of linguistic models, with explanations based on rules – SystemER – interactive entity resolution, also with explanations based on rules – [Chen et al, IJCAI 2020] • Multi-turn explainable conversation, leads to better recommendations – [Gao et al, IntelliSys 2021] • Proactively explains the AI system to the user ∃ verb ∈ 𝑆: verb.theme ∈ ThemeDict and verb.voice = passive and verb.tense = future Machine-generated rules Human feedback HEIDL
SystemER: Interactive Learning of Explainable Models corporation Zip City address Trader Joe’s 95118 San Jose 5353 Almaden Express Way company Zip City address Trader Joe’s 95110 San Jose 635 Coleman Ave Rule Learning Algorithm R.corporation = S.company and R.city = S.city Candidate rule learning R.corporation = S.company and R.city = S.city and R.zip = S.zip corporation Zip City address KFC 90036 LA 5925 W 3rd St. Not a match refinement company Zip City address KFC 90004 LA 340 N Western Ave Qian et al, SystemER: A human-in-the-loop System for Explainable Entity Resolution. VLDB’17 Demo (also full paper in CIKM’17) Iteratively learns an ER model via active learning from few labels. • Rules: explainable representation for intermediate states and final model • Rules: used as mechanism to generate and explain the examples brought back for labeling Large datasets to match Example selection Domain expert labeling 121
U: D1 x D2 Matches (R1) + + + + + + + + + + + + + + + + + + + + + - - - - - SystemER: Example Selection (Likely False Positives) To refine R1 so that it becomes high-precision, need to find examples from the matches of R1 that are likely to be false positives Negative examples will enhance precision Candidate rule R1 122
U: D1 x D2 Matches (R1) To refine R1 so that it becomes high-precision, need to find examples from the matches of R1 that are likely to be false positives + + + + + + + + + + + + + + + + + + + + + Negative examples will enhance precision - - - - - SystemER: Example Selection (Likely False Negatives) + + Rule-Minus1 + + Rule-Minus2 + Rule-Minus3 Rule-Minus heuristic: explore beyond the current R1 to find what it misses (likely false negatives) R1: i.firstName = t.first AND i.city = t.city AND i.lastName = t.last i.firstName = t.first AND i.city = t.city AND i.lastName = t.last Rule-Minus1 i.firstName = t.first AND i.city = t.city AND i.lastName = t.last Rule-Minus3 i.firstName = t.first AND i.city = t.city AND i.lastName = t.last Rule-Minus2 Candidate rule R1 New positive examples will enhance recall At each step, examples are explained/visualized based on the predicates they satisfy or not satisfy 123
124 Explainable Conversational Recommendation • Multi-turn user vs. model conversation, based on (recommendation + explanation). • Enables continuous improvement of both recommendation and explanation quality. • Suitable for exploration, where users are wandering for interesting items. Chen et al, Towards Explainable Conversational Recommendation. IJCAI 2020. Explanations are the foundation for user feedback: • Users inform the system whether they like, or dislike features mentioned in the explanations • The features in the explanations may also trigger users to suggest related features they like. (-) comedy (-) anime (+) action movie
125 Chat-XAI Design Principles Gao et al, Chat-XAI: A New Chatbot to Explain Artificial Intelligence, IntelliSys 2021 • Key premise: many users do not understand what the AI system can do, and often assume the AI is perfect. • Proactive interactions → better communication between AI and human.
Outline of Part II • Literature review methodology • Categorization of different types of explanation – Local vs. Global – Self-explaining vs. Post-hoc processing • Generating and presenting explanations – Explainability techniques • Common operations to enable explainability – Visualization techniques – Communication Paradigm • Other insights (XNLP website) – Relationships among explainability and visualization techniques • Evaluation of Explanations 126
XNLP • We built an interactive website for exploring the domain • It’s self-contained • It provides different ways for exploring the domain – Cluster View (group papers based on explainability and visualization) – Tree view (categorize papers in a tree like structure) – Citation graphs (show the evolution of the field and also show influential works) – List view (list the set of papers in a table) – Search view (support keyword search and facet search) • IUI’2021 demo 127
XNLP - various ways to explore the literature 128 Link to the website: https://xainlp2020.github.io/xainlp/home
Outline of Part II • Literature review methodology • Categorization of different types of explanation – Local vs. Global – Self-explaining vs. Post-hoc processing • Generating and presenting explanations – Explainability techniques • Common operations to enable explainability – Visualization techniques – Communication Paradigm • Other insights (XNLP website) – Relationships among explainability and visualization techniques • Evaluation of Explanations 129
Evaluation Techniques Informal Examination Explaining How Well Human Evaluation Comparison to Ground Truth 60% 25% 15% [Xie et al., 2017] [Ling et al., 2017] [Sydorova et al., 2019]
Evaluation – Comparison to ground truth • Idea: compare generated explanations to ground truth explanations • Metrics used – Precision/Recall/F1 (Carton et al.,2018) – BLEU (Ling et al., 2017; Rajani et al., 2019b) • Benefit – A quantitative way to measure explainability • Pitfalls – Quality of the ground truth data – Alternative valid explanations may exit • Some attempts to avoid these issues – Having multiple annotators (with inter-annotator agreement) – Evaluating at different granularities (e.g., token-wise vs. phrase-wise) 131
Evaluation – Human Evaluation • Idea: ask humans to evaluate the effectiveness of the generated explanations. • Benefit – Avoiding the assumption that there is only one good explanation that could serve as ground truth • Important aspects – It is important to have multiple annotators (with inter-annotator agreement, avoid subjectivity) • Observations from our literature review – Single-human evaluation (Mullenbach et al., 2018) – Multiple-human evaluation (Sydorova et al., 2019) – Rating the explanations of a single approach (Dong et al., 2019) – Comparing explanations of multiple techniques (Sydorova et al., 2019) • Very few well-established human evaluation strategies. 132
Human Evaluation – Trust Evaluation (Sydorova et al., 2019) • Borrowed the trust evaluation idea used in CV (Selvaraju et al. 2016) • A comparison approach – given two models, find out which one produces more intuitive explanations Model A Model B examples Examples that the two models predict the same labels Explanation 1 Explanation 2 Blind evaluation 133
Human Evaluation – Trust Evaluation (Sydorova et al., 2019) 134
Quality of Explanation – Predictive Process Coverage • The predictive process starts from input to final prediction • Not all XAI approaches cover the whole process • Ideally, the explanation covers the whole process or most of the process • However, many of the approaches cover only a small part of the process • The end user has to fill up the gaps 135
Quality of Explanation – Predictive Process Coverage • Take MathQA (Amini et al. 2019) as an example Induced mathematical operations 136
Quality of Explanation – Predictive Process Coverage • Take MathQA (Amini et al. 2019) as an example 137
Fidelity and faithfulness • Often an issue arises in surrogate model • Surrogate model is flexible since its model-agnostic • Local fidelity vs. global fidelity • Logic fidelity • Surrogate model may use completely different reasoning mechanism • Neglected in the past, now an emerging topic • General guidelines for evaluating faithfulness and fidelity • “Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?” Jacovi and Goldberg. ACL 2020 • Use fidelity to measure the goodness of an explanation approach 138 Evaluating Explanation Methods for Neural Machine Translation. Li et. al, ACL 2020 [Ribeiro et al. KDD]
139 Faithfulness/Fidelity • [Jacovi and Goldberg, 2020] provides a in-depth discussion about faithfulness • Guidelines for evaluating faithfulness: – Be explicit in what you evaluate (Plausibility vs. faithfulness) – Should not involve human-judgement on the quality of the interpretation. – Should not involve human-provided gold labels – Do not trust “inherent interpretability” claims – Evaluation of IUI systems should not rely on user performance. • This work aims to pave the future avenue for faithfulness evaluation research. “Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?” Jacovi and Goldberg. ACL 2020
Evaluate fidelity in Neural Machine Translation Evaluating Explanation Methods for Neural Machine Translation. Li et. al, ACL 2020 Image source: https://developer.nvidia.com/blog/introduction-neural-machine-translation-gpus-part-3/ For a target word predicted, the explanation is usually given by extracting the most relevant words. Different methods may highlight/extract different relevant words, how to measure the quality of the extracted/highlighted words?
Fidelity-based evaluation metric • Construct proxy models for extracted words produced by different explainable approaches • Compare these proxy models with the original NMT model • The explanation method that leads to the the proxy model that agrees with the original NMT model the best wins NMT model Explanation Approach 1 Explanation Approach 2 Word set 1 Word set 2 Proxy model selection Proxy model 1 Proxy model 2 NMT model NMT model vs. vs. The one with lower disagreement rate wins Explanation operations tested • Attention • Gradient-based (first-derivative saliency) • Prediction Difference Proxy model considered • MLP • RNN • Self-attention
PART III – Explainability & Case Study Anbang Xu
Evaluating explanations User-dependent measures2 Comprehensability, User satisfaction Task Oriented Measures2 Task performance Impact on AI interaction - Trust (calibration) in model Task or AI system satisfaction Inherent “goodness” measure2 Fidelity/faithfulness, Stability, Compactness Explainable AI in Industry – KDD ‘19 Comprehensability1 Actionabilty1 Reusability1 Accuracy1 Completeness1 Succintness1 Introduction to Explainable AI – CHI 2021 1. Gade, K., Geyik, S. C., Kenthapadi, K., Mithal, V., & Taly, A. (2019, July). Explainable AI in industry. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 3203-3204). 2. Liao, Q. V., Singh, M., Zhang, Y., & Bellamy, R. (2021, May). Introduction to explainable ai. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-3). 3. Schmidt, P., & Biessmann, F. (2019). Quantifying interpretability and trust in machine learning systems. arXiv preprint arXiv:1901.08558. Trust metric based on the quality of interpretability methods3 1 2 3
Human evaluation of explainability techniques 1 . Zhu, Y., Xian, Y., Fu, Z., de Melo, G., & Zhang, Y. (2021, June). Faithfully Explainable Recommendation via Neural Logic Reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 3083-3090). 2 Doogan, C., & Buntine, W. (2021, June). Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 3824-3848). 3. Wu, H., Chen, W., Xu, S., & Xu, B. (2021, June). Counterfactual Supporting Facts Extraction for Explainable Medical Record Based Diagnosis with Graph Network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 1942-1955). Prior evaluation with a limited number of users: • KG-enhanced explanation paths that connect user to recommendation1 faithfulness • Evaluating topic interpretability2 • Text extraction to support medical diagnosis3 comprehensiveness, trustworthiness
14 5 Explainability Human-Centered Problem
14 6 The role of explanations - Improve trust Decision making with machine assistance e.g. deception detection1 Aim for human-AI complementary performance2 – Does the whole exceed sum of its part? – Slow down to reduce anchor effects – Foster appropriate reliance Full human agency Only explanations Predicted labels with accuracy Predicted labels and explanation Full automation Spectrum of machine assistance 1. Lai, V., & Tan, C. (2019, January). On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency (pp. 29-38). 2 .Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., ... & Weld, D. (2021, May). Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-16).
14 7 Human-centered perspective on XAI • Explanations in social sciences1 Contrastive, Selected, Social. • Leveraging human reasoning processes to craft explanations2 Map explainability techniques with cognitive patterns. 1. Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1-38. 2. Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019, May). Designing theory-driven user-centric explainable AI. In Proceedings of the 2019 CHI conference on human factors in computing systems (pp. 1-15).
14 8 Human-centered perspective on XAI • Explanations embedded within a socio-technical system • Explainability scenarios of possible uses • Human-AI interaction guidelines Ehsan, U., & Riedl, M. O. (2020, July). Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach. In International Conference on Human-Computer Interaction (pp. 449-466). Springer, Cham. Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., ... & Eckersley, P. (2020, January). Explainable machine learning in deployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (pp. 648-657). Wolf, C. T. (2019, March). Explainability scenarios: towards scenario-based XAI design. In Proceedings of the 24th International Conference on Intelligent User Interfaces (pp. 252-257). Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., ... & Teevan, J. (2019, May). Guidelines for human-AI interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-13).
14 9 Broadening XAI Who • Designing for multi-users systems3 • Creator-consumer gap w/r/t AI backgrounds2 • Leveraging questions from designers to design explanations1 1. Liao, Q. V., Gruen, D., & Miller, S. (2020, April). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-15). 2. Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I., Muller, M., & Riedl, M. O. (2021). The Who in Explainable AI: How AI Background Shapes Perceptions of AI Explanations. arXiv preprint arXiv:2107.13509. 3. Jacobs, M., He, J., F. Pradier, M., Lam, B., Ahn, A. C., McCoy, T. H., ... & Gajos, K. Z. (2021, May). Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-14).
Broadening XAI 15 0 Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021, May). Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-16). Who
15 1 Broadening XAI When Onboarding1 AI lifecycle as a lens2,3 1. Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019). " Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making. Proceedings of the ACM on Human-computer Interaction, 3(CSCW), 1-24. 2. Dhanorkar, S., Wolf, C. T., Qian, K., Xu, A., Popa, L., & Li, Y. (2021, June). Who needs to know what, when?: Broadening the Explainable AI (XAI) Design Space by Looking at Explanations Across the AI Lifecycle. In Designing Interactive Systems Conference 2021 (pp. 1591-1602). 3.Hong, S. R., Hullman, J., & Bertini, E. (2020). Human factors in model interpretability: Industry practices, challenges, and needs. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW1), 1-26. 4. Anik, A. I., & Bunt, A. (2021, May). Data-Centric Explanations: Explaining Training Data of Machine Learning Systems to Promote Transparency. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-13). Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021, May). Expanding explainability: Towards social transparency in ai systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-19). What – Model outputs – Inner model working – Data-centric explanations4 – Visibility into context of explanations5
15 2 Human-Centered Approach to Explainabilty Human-Centered approach to Explainability AI lifecycle as a lens: User Study + Case Study Dhanorkar, S., Wolf, C. T., Qian, K., Xu, A., Popa, L., & Li, Y. (2021, June). Who needs to know what, when?: Broadening the Explainable AI (XAI) Design Space by Looking at Explanations Across the AI Lifecycle. In Designing Interactive Systems Conference 2021 (pp. 1591-1602).
15 3 Explainability in Practice RQ2: What is the nature of explanations and explainability concerns in industrial AI projects? RQ1: Over the course of a model’s lifecycle, who seeks and provides explanations? When do explanation needs arise?
Domains Drug safety, key opinion leader mining, information extraction from business communications e.g., emails, contract document understanding, HR, chatbots, public health monitoring …. Interview Participants AI Engineers -7 Data Scientists - 13 Designers – 2 Information Vis Specialists - 3 Product Managers - 3 Technical Strategists - 5 *some individuals were overlapping in two roles
15 5 Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Business Users Product Managers Data Scientists Domain Experts Business Users Data Providers AI Engineers Data Scientists AI Engineers Data Scientists Product Managers Model Validators Business Users Designers Software Engineers Product Managers AI Engineers Business Users AI Operations Product Managers Model Validators
15 6 AI Lifecycle Touchpoints Initial Model building Model validation during proof-of-concept Model in-production Audience (Whom does the AI model interface with) Model Developers Data Scientists Product Managers Domain Experts Business Users Business IT Operations Model Developers Data Scientists | Technical Strategists Product Managers | Design teams BusinessUsers Business IT Operations Explainability Motivations (Information needs) - Peeking inside models to understand their inner workings - Improving model design (e.g., how should the model be retrained, re- tuned) - Selecting the right model - Characteristics of data (proprietary, public, training data) - Understanding model design - Ensuring ethical model development - Expectation mismatch - Augmenting business workflow and business actionability
15 7 Explanations during model development Understanding AI models inner workings “Explanations can be useful to NLP researchers explore and come up with hypothesis about why their models are working well… If you know the layer and the head then you have all the information you need to remove its influence... by looking, you could say, oh this head at this layer is only causing adverse effects, kill it and retrain or you could tweak it perhaps in such a way to minimize bias (I-19, Data scientist) “Low-level details like hyper-parameters would be discussed for debugging or brainstorming. (I-12, Data Scientist)
15 8 Explanations during model validation Details about data over which model is built “Domain experts want to know more about the public medical dataset that the model is trained on to gauge if it can adapt well to their proprietary data (I-4, Data Scientist) Model design at a high level “Initially we presented everything in the typical AI way (i.e., showing the diagram of the model). We even put equations but realized this will not work... After a few weeks, we started to only show examples and describing how the model works at a high level with these examples (I-4)
15 9 Explanations during model validation “How do we know the model is safe to use? ... Users will ask questions about regulatory or compliance-related factors: Does the model identify this particular part of the GDPR law? (I-16, Product Manager) Ethical Considerations
16 0 Explanations during model in-production Expectation mismatch “Data quality issues might arise resulting from expectation mismatch, but the list of recommendations must at least be as good as the manual process ... If they [the clients] come back with data quality issues ... we need to [provide] an explanation of what happened (I-5, Technical Strategist) “Model mistakes versus decision disagreements” (I-28, Technical Strategist) Explanation in service of business actionability “Is the feature it is pointing to something I can change in my business? If not, how does knowing that help me? (I-22, AI Engineer)
Explainability in HEIDL HEIDL (Human-in-the loop linguistic Expressions wIth Deep Learning) Explainability takes on two dimensions here: models are fully explainable users involved In model co-creation with immediate feedback HEIDL video
16 2 Design Implications of AI Explainability Balancing external stakeholders needs with their AI knowledge “We have to balance the information you are sharing about the AI underpinnings, it can overwhelm the user (I-21, UX Designer) Group collaboration persona describing distinct types of members “Loss of control making the support and maintenance of explainable models hard (I-8, Data Scientist) Simplicity versus complexity tradeoff “The design space for (explaining) models to end users is in a way more constrained ... you have to assume that you have to put in a very, very very shallow learning curve (I-27 HCI Researcher) Privacy “Explanatory features can reveal identities (e.g., easily inferring employee, department, etc.) (I- 24, HCI researcher)
16 3 Future Work - User-Centered Design of AI Explainability Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Image source: https://www.slideshare.net/agaszostek/history-and-future-of-human-computer-interaction-hci-and-interaction-design Business Users Product Managers AI Engineers Data Scientists End-Users
16 4 Future Work - User-Centered Design of AI Explainability Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Image source: https://www.slideshare.net/agaszostek/history-and-future-of-human-computer-interaction-hci-and-interaction-design Business Users Product Managers AI Engineers Data Scientists End-Users
16 5 Future Work - User-Centered Design of AI Explainability Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Image source: https://www.slideshare.net/agaszostek/history-and-future-of-human-computer-interaction-hci-and-interaction-design Business Users Product Managers AI Engineers Data Scientists End-Users
PART IV – Open Challenges & Concluding Remarks Marina Danilevsky Yunyao Li Lucian Popa Kun Qian Anbang Xu
Explainability for NLP is an Emerging Field • AI has become an important part of our daily lives • Explainability is becoming increasingly important • Explainability for NLP is still at its early stage
Challenge 1: Standardized Terminology Interpretability explainability transparency Directly interpretable Self-explaining Understandability Comprehensibility Black-box-ness White-box Gray-box … … … 168
Challenge 2: What counts as explainable? 169 Attention weights indicate most predictive tokens Attention mechanisms are noisy – even for intermediate input components’ importance Attention weights are uncorrelated with gradient-based feature importance measures Different attention distributions can yield equivalent predictions May be hard to say tokens are “responsible for” model output – should not treat attention as decision justification Attention can give a plausible reconstruction though with no guarantee of faithfulness A model can be forced to reduce attention on predictive tokens, corrupting the attention-based explanation Depends on what you’re looking for: plausible or faithful explanations?
Challenge 3: Stakeholder-Oriented Explainability Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Business Users Product Managers AI Engineers Data Scientists End-Users 170
Challenge 3: Stakeholder-Oriented Explainability Statement of Work Company A Company B … Time is of essence of this Agreement. …. label = Risky Contains($sentence, $time) = False AI engineer Statement of Work Company A Company B … Time is of essence of this Agreement. …. … Risky. Need to define time frame to make this clause meaningful. Potential risk: If no time specified, courts will apply a "reasonable" time to perform. What is a "reasonable time" is a question of fact - so it will up to a jury. legal professional Different stakeholders have different expectations and skills 171
Challenge 4: Explainability AND Accuracy Current Focus Holy Grail Explainability Accuracy H H L L Explain Blackbox ML Models Fully Explainable Models Source: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature. Machine Intelligence. Surrogate models: • OK to explain a black-box model, but fidelity suffers Model design geared towards explainability • Can be both explainable and high-accuracy! 172
Challenge 5: Appropriate Trust-Oriented Explainability • Engender appropriate trust – Help human trust the AI, when it is right – Expose the AI’s errors, when it is wrong • Maximize AI and explanation’s helpfulness • Take nature of the task into account 173 Credit: Dan Weld. Keynote talk “Optimizing Human-AI Teams”. DaSH-LA @NAACL’2021 Source: “Does the whole exceed its parts? The effect of AI explanations on complementary team performance. CHI’2021c
Challenge 6: Evaluation Methodology & Metrics • Need more well-established evaluation methodologies • Need more fine-grain evaluation metrics – Coverage of the prediction – Fidelity • Take into account the stakeholders as well! 174
Path Forward • Explainability research for NLP is an interdisciplinary topic – Computational Linguistics – Machine learning – Human computer interaction • NLP community and HCI community can work closer in the future – Explainability research is essentially user-oriented – but most of the papers we reviewed did not have any form of user study – Get the low-hanging fruit from the HCI community This tutorial is on a hot area in NLP and takes an innovative approach by weaving in not only core technical research but also user-oriented research. I believe this tutorial has the potential to draw a reasonably sized crowd. – anonymous tutorial proposal reviewer “Furthermore, I really appreciate the range of perspectives (social sciences, HCI, and NLP) represented among the presenters of this tutorial.” – anonymous tutorial proposal reviewer NLP people appreciate the idea of bringing NLP and HCI people together 175
Thank You • Project website: https://xainlp.github.io/ – Tutorial website: https://xainlp.github.io/kddtutorial/ • Questions and discussion 176

Recommended

Explainability for Natural Language Processing
Explainability for Natural Language ProcessingExplainability for Natural Language Processing
Explainability for Natural Language Processing
Yunyao Li
 
Explainability for Natural Language Processing
Explainability for Natural Language ProcessingExplainability for Natural Language Processing
Explainability for Natural Language Processing
Yunyao Li
 
Explainable AI in Industry (WWW 2020 Tutorial)
Explainable AI in Industry (WWW 2020 Tutorial)Explainable AI in Industry (WWW 2020 Tutorial)
Explainable AI in Industry (WWW 2020 Tutorial)
Krishnaram Kenthapadi
 
Explainable AI in Industry (FAT* 2020 Tutorial)
Explainable AI in Industry (FAT* 2020 Tutorial)Explainable AI in Industry (FAT* 2020 Tutorial)
Explainable AI in Industry (FAT* 2020 Tutorial)
Krishnaram Kenthapadi
 
Explainable AI (XAI)
Explainable AI (XAI)Explainable AI (XAI)
Explainable AI (XAI)
Manojkumar Parmar
 
Explainable AI in Industry (KDD 2019 Tutorial)
Explainable AI in Industry (KDD 2019 Tutorial)Explainable AI in Industry (KDD 2019 Tutorial)
Explainable AI in Industry (KDD 2019 Tutorial)
Krishnaram Kenthapadi
 
Explainable AI - making ML and DL models more interpretable
Explainable AI - making ML and DL models more interpretableExplainable AI - making ML and DL models more interpretable
Explainable AI - making ML and DL models more interpretable
Aditya Bhattacharya
 
Explainable AI is not yet Understandable AI
Explainable AI is not yet Understandable AIExplainable AI is not yet Understandable AI
Explainable AI is not yet Understandable AI
epsilon_tud
 

Recommended

Explainability for Natural Language Processing
Explainability for Natural Language ProcessingExplainability for Natural Language Processing
Explainability for Natural Language Processing
Yunyao Li
 
Explainability for Natural Language Processing
Explainability for Natural Language ProcessingExplainability for Natural Language Processing
Explainability for Natural Language Processing
Yunyao Li
 
Explainable AI in Industry (WWW 2020 Tutorial)
Explainable AI in Industry (WWW 2020 Tutorial)Explainable AI in Industry (WWW 2020 Tutorial)
Explainable AI in Industry (WWW 2020 Tutorial)
Krishnaram Kenthapadi
 
Explainable AI in Industry (FAT* 2020 Tutorial)
Explainable AI in Industry (FAT* 2020 Tutorial)Explainable AI in Industry (FAT* 2020 Tutorial)
Explainable AI in Industry (FAT* 2020 Tutorial)
Krishnaram Kenthapadi
 
Explainable AI (XAI)
Explainable AI (XAI)Explainable AI (XAI)
Explainable AI (XAI)
Manojkumar Parmar
 
Explainable AI in Industry (KDD 2019 Tutorial)
Explainable AI in Industry (KDD 2019 Tutorial)Explainable AI in Industry (KDD 2019 Tutorial)
Explainable AI in Industry (KDD 2019 Tutorial)
Krishnaram Kenthapadi
 
Explainable AI - making ML and DL models more interpretable
Explainable AI - making ML and DL models more interpretableExplainable AI - making ML and DL models more interpretable
Explainable AI - making ML and DL models more interpretable
Aditya Bhattacharya
 
Explainable AI is not yet Understandable AI
Explainable AI is not yet Understandable AIExplainable AI is not yet Understandable AI
Explainable AI is not yet Understandable AI
epsilon_tud
 
Fairness and Privacy in AI/ML Systems
Fairness and Privacy in AI/ML SystemsFairness and Privacy in AI/ML Systems
Fairness and Privacy in AI/ML Systems
Krishnaram Kenthapadi
 
220206 transformer interpretability beyond attention visualization
220206 transformer interpretability beyond attention visualization220206 transformer interpretability beyond attention visualization
220206 transformer interpretability beyond attention visualization
taeseon ryu
 
Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...
Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...
Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...
Sri Ambati
 
Synthetic Data Generation with DoppelGanger
Synthetic Data Generation with DoppelGangerSynthetic Data Generation with DoppelGanger
Synthetic Data Generation with DoppelGanger
QuantUniversity
 
Explainability and bias in AI
Explainability and bias in AIExplainability and bias in AI
Explainability and bias in AI
Bill Liu
 
Explainable AI in Industry (AAAI 2020 Tutorial)
Explainable AI in Industry (AAAI 2020 Tutorial)Explainable AI in Industry (AAAI 2020 Tutorial)
Explainable AI in Industry (AAAI 2020 Tutorial)
Krishnaram Kenthapadi
 
Developing Data Products
Developing Data ProductsDeveloping Data Products
Developing Data Products
Peter Skomoroch
 
Explainable AI
Explainable AIExplainable AI
Explainable AI
Wagston Staehler
 
eScience SHAP talk
eScience SHAP talkeScience SHAP talk
eScience SHAP talk
Scott Lundberg
 
Explainable AI (XAI) - A Perspective
Explainable AI (XAI) - A Perspective Explainable AI (XAI) - A Perspective
Explainable AI (XAI) - A Perspective
Saurabh Kaushik
 
Unified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIMEUnified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIME
Databricks
 
Responsible AI in Industry (Tutorials at AAAI 2021, FAccT 2021, and WWW 2021)
Responsible AI in Industry (Tutorials at AAAI 2021, FAccT 2021, and WWW 2021)Responsible AI in Industry (Tutorials at AAAI 2021, FAccT 2021, and WWW 2021)
Responsible AI in Industry (Tutorials at AAAI 2021, FAccT 2021, and WWW 2021)
Krishnaram Kenthapadi
 
Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...
Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...
Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...
Sri Ambati
 
Interpretable machine learning : Methods for understanding complex models
Interpretable machine learning : Methods for understanding complex modelsInterpretable machine learning : Methods for understanding complex models
Interpretable machine learning : Methods for understanding complex models
Manojit Nandi
 
Federated Learning
Federated LearningFederated Learning
Federated Learning
DataWorks Summit
 
An Introduction to XAI! Towards Trusting Your ML Models!
An Introduction to XAI! Towards Trusting Your ML Models!An Introduction to XAI! Towards Trusting Your ML Models!
An Introduction to XAI! Towards Trusting Your ML Models!
Mansour Saffar
 
Shap
ShapShap
Shap
Giovanni Bruner
 
Introduction to Interpretable Machine Learning
Introduction to Interpretable Machine LearningIntroduction to Interpretable Machine Learning
Introduction to Interpretable Machine Learning
Nguyen Giang
 
Introduction to machine learning
Introduction to machine learningIntroduction to machine learning
Introduction to machine learning
Koundinya Desiraju
 
ML Drift - How to find issues before they become problems
ML Drift - How to find issues before they become problemsML Drift - How to find issues before they become problems
ML Drift - How to find issues before they become problems
Amy Hodler
 
How Will Your ML Project Fail
How Will Your ML Project FailHow Will Your ML Project Fail
How Will Your ML Project Fail
Elena Samuylova
 
Machine Learning for Finance Master Class
Machine Learning for Finance Master Class Machine Learning for Finance Master Class
Machine Learning for Finance Master Class
QuantUniversity
 

More Related Content

What's hot

Fairness and Privacy in AI/ML Systems
Fairness and Privacy in AI/ML SystemsFairness and Privacy in AI/ML Systems
Fairness and Privacy in AI/ML Systems
Krishnaram Kenthapadi
 
220206 transformer interpretability beyond attention visualization
220206 transformer interpretability beyond attention visualization220206 transformer interpretability beyond attention visualization
220206 transformer interpretability beyond attention visualization
taeseon ryu
 
Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...
Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...
Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...
Sri Ambati
 
Synthetic Data Generation with DoppelGanger
Synthetic Data Generation with DoppelGangerSynthetic Data Generation with DoppelGanger
Synthetic Data Generation with DoppelGanger
QuantUniversity
 
Explainable AI in Industry (AAAI 2020 Tutorial)
Explainable AI in Industry (AAAI 2020 Tutorial)Explainable AI in Industry (AAAI 2020 Tutorial)
Explainable AI in Industry (AAAI 2020 Tutorial)
Krishnaram Kenthapadi
 
Unified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIMEUnified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIME
Databricks
 
Responsible AI in Industry (Tutorials at AAAI 2021, FAccT 2021, and WWW 2021)
Responsible AI in Industry (Tutorials at AAAI 2021, FAccT 2021, and WWW 2021)Responsible AI in Industry (Tutorials at AAAI 2021, FAccT 2021, and WWW 2021)
Responsible AI in Industry (Tutorials at AAAI 2021, FAccT 2021, and WWW 2021)
Krishnaram Kenthapadi
 
Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...
Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...
Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...
Sri Ambati
 
Federated Learning
Federated LearningFederated Learning
Federated Learning
DataWorks Summit
 
An Introduction to XAI! Towards Trusting Your ML Models!
An Introduction to XAI! Towards Trusting Your ML Models!An Introduction to XAI! Towards Trusting Your ML Models!
An Introduction to XAI! Towards Trusting Your ML Models!
Mansour Saffar
 

What's hot (20)

Fairness and Privacy in AI/ML Systems
Fairness and Privacy in AI/ML SystemsFairness and Privacy in AI/ML Systems
Fairness and Privacy in AI/ML Systems
 
220206 transformer interpretability beyond attention visualization
220206 transformer interpretability beyond attention visualization220206 transformer interpretability beyond attention visualization
220206 transformer interpretability beyond attention visualization
 
Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...
Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...
Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shaple...
 
Synthetic Data Generation with DoppelGanger
Synthetic Data Generation with DoppelGangerSynthetic Data Generation with DoppelGanger
Synthetic Data Generation with DoppelGanger
 
Explainability and bias in AI
Explainability and bias in AIExplainability and bias in AI
Explainability and bias in AI
 
Explainable AI in Industry (AAAI 2020 Tutorial)
Explainable AI in Industry (AAAI 2020 Tutorial)Explainable AI in Industry (AAAI 2020 Tutorial)
Explainable AI in Industry (AAAI 2020 Tutorial)
 
Developing Data Products
Developing Data ProductsDeveloping Data Products
Developing Data Products
 
Explainable AI
Explainable AIExplainable AI
Explainable AI
 
eScience SHAP talk
eScience SHAP talkeScience SHAP talk
eScience SHAP talk
 
Explainable AI (XAI) - A Perspective
Explainable AI (XAI) - A Perspective Explainable AI (XAI) - A Perspective
Explainable AI (XAI) - A Perspective
 
Unified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIMEUnified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIME
 
Responsible AI in Industry (Tutorials at AAAI 2021, FAccT 2021, and WWW 2021)
Responsible AI in Industry (Tutorials at AAAI 2021, FAccT 2021, and WWW 2021)Responsible AI in Industry (Tutorials at AAAI 2021, FAccT 2021, and WWW 2021)
Responsible AI in Industry (Tutorials at AAAI 2021, FAccT 2021, and WWW 2021)
 
Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...
Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...
Interpretable Machine Learning Using LIME Framework - Kasia Kulma (PhD), Data...
 
Interpretable machine learning : Methods for understanding complex models
Interpretable machine learning : Methods for understanding complex modelsInterpretable machine learning : Methods for understanding complex models
Interpretable machine learning : Methods for understanding complex models
 
Federated Learning
Federated LearningFederated Learning
Federated Learning
 
An Introduction to XAI! Towards Trusting Your ML Models!
An Introduction to XAI! Towards Trusting Your ML Models!An Introduction to XAI! Towards Trusting Your ML Models!
An Introduction to XAI! Towards Trusting Your ML Models!
 
Shap
ShapShap
Shap
 
Introduction to Interpretable Machine Learning
Introduction to Interpretable Machine LearningIntroduction to Interpretable Machine Learning
Introduction to Interpretable Machine Learning
 
Introduction to machine learning
Introduction to machine learningIntroduction to machine learning
Introduction to machine learning
 
ML Drift - How to find issues before they become problems
ML Drift - How to find issues before they become problemsML Drift - How to find issues before they become problems
ML Drift - How to find issues before they become problems
 

Similar to Explainability for Natural Language Processing

Model governance in the age of data science & AI
Model governance in the age of data science & AIModel governance in the age of data science & AI
Model governance in the age of data science & AI
QuantUniversity
 
Responsible AI in Industry: Practical Challenges and Lessons Learned
Responsible AI in Industry: Practical Challenges and Lessons LearnedResponsible AI in Industry: Practical Challenges and Lessons Learned
Responsible AI in Industry: Practical Challenges and Lessons Learned
Krishnaram Kenthapadi
 
If You Are Not Embedding Analytics Into Your Day To Day Processes, You Are Do...
If You Are Not Embedding Analytics Into Your Day To Day Processes, You Are Do...If You Are Not Embedding Analytics Into Your Day To Day Processes, You Are Do...
If You Are Not Embedding Analytics Into Your Day To Day Processes, You Are Do...
Dell World
 
UCLA MSBA LinkedIn Industry Seminar 2019-02-27
UCLA MSBA LinkedIn Industry Seminar 2019-02-27UCLA MSBA LinkedIn Industry Seminar 2019-02-27
UCLA MSBA LinkedIn Industry Seminar 2019-02-27
Jimmy Wong
 
Streaming analytics webinar | 9.13.16 | Guest: Mike Gualtieri from Forrester
Streaming analytics webinar | 9.13.16 | Guest: Mike Gualtieri from ForresterStreaming analytics webinar | 9.13.16 | Guest: Mike Gualtieri from Forrester
Streaming analytics webinar | 9.13.16 | Guest: Mike Gualtieri from Forrester
Cubic Corporation
 

Similar to Explainability for Natural Language Processing (20)

How Will Your ML Project Fail
How Will Your ML Project FailHow Will Your ML Project Fail
How Will Your ML Project Fail
 
Machine Learning for Finance Master Class
Machine Learning for Finance Master Class Machine Learning for Finance Master Class
Machine Learning for Finance Master Class
 
Ai design sprint - Finance - Wealth management
Ai design sprint - Finance - Wealth managementAi design sprint - Finance - Wealth management
Ai design sprint - Finance - Wealth management
 
Self-service Analytic for Business Users-19july2017-final
Self-service Analytic for Business Users-19july2017-finalSelf-service Analytic for Business Users-19july2017-final
Self-service Analytic for Business Users-19july2017-final
 
Model governance in the age of data science & AI
Model governance in the age of data science & AIModel governance in the age of data science & AI
Model governance in the age of data science & AI
 
Responsible AI in Industry: Practical Challenges and Lessons Learned
Responsible AI in Industry: Practical Challenges and Lessons LearnedResponsible AI in Industry: Practical Challenges and Lessons Learned
Responsible AI in Industry: Practical Challenges and Lessons Learned
 
If You Are Not Embedding Analytics Into Your Day To Day Processes, You Are Do...
If You Are Not Embedding Analytics Into Your Day To Day Processes, You Are Do...If You Are Not Embedding Analytics Into Your Day To Day Processes, You Are Do...
If You Are Not Embedding Analytics Into Your Day To Day Processes, You Are Do...
 
Get your data analytics strategy right!
Get your data analytics strategy right!Get your data analytics strategy right!
Get your data analytics strategy right!
 
Robert Brooks, PwC
Robert Brooks, PwCRobert Brooks, PwC
Robert Brooks, PwC
 
State of the Market - Data Quality in 2023
State of the Market - Data Quality in 2023State of the Market - Data Quality in 2023
State of the Market - Data Quality in 2023
 
Data Science and Analytics
Data Science and Analytics Data Science and Analytics
Data Science and Analytics
 
Using the power of OpenAI with your own data: what's possible and how to start?
Using the power of OpenAI with your own data: what's possible and how to start?Using the power of OpenAI with your own data: what's possible and how to start?
Using the power of OpenAI with your own data: what's possible and how to start?
 
AI based Testing in Finance & Retail Breakfast Briefing
AI based Testing in Finance & Retail Breakfast BriefingAI based Testing in Finance & Retail Breakfast Briefing
AI based Testing in Finance & Retail Breakfast Briefing
 
UCLA MSBA LinkedIn Industry Seminar 2019-02-27
UCLA MSBA LinkedIn Industry Seminar 2019-02-27UCLA MSBA LinkedIn Industry Seminar 2019-02-27
UCLA MSBA LinkedIn Industry Seminar 2019-02-27
 
Recommender System- Analyzing products by mining Data Streams
Recommender System- Analyzing products by mining Data StreamsRecommender System- Analyzing products by mining Data Streams
Recommender System- Analyzing products by mining Data Streams
 
Projects
ProjectsProjects
Projects
 
Machine Learning for Business - Eight Best Practices for Getting Started
Machine Learning for Business - Eight Best Practices for Getting StartedMachine Learning for Business - Eight Best Practices for Getting Started
Machine Learning for Business - Eight Best Practices for Getting Started
 
Context Is Critical for IT Operations - How Rich Data Yields Richer Results
Context Is Critical for IT Operations - How Rich Data Yields Richer Results Context Is Critical for IT Operations - How Rich Data Yields Richer Results
Context Is Critical for IT Operations - How Rich Data Yields Richer Results
 
Algorithmic auditing 1.0
Algorithmic auditing 1.0Algorithmic auditing 1.0
Algorithmic auditing 1.0
 
Streaming analytics webinar | 9.13.16 | Guest: Mike Gualtieri from Forrester
Streaming analytics webinar | 9.13.16 | Guest: Mike Gualtieri from ForresterStreaming analytics webinar | 9.13.16 | Guest: Mike Gualtieri from Forrester
Streaming analytics webinar | 9.13.16 | Guest: Mike Gualtieri from Forrester
 

More from Yunyao Li

Building, Growing and Serving Large Knowledge Graphs with Human-in-the-Loop
Building, Growing and Serving Large Knowledge Graphs with Human-in-the-LoopBuilding, Growing and Serving Large Knowledge Graphs with Human-in-the-Loop
Building, Growing and Serving Large Knowledge Graphs with Human-in-the-Loop
Yunyao Li
 
Meaning Representations for Natural Languages: Design, Models and Applications
Meaning Representations for Natural Languages: Design, Models and ApplicationsMeaning Representations for Natural Languages: Design, Models and Applications
Meaning Representations for Natural Languages: Design, Models and Applications
Yunyao Li
 
Taming the Wild West of NLP
Taming the Wild West of NLPTaming the Wild West of NLP
Taming the Wild West of NLP
Yunyao Li
 
Towards Deep Table Understanding
Towards Deep Table UnderstandingTowards Deep Table Understanding
Towards Deep Table Understanding
Yunyao Li
 
Human in the Loop AI for Building Knowledge Bases
Human in the Loop AI for Building Knowledge Bases Human in the Loop AI for Building Knowledge Bases
Human in the Loop AI for Building Knowledge Bases
Yunyao Li
 
Towards Universal Language Understanding
Towards Universal Language UnderstandingTowards Universal Language Understanding
Towards Universal Language Understanding
Yunyao Li
 
Towards Universal Language Understanding (2020 version)
Towards Universal Language Understanding (2020 version)Towards Universal Language Understanding (2020 version)
Towards Universal Language Understanding (2020 version)
Yunyao Li
 
Towards Universal Semantic Understanding of Natural Languages
Towards Universal Semantic Understanding of Natural LanguagesTowards Universal Semantic Understanding of Natural Languages
Towards Universal Semantic Understanding of Natural Languages
Yunyao Li
 
An In-depth Analysis of the Effect of Text Normalization in Social Media
An In-depth Analysis of the Effect of Text Normalization in Social MediaAn In-depth Analysis of the Effect of Text Normalization in Social Media
An In-depth Analysis of the Effect of Text Normalization in Social Media
Yunyao Li
 
Exploiting Structure in Representation of Named Entities using Active Learning
Exploiting Structure in Representation of Named Entities using Active LearningExploiting Structure in Representation of Named Entities using Active Learning
Exploiting Structure in Representation of Named Entities using Active Learning
Yunyao Li
 
K-SRL: Instance-based Learning for Semantic Role Labeling
K-SRL: Instance-based Learning for Semantic Role LabelingK-SRL: Instance-based Learning for Semantic Role Labeling
K-SRL: Instance-based Learning for Semantic Role Labeling
Yunyao Li
 
Natural Language Data Management and Interfaces: Recent Development and Open ...
Natural Language Data Management and Interfaces: Recent Development and Open ...Natural Language Data Management and Interfaces: Recent Development and Open ...
Natural Language Data Management and Interfaces: Recent Development and Open ...
Yunyao Li
 
Polyglot: Multilingual Semantic Role Labeling with Unified Labels
Polyglot: Multilingual Semantic Role Labeling with Unified LabelsPolyglot: Multilingual Semantic Role Labeling with Unified Labels
Polyglot: Multilingual Semantic Role Labeling with Unified Labels
Yunyao Li
 
The Power of Declarative Analytics
The Power of Declarative AnalyticsThe Power of Declarative Analytics
The Power of Declarative Analytics
Yunyao Li
 
Enterprise Search in the Big Data Era: Recent Developments and Open Challenges
Enterprise Search in the Big Data Era: Recent Developments and Open ChallengesEnterprise Search in the Big Data Era: Recent Developments and Open Challenges
Enterprise Search in the Big Data Era: Recent Developments and Open Challenges
Yunyao Li
 

More from Yunyao Li (20)

The Role of Patterns in the Era of Large Language Models
The Role of Patterns in the Era of Large Language ModelsThe Role of Patterns in the Era of Large Language Models
The Role of Patterns in the Era of Large Language Models
 
Building, Growing and Serving Large Knowledge Graphs with Human-in-the-Loop
Building, Growing and Serving Large Knowledge Graphs with Human-in-the-LoopBuilding, Growing and Serving Large Knowledge Graphs with Human-in-the-Loop
Building, Growing and Serving Large Knowledge Graphs with Human-in-the-Loop
 
Meaning Representations for Natural Languages: Design, Models and Applications
Meaning Representations for Natural Languages: Design, Models and ApplicationsMeaning Representations for Natural Languages: Design, Models and Applications
Meaning Representations for Natural Languages: Design, Models and Applications
 
Taming the Wild West of NLP
Taming the Wild West of NLPTaming the Wild West of NLP
Taming the Wild West of NLP
 
Towards Deep Table Understanding
Towards Deep Table UnderstandingTowards Deep Table Understanding
Towards Deep Table Understanding
 
Human in the Loop AI for Building Knowledge Bases
Human in the Loop AI for Building Knowledge Bases Human in the Loop AI for Building Knowledge Bases
Human in the Loop AI for Building Knowledge Bases
 
Towards Universal Language Understanding
Towards Universal Language UnderstandingTowards Universal Language Understanding
Towards Universal Language Understanding
 
Towards Universal Language Understanding (2020 version)
Towards Universal Language Understanding (2020 version)Towards Universal Language Understanding (2020 version)
Towards Universal Language Understanding (2020 version)
 
Towards Universal Semantic Understanding of Natural Languages
Towards Universal Semantic Understanding of Natural LanguagesTowards Universal Semantic Understanding of Natural Languages
Towards Universal Semantic Understanding of Natural Languages
 
An In-depth Analysis of the Effect of Text Normalization in Social Media
An In-depth Analysis of the Effect of Text Normalization in Social MediaAn In-depth Analysis of the Effect of Text Normalization in Social Media
An In-depth Analysis of the Effect of Text Normalization in Social Media
 
Exploiting Structure in Representation of Named Entities using Active Learning
Exploiting Structure in Representation of Named Entities using Active LearningExploiting Structure in Representation of Named Entities using Active Learning
Exploiting Structure in Representation of Named Entities using Active Learning
 
K-SRL: Instance-based Learning for Semantic Role Labeling
K-SRL: Instance-based Learning for Semantic Role LabelingK-SRL: Instance-based Learning for Semantic Role Labeling
K-SRL: Instance-based Learning for Semantic Role Labeling
 
Coling poster
Coling posterColing poster
Coling poster
 
Coling demo
Coling demoColing demo
Coling demo
 
Natural Language Data Management and Interfaces: Recent Development and Open ...
Natural Language Data Management and Interfaces: Recent Development and Open ...Natural Language Data Management and Interfaces: Recent Development and Open ...
Natural Language Data Management and Interfaces: Recent Development and Open ...
 
Polyglot: Multilingual Semantic Role Labeling with Unified Labels
Polyglot: Multilingual Semantic Role Labeling with Unified LabelsPolyglot: Multilingual Semantic Role Labeling with Unified Labels
Polyglot: Multilingual Semantic Role Labeling with Unified Labels
 
Transparent Machine Learning for Information Extraction: State-of-the-Art and...
Transparent Machine Learning for Information Extraction: State-of-the-Art and...Transparent Machine Learning for Information Extraction: State-of-the-Art and...
Transparent Machine Learning for Information Extraction: State-of-the-Art and...
 
The Power of Declarative Analytics
The Power of Declarative AnalyticsThe Power of Declarative Analytics
The Power of Declarative Analytics
 
Enterprise Search in the Big Data Era: Recent Developments and Open Challenges
Enterprise Search in the Big Data Era: Recent Developments and Open ChallengesEnterprise Search in the Big Data Era: Recent Developments and Open Challenges
Enterprise Search in the Big Data Era: Recent Developments and Open Challenges
 
SystemT: Declarative Information Extraction
SystemT: Declarative Information ExtractionSystemT: Declarative Information Extraction
SystemT: Declarative Information Extraction
 

Recently uploaded

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
sammart93
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Jeffrey Haguewood
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 

Recently uploaded (20)

Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 

Explainability for Natural Language Processing

  • 1. 1 Explainability for Natural Language Processing 1 IBM Research – Almaden 2 Pennnsylvania State University. 3 Amazon 4 IBM Watson Lecture-style tutorial@KDD'2021 https://xainlp.github.io/ 1 Marina Danilevsky 2 Shipi Dhanorkar 1 Yunyao Li 1 Lucian Popa 3 Kun Qian 4 Anbang Xu
  • 2. Outline of this Tutorial • PART I – Introduction • PART II - Current State of XAI Research for NLP • PART III – Explainability and Case Study • PART IV - Open Challenges & Concluding Remarks 2
  • 3. 3 PART I - Introduction Yunyao Li
  • 4. 4 AI vs. NLP source: https://becominghuman.ai/alternative-nlp-method-9f94165802ed
  • 5. 5 AI is Becoming Increasing Widely Adopted … …
  • 6. 6 AI Life Cycle Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize
  • 8. 8 Why Explainability: Model Development Feature Engineering (e.g. POS, LM) Model Selection/Refine ment Model Training Model Evaluation Error Analysis Error Analysis e.g. “$100 is due at signing” is not classified as <Obligation> Root Cause Identification e.g. <Currency> is not considrerd in the model AI engineers/ data scientists Model Improvement e.g. Include <Currency> as a feature / or introduce entity-aware attention Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Source: ModelLens: An Interactive System to Support the Model Improvement Practices of Data Science Teams. CSCW’2019
  • 9. 9 Why Explainability: Monitor & Optimize Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize User Feedback (implicit/explicit) Feedback Analysis Model Improvement User Feedback e.g. Clauses related to trademark should not classified as <IP> Feedback Analysis e.g. Trademark-related clauses are considered part of <IP> category by The out-of-box model AI engineers Model Improvement e.g. Create a customized model to distinguish trademark-related clauses from other IP-related clauses. Source: Role of AI in Enterprise Applications. https://wp.sigmod.org/?p=2577
  • 12. 12 Needs for Trust in AI à Needs for Explainability What does it take? - Being able to say with certainty how AI reaches decisions - Building mindful of how it is brought into the workplace.
  • 13. Why Explainability: Business Understanding Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize When Explainability is not Needed • No significant consequences for unacceptable results à E.g., ads, search, movie recommendations • Sufficiently well-studied and validated in real applications à we trust the system’s decision, even if it is not perfect E.g. postal code sorting, airborne collision avoidance systems Source: Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang. EMNLP’2017
  • 14. 14 Regulations Credit: Gade, Geyik, Kenthapadi, Mithal, Taly. Explainable AI in Industry, WWW’2020 General Data Protection Regulation (GDPR): Article 22 empowers individuals with the right to demand an explanation of how an automated system made a decision that affects them. Algorithmic Accountability Act 2019: Requires companies to provide an assessment of the risks posed by the automated decision system to the privacy or security and the risks that contribute to inaccurate, unfair, biased, or discriminatory decisions impacting consumers California Consumer Privacy Act: Requires companies to rethink their approach to capturing, storing, and sharing personal data to align with the new requirements by January 1, 2020. Washington Bill 1655: Establishes guidelines for the use of automated decision systems to protect consumers, improve transparency, and create more market predictability. Massachusetts Bill H.2701: Establishes a commission on automated decision-making, transparency, fairness, and individual rights. Illinois House Bill 3415: States predictive data analytics determining creditworthiness or hiring decisions may not include information that correlates with the applicant race or zip code.
  • 16. 16 Why Explainability: Data Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize • Datasets may contain significant bias • Models can further amplify existing bias Source: Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang. EMNLP’2017 man woman Training Data 33% 67% Model Prediction 16% (↓17%) 84% (↑17%) agent role in cooking images
  • 17. 17 Why Explainability: Data Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize • Explicitly capture measures to drive the creation of better, more inclusive algorithms. Source: https://datanutrition.org More: Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Emily M. Bender, Batya Friedman TACL’2018
  • 18. A core element of model risk management (MRM) • Verify models are performing as intended • Ensure models are used as intended 19 Why Explainability: Model Validation Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Source: https://dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/ https://www.mckinsey.com/business-functions/risk/our-insights/the-evolution-of-model-risk-management Defective Model
  • 19. 20 Why Explainability: Model Validation Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize A core element of model risk management (MRM) • Verify models are performing as intended • Ensure models are used as intended Source: https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon- scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G Defective Model
  • 20. 21 Why Explainability: Model Validation Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize A core element of model risk management (MRM) • Verify models are performing as intended • Ensure models are used as intended Source: https://www.theguardian.com/business/2013/sep/19/jp-morgan-920m-fine-london-whale https://financetrainingcourse.com/education/2014/04/london-whale-casestudy-timeline/ https://www.reuters.com/article/us-jpmorgan-iksil/london-whale-took-big-bets-below-the-surface- idUSBRE84A12620120511 Incorrect Use of Model
  • 21. 22 Why Explainability: Model Validation Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize A core element of model risk management (MRM) • Verify models are performing as intended • Ensure models are used as intended Source: https://www.digitaltrends.com/cars/tesla-autopilot-in-hot-seat-again-driver-misuse/ Incorrect Use of Model
  • 22. 23 Why Explainability: Model Deployment Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize • Compliant to regulatory and legal requirements • Foster trust with key stakeholders Source: https://algoritmeregister.amsterdam.nl/en/ai-register/
  • 23. 24 Why Explainability: Model Deployment Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize • Compliant to regulatory and legal requirements • Foster trust with key stakeholders Source: https://algoritmeregister.amsterdam.nl/en/ai-register/
  • 24. 26 Why Explainability: Monitor & Optimize Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize • Carry out regular checks over real data to make sure that the systems are working and used as intended. • Establish KPIs and a quality assurance program to measure the continued effectiveness Source: https://www.huffpost.com/entry/microsoft-tay-racist-tweets_n_56f3e678e4b04c4c37615502
  • 25. 27 Why Explainability: Monitor & Optimize Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize • Carry out regular checks over real data to make sure that the systems are working and used as intended. • Establish KPIs and a quality assurance program to measure the continued effectiveness Source: https://www.technologyreview.com/2020/10/08/1009845/a-gpt-3-bot-posted-comments-on-reddit-for- a-week-and-no-one-noticed/
  • 27. 29 Explainable AI Credit: Gade, Geyik, Kenthapadi, Mithal, Taly. Explainable AI in Industry, WWW’2020
  • 28. Input A test example for which two different classifiers predict the same class A test example which one classifier predicts with high confidence A test example which the classifier predicts with low confidence Information Predicted class & top-m evidences of each model Predicted class & top-m evidences Predicted class & top-m evidences (& possibly top-n counter-evidences) What is the role of an explanation? Introduction Table adapted from [Lertvittayakumjorn and Toni, 2019] Reveal model behavior Justify model predictions Investigate uncertain predictions example task: text classification This tutorial’s focus: Outcome explanation problem
  • 29. Explainability vs. Interpretability 31 Interpretability The degree to which a human can consistently predict the model's result à May not know why Explainability The degree to which a human can understand the cause of the model’s result. à Know why Source: Examples are not enough, learn to criticize! Criticism for interpretability. NIPS’2016 Explaining Explanations: An Overview of Interpretability of Machine Learning. DASS. 2018 explainable à interpretable interpretable !à explainable Explainability Interpretability
  • 30. 32 WHAT MAKES EXPLAINABILITY FOR NLP DIFFERENT?
  • 31. Source: “Why Should I Trust You?” Explaining the Predictions of Any Classifier. KDD’2016 Example: Explanability for Computer Vision
  • 32. Example: Explainability for NLP Source: Outpatient Text Classification Using Attention-Based Bidirectional LSTM for Robot-Assisted Servicing in Hospital. Information 2020
  • 33. 35 Which One Takes You Longer to Understand?
  • 34. 36 Which One Takes You Longer to Understand? Understanding explanation for imagine classification à Perceptual Task: a task we are hardwired to do Source: Why a diagram is (sometimes) worth ten thousand words. Jill H. Larkin Herbert A. Simon. Cognitive Science. 1987 Graphs as Aids to Knowledge Construction: Signaling Techniques for Guiding the Process of Graph Comprehension November 1999. Journal of Educational Psychology Understanding explanation for text classification à Graph Comprehension Task - encode the visual array - identify the important visual features - identify the quantitative facts or relations that those features represent - relate those quantitative relations to the graphic variables depicted
  • 35. 37 Which One Takes You Longer to Understand? Understanding explanation for imagine classification à Perceptual Task: a task we are hardwired to do Source: Why a diagram is (sometimes) worth ten thousand words. Jill H. Larkin Herbert A. Simon. Cognitive Science. 1987 Graphs as Aids to Knowledge Construction: Signaling Techniques for Guiding the Process of Graph Comprehension November 1999. Journal of Educational Psychology Understanding explanation for text classification à Graph Comprehension Task - encode the visual array - identify the important visual features - identify the quantitative facts or relations that those features represent - relate those quantitative relations to the graphic variables depicted
  • 36. 38 Challenges in Explainability for NLP Model Explanation Generation Explanation Interface Text modality à Additional challenges Source: SoS TextVis: An Extended Survey of Surveys on Text Visualization. February 2019. Computers 8(1):17 Generate explanation Display explanation Comprehend explanation
  • 38. 40 Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Business Users Product Managers Data Scientists Domain Experts Business Users Data Providers AI Engineers Data Scientists Product managers AI engineers Data scientists Product managers Model validators Business Users Designers Software Engineers Product Managers AI Engineers Business Users AI Operations Product Managers Model Validators Appropriate Explanations for Different Stakeholders Whether and how explanations should be offered to stakeholders with differing levels of expertise and interests.
  • 39. 41 Are Explanations Helpful? Credit: Dan Weld. Keynote talk “Optimizing Human-AI Teams”. DaSH-LA @NAACL’2021 Source: Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. CHI’2021 AI outperforming human AI performing comparably as human
  • 40. AI Correct AI Incorrect Explanations ARE Convincing Team (R+Human Expl.) Credit: Dan Weld. Keynote talk “Optimizing Human-AI Teams”. DaSH-LA @NAACL’2021 Source: “Does the whole exceed its parts? The effect of AI explanations on complementary team performance. CHI’2021c Regardless of whether AI is right or mistaken ”Better” explanation increases trust but NOT appropriate trust
  • 41. Appropriate Explanations • Need to engender appropriate trust?! • Nature of task may determine affect of explanation on appropriate trust 43 Credit: Dan Weld. Keynote talk “Optimizing Human-AI Teams”. DaSH-LA @NAACL’2021 Source: Does the whole exceed its parts? The effect of AI explanations on complementary team performance. CHI’2021 Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA. ACL’2021 Yet another reason for focusing on explainability for NLP Open-domain QA
  • 42. PART II – Current State of Explainable AI for NLP Marina Danilevsky Lucian Popa Kun Qian
  • 43. Outline – Part II • Literature review methodology • Categorization of different types of explanation – Local vs. Global – Self-explaining vs. Post-hoc processing • Generating and presenting explanations – Explainability techniques • Common operations to enable explainability – Visualization techniques – Communication Paradigm • Other insights (XNLP website) – Relationships among explainability and visualization techniques • Evaluation of Explanations 45
  • 44. Outline of Part II • Literature review methodology • Categorization of different types of explanation – Local vs. Global – Self-explaining vs. Post-hoc processing • Generating and presenting explanations – Explainability techniques • Common operations to enable explainability – Visualization techniques – Communication Paradigm • Other insights (XNLP website) – Relationships among explainability and visualization techniques • Evaluation of Explanations 46
  • 45. Literature Review Methodology: • The purpose is NOT to provide an exhaustive list of papers • Major NLP/Data Science conferences: – KDD, ACL, EMNLP, NAACL, COLING, AAAI, IJCAI, WWW – Years: 2013 – 2021 – > 240 papers screened • What makes a candidate paper: – contain XAI keywords (lemmatized) in their title (e.g., explainable, interpretable, transparent, etc) • Reviewing process – A couple of NLP researchers first go through all the candidate papers to make sure they are truly about XAI for NLP – Every paper is then thoroughly reviewed by at least 2 NLP researchers • Category of the explanation • Explainability & visualization techniques • Evaluation methodology – Consult additional reviewers in the case of disagreement 47
  • 46. Outline of Part II • Literature review methodology • Categorization of different types of explanation – Local vs. Global – Self-explaining vs. Post-hoc processing • Generating and presenting explanations – Explainability techniques • Common operations to enable explainability – Visualization techniques – Communicating Explanations • Other insights (XNLP website) – Relationships among explainability and visualization techniques • Evaluation of Explanations 48
  • 47. How should we differentiate explanations? Multiple different taxonomies exist… [Arya et al., 2019] [Arrietaa et al, 2020] 49 Madsen et al., 2021
  • 48. How should we differentiate explanations? Two fundamental aspects apply to any XAI problem Is the explanation for an individual instance or for an AI model ? Is the explanation obtained directly from the prediction or requiring post-processing ? Local explanation: individual instance Global explanation: internal mechanism of a model Self-explaining: directly interpretable Post-hoc: a second step is needed to get explanation 50
  • 49. Local Explanation • We understand only the reasons for a specific decision made by an AI model • Only the single prediction/decision is explained (Guidotti et al. 2018) Machine translation Input-output alignment matrix changes based on instances [NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE. Bahdanau et al., ICRL 2015] 51
  • 50. Global explanation • We understand the whole logic of a model • We can follow the entire reasoning leading to all different possible outcomes Img source: https://medium.com/@fenjiro/data-mining-for-banking-loan-approval-use-case-e7c2bc3ece3 Decision tree built for credit card application decision making 52
  • 51. Self-explaining vs. Post-hoc • Self-explaining: we can directly get the explanation with the prediction • Post-hoc: explanation does not come directly with the prediction Only one model Model 1 Model 2 53
  • 52. Task - how to differentiate different explanations Is the explanation for an individual instance or for an AI model ? Is the explanation obtained directly from the prediction or requiring post-processing ? Local explanation: individual instance Global explanation: internal mechanism of a model Self-explaining: directly interpretable Post-hoc: a second step is needed to get explanation orthogonal aspects 54 Two fundamental aspects apply to any XAI problem
  • 54. Outline of Part II • Literature review methodology • Categorization of different types of explanation – Local vs. Global – Self-explaining vs. Post-hoc processing • Generating and presenting explanations – Explainability techniques • Common operations to enable explainability – Visualization techniques – Communication Paradigm • Other insights (XNLP website) – Relationships among explainability and visualization techniques • Evaluation of Explanations 56
  • 55. From model prediction to user understanding XAI model Explained predictions End user Are explanations directly sent from XAI models to users? 57
  • 56. Two Aspects: Generation & Presentation XAI model Mathematical justifications End user Visualizing justifications 1. Explanation Generation 2. Explanation Presentation UX engineers AI engineers 58
  • 57. What’s next XAI model Mathematical justifications End user Visualizing justifications 1. Explanation Generation 2. Explanation Presentation UX engineers AI engineers Explainability techniques 59
  • 58. What is Explainability? • The techniques used to generate raw explanations – Ideally, created by AI engineers • Representative techniques: – Feature importance – Surrogate model – Example-driven – Provenance-based – Declarative induction • Common operations to enable explainability – Attention – First-derivative saliency – Layer-wise Relevance Propagation – Integrated Gradients – Explainability-aware architecture design 60
  • 59. Explainability - Feature Importance The main idea of Feature Importance is to derive explanation by investigating the importance scores of different features used to output the final prediction. Can be built on different types of features Manual features from feature engineering [Voskarideset al., 2015] Lexical features including words/tokens and N-gram [Godin et al., 2018] [Mullenbachet al., 2018] Latent features learned by neural nets [Xie et al., 2017] [Bahdanau et al., 2015] [Li et al., 2015] Definition Usually used in parallel [Ghaeini et al., 2018] 61 [Ramnath et al., 2020]
  • 60. Feature Importance – Machine Translation [Bahdanau et al., 2015] [NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE. Bahdanau et al., ICRL 2015] A well-known example of feature importance is the Attention Mechanism first used in machine translation Image courtesy: https://smerity.com/articles/2016/google_nmt_arch.html Traditional encoder-decoder architecture for machine translation Only the last hidden state from encoder is used 62
  • 61. [NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE. Bahdanau et al., ICRL 2015] Image courtesy: https://smerity.com/articles/2016/google_nmt_arch.html Attention mechanism uses the latent features learned by the encoder • a weighted combination of the latent features Weights become the importance scores of these latent features (also corresponds to the lexical features) for the decoded word/token. Similar approach in [Mullenbachet al., 2018] Local & Self-explaining Input-output alignment 63 Feature Importance – Machine Translation [Bahdanau et al., 2015]
  • 62. Feature Importance – token-level explanations [Thorne et al] Another example that uses attention to enable feature importance NLP Task: Natural Language Inference (determining whether a “hypothesis” is true (entailment), false (contradiction), or underdetermined (neutral) given a “premise” Premise: Children Smiling and Waving at a camera Hypothesis: The kids are frowning GloVe LSTM ℎ! ℎ" Attention c h i l d r e n S m i l i n g a n d W a v i n g a t a c a m e r a The kid are frowning MULTIPLE INSTANCE LEARNING to get appropriate thresholds for attention matrix 64
  • 63. 65 Interpreting BERT for Reading Comprehension Based QA • RCQA: given a document and a question, find the answer in the document. Towards Interpreting BERT for Reading Comprehension Based QA. Ramnath et. al. EMNLP 2020 What are the roles of each BERT layer plays?
  • 64. 66 Interpreting BERT for Reading Comprehension Based QA • Use Integrated Gradients (IG) to compute the importance of different features in different BERT layers – IG is one of the most popular techniques to interpret DL models – Will be discussed in later operation section in details. • Observations: – Initial layers: query-passage interaction – Later layers: contextual understanding and enhancing the answer prediction. Towards Interpreting BERT for Reading Comprehension Based QA. Ramnath et. al. EMNLP 2020
  • 65. Explainability – Surrogate Model Model predictions are explained by learning a second, usually more explainable model, as a proxy. Definition A black-box model or a hard-to-explain model Input data Predictions without explanations or hard to explain. An interpretable model that approximates the black-box model Interpretable predictions [Liu et al. 2018] [Ribeiro et al. KDD] [Alvarez-melis and Jaakkola] [Sydorova et al, 2019] 67
  • 66. Surrogate Model – LIME [Ribeiro et al. KDD] Blue/Pink area are the decision boundary of a complex model 𝒇, which is a black-box model Cannot be well approximated by a linear function Instance to be explained Sampled instances, weighted by the proximity to the target instance An Interpretable classifier with local fidelity Local & Post-hoc LIME: Local Interpretable Model-agnostic Explanations 68
  • 67. Surrogate Model – LIME [Ribeiro et al. KDD] Local & Post-hoc High flexibility – free to choose any interpretable surrogate models 69
  • 68. Surrogate Model - Explaining Network Embedding [Liu et al. 2018] Global & Post-hoc A learned network embedding Induce a taxonomy from the network embedding 70
  • 69. Surrogate Model - Explaining Network Embedding [Liu et al. 2018] Induced from 20NewsGroups Network Bold text are attributes of the cluster, followed by keywords from documents belong to the cluster. 71
  • 70. Explainability – Example-driven • Similar to the idea of nearest neighbors [Dudani, 1976] • Have been applied to solve problems including – Text classification [Croce et al., 2019] – Question Answering [Abujabal et al., 2017] Such approaches explain the prediction of an input instance by identifying and presenting other instances, usually from available labeled data, that are semantically similar to the input instance. Definition 72
  • 71. Explainability – Text Classification [Croce et al., 2019] “What is the capital of Germany?” refers to a Location. WHY? “What is the capital of California?” which also refers to a Location in the training data Because 73
  • 72. Explainability – Text Classification [Croce et al., 2019] Layer-wise Relevance Propagation in Kernel-based Deep Architecture 74
  • 73. Explainability – Text Classification [Croce et al., 2019] Need to understand the work Thoroughly 75
  • 74. Explainability – Provenance-based • An intuitive and effective explainability technique when the final prediction is the result of a series of reasoning steps. • Several question answering papers adopt this approach. – [Abujabal et al., 2017] – Quint system for KBQA (knowledge-base question answering) – [Zhou et al., 2018] – Multi-relation KBQA – [Amini et al., 2019] – MathQA, will be discussed in the declarative induction part (as a representative example of program generation) Explanations are provided by illustrating (some of) the prediction derivation process Definition 76
  • 75. Provenance-Based: Quint System A. Abujabal et al. QUINT: Interpretable Question Answering over Knowledge Bases. EMNLP System Demonstration, 2017. • When QUINT answers a question, it shows the complete derivation sequence: how it understood the question: o Entity linking: entity mentions in text ⟼ actual KB entities o Relation linking: relations in text ⟼ KB predicates the SPARQL query used to retrieve the answer: “Eisleben” “Where was Martin Luther raised?” place_of_birth Martin_Luther “ … a German professor of theology, composer, priest, monk and …” desc Martin_Luther_King_Jr. desc Learned from query templates based on structurally similar instances (“Where was Obama educated?”) Eisleben Derivation provides insights towards reformulating the question (in case of errors) 77
  • 76. Provenance-Based: Multi-Relation Question Answering M. Zhou et al. An Interpretable Reasoning Network for Multi- Relation Question Answering. COLING, 2018. IRN (Interpretable Reasoning Network): • Breaks down the KBQA problem into multi-hop reasoning steps. • Intermediate entities and relations are predicted in each step: Reasoning path: John_Hays_Hammond ⟶ e ⟶ a Child Profession Facilitates correction by human (e.g., disambiguation) • multiple answers in this example (3 different children) 78
  • 77. Explainability – Declarative Induction The main idea of Declarative Induction is to construct human-readable representations such as trees, programs, and rules, which can provide local or global explanations. Explanations are produced in a declarative specification language, as humans often would do. Trees [Jiang et al., ACL 2019] - express alternative ways of reasoning in multi-hop reading comprehension [Wu et al., ACL 2020] – Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT Rules: [A. Ebaid et al., ICDE Demo, 2019] - Bayesian rule lists to explain entity matching decisions [Pezeshkpour et al, NAACL 2019] - rules for interpretability of link prediction [Sen et al, EMNLP 2020] - linguistic expressions as sentence classification model [Qian et al, CIKM 2017] - rules as entity matching model Definition Examples of global explainability (although rules can also be used for local explanations) (Specialized) Programs: [A. Amini et al., NAACL, 2019] – MathQA, math programs for explaining math word problems Examples of local explainability (per instance) 79 79
  • 78. Declarative Induction: Reasoning Trees for Multi-Hop Reading Comprehension Task: Explore and connect relevant information from multiple sentences/documents to answer a question. Y. Jiang et al. Explore, Propose, and Assemble: An Interpretable Model for Multi-Hop Reading Comprehension. ACL, 2019. Question subject: “Sulphur Spring” Question body: located in administrative territorial entity Reasoning trees play dual role: • Accumulate the information needed to produce the answer • Each root-to-leaf path represents a possible answer and its derivation • Final answer is aggregated across paths • Explain the answer via the tree 80
  • 79. Declarative Induction: (Specialized) Program Generation Task: Understanding and solving math word problems A. Amini et al. MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. NAACL-HLT, 2019. • Seq-2-Program translation produces a human- interpretable representation, for each math problem • Does not yet handle problems that need complicated or long chains of mathematical reasoning, or more powerful languages (logic, factorization, etc). • Also fits under provenance-based explainability: • provides answer, and how it was derived. Domain-specific program based on math operations • Representative example of a system that provides: • Local (“input/output”) explainability • But no global explanations or insights into how the model operates (still “black-box”) 81
  • 80. Declarative Induction: Post-Hoc Explanation for Entity Resolution Black-box model ExplainER • Local explanations (e.g., how different features contribute to a single match prediction – based on LIME) • Global explanations Approximate the model with an interpretable representation (Bayesian Rule List -- BRL) • Representative examples Small, diverse set of <input, output> tuples to illustrate where the model is correct or wrong • Differential analysis Compare with other ER model (disagreements, compare performance) A. Ebaid et al. ExplainER: Entity Resolution Explanations. ICDE Demo, 2019. M. T. Ribeiro et al. Why should I trust you: Explaining predictions of any classifier. In SIGKDD, pp. 1135—1144, 2016. B. Letham et al. Interpretable classifiers using rules and Bayesian analysis. Annals of Applied Stat., 9(3), pp. 1350-1371, 2015. Can also be seen as a tree (of probabilistic IF-THEN rules) Entity resolution (aka ER, entity disambiguation, record linking or matching) 82
  • 81. Declarative Induction: Rules for Link Prediction Task: Link prediction in a knowledge graph, with focus on robustness and explainability P. Pezeshkpour et al. Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications. NAACL 2019. Uses adversarial modifications (e.g., removing facts): 1. Identify the KG nodes/facts most likely to influence a link 2. Aggregate the individual explanations into an extracted set of rules operating at the level of entire KG: • Each rule represents a frequent pattern for predicting the target link Works on top of existing, black-box models for link prediction: DistMult: B. Yang et al. Embedding entities and relations for learning and inference in knowledge bases. ICLR 2015 ConvE: T. Dettmers et al. Convolutional 2d knowledge graph embeddings. AAAI 2018 Global explainability able to pinpoint common mistakes in the underlying model (wrong inference in ConvE) 83
  • 82. 84 Declarative Induction – Interpreting BERT with Perturbed Masking Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT. Wu et. al,. ACL 2020 • BERT is acclaimed for learning useful representations for different NLP tasks • Probing classifiers are used for measure the quality of representations learned by BERT – a small NN that uses BERT embeddings as input for a specific task – Problem: additional NN with its own parameters. • Can we draw insights directly from BERT’s parameters, that is, in a probe-free fashion? • This paper introduces a parameter-free probing technique called Perturbed Masking.
  • 83. 85 Declarative Induction – Interpreting BERT with Perturbed Masking Perturb input sentence and extract an impact matrix Use graph-based dependency parsing algorithm to extract dependency trees out of impact matrix Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT. Wu et. al,. ACL 2020
  • 84. • Previous approach for rule induction: – The result is a set of logical rules (Horn rules of length 2) that are built on existing predicates from the KG (e.g., Yago). • Other approaches also learn logical rules, but they are based on different vocabularies: – Next: o Logical rules built on linguistic predicates for Sentence Classification – [SystemER – see later] o Logical rules built on similarity features for Entity Resolution 86
  • 85. Declarative Induction: Linguistic Rules for Sentence Classification Notices will be transmitted electronically, by registered or certified mail, or courier Syntactic and Semantic NLP Operators (Explainable) Deep Learning ∃ verb ∈ 𝑆: verb.theme ∈ ThemeDict and verb.voice = passive and verb.tense = future ThemeDict communication notice Dictionaries Rules (predicates based on syntactic/semantic parsing) “Communication” Sen et al, Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification. EMNLP 2020. • Representative example of system that uses a neural model to learn another model (the rules) • The rules are used for classification but also act as global explanation • Joint learning of both rules and domain-specific dictionaries • Explainable rules generalize better and are more conducive for interaction with the expert (see later – Visualization) Label … “Terms and Termination” … Task: Sentence classification in a domain (e.g., legal/contracts domain) 87
  • 86. Common Operations to enable explainability • A few commonly used operations that enable different explainability – Attention – First-derivative saliency – Layer-wise relevance propagation – Integrated Gradients – Input perturbations – Explainability-aware architecture design 88
  • 87. Operations – Attention [NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE. Bahdanau et al., ICRL 2015] Image courtesy: https://smerity.com/articles/2016/google_nmt_arch.html Attention mechanism uses the latent features learned by the encoder • a weighted combination of the latent features Weights become the importance scores of these latent features (also corresponds to the lexical features) for the decoded word/token. Naturally, can be used to enable feature importance [Ghaeini et al., 2018] 89
  • 88. 90 Operations – Attention – Self-attention [images from Stanford CS224] Transformer-based models such as BERT use a multi-head self-attention mechanism Attention weights are still used as importance scores
  • 89. 91 Operations – Attention – Self-attention • Attention score (left) and attribution score (right) of a single head in BERT. • The color is darker for larger values. • Model prediction for the sentence is contradiction. • Attribution tends to identify more sparse word interactions that contribute to the final model decision [Hao et al., 2021]
  • 90. Operations – First-derivative saliency • Mainly used for enabling Feature Importance – [Li et al., 2016 NAACL], [Aubakirova and Bansal, 2016] • Inspired by the NN visualization method proposed for computer vision – Originally proposed in (Simonyan et al. 2014) – How much each input unit contributes to the final decision of the classifier. Input embedding layer Output of neural network Gradients of the output wrt the input 92
  • 91. Operations – First-derivative saliency Sentiment Analysis Naturally, it can be used to generate explanations presented by saliency-based visualization techniques [Li et al., 2016 NAACL] 93
  • 92. Operations – Layer-wise relevance propagation • For a good overview, refer to • An operation that is specific to neural networks – Input can be images, videos, or text • Main idea is very similar to first-derivative saliency – Decomposing the prediction of a deep neural network computed over a sample, down to relevance scores for the single input dimensions of the sample. [Montavon et al., 2019] [Montavon et al., 2019] 94
  • 93. Operations – Layer-wise relevance propagation • Key difference to first-derivative saliency: – Assigns to each dimension (or feature), 𝑥#, a relevance score 𝑅# (%) such that • Propagate the relevance scores using purposely designed local propagation rules – Basic Rule (LRP-0) – Epsilon Rule (LRP-ϵ) – Gamma Rule (LRP-γ) 𝑓 𝑥 ≈ ( # 𝑅# (%) [Croce et al] 𝑅# (%) > 0 – the feature in favor the prediction 𝑅# (%) < 0 – the feature against the prediction [Montavon et al., 2019] 95
  • 94. 96 Integrated Gradients (IG) • An attribution-based approach that enables Feature Importance over DL models. – Hands-on tutorial on IG using TensorFlow: https://www.tensorflow.org/tutorials/interpretability/inte grated_gradients • Introduced two axioms that every attribution method should satisfy – Sensitivity: for every input and baseline that differ in one feature but have different predictions then the differing feature should be given a non-zero attribution – Implementation invariance: . Two networks are functionally equivalent if their outputs are equal for all inputs, despite having very different implementations. Axiomatic Attribution for Deep Networks. Sundararajan, Taly, Yan. ICML 2017 Image source: https://www.tensorflow.org/tutorials/interpretability/integrated_gradients
  • 95. 97 Integrated Gradients (IG) • Start with a baseline input (e.g., empty input text or all zero embeddings) • Iteratively “move” the baseline input to the final input in the high-dimensional space – cumulate the gradients along the (straight) path from baseline to the final input – The integrated gradient along the i-th dimension for an input x and baseline x’ is defined as • Nice properties of IG: – Satisfy the two aforementioned properties – No additional parameters needed – No training required – Simple to implement Axiomatic Attribution for Deep Networks. Sundararajan, Taly, Yan. ICML 2017 Integrated Gradients is included in Captum: an XAI package for pytorch approximation Image source: https://www.tensorflow.org/tutorials/interpretability/integrated_gradients
  • 96. Operations – Input perturbation • Usually used to enable local explanation for a particular instance – Often combined with surrogate models 𝑋 𝑓 𝑓(𝑋) An input instance A classifier Usually not interpretable Prediction made by the classifier 𝑋! , 𝑋" , … 𝑋# 𝑓(𝑋! ), 𝑓(𝑋" ), … 𝑓(𝑋# ) Perturbed instances Prediction made by the original classifier 𝒈 A surrogate classifier usually interpretable 𝒈(𝑋) Explanation to humans 98
  • 97. Operations – Input perturbation • Different ways to generate perturbations – Sampling method (e.g., [Ribeiro et al. KDD] - LIME ) – Perturbating intermediate vector representation (e.g., [Alvarez-Melis and Jaakkola, 2017]) • Sampling method – Directly use examples within labeled data that are semantically similar • Perturbating intermediate vector representation – Specific to neural networks – Introduce minor perturbations to the vector representation of input instance 99
  • 98. Input perturbation – Sampling method Find examples that are both semantically “close” and “far away” from the instance in question. - Need to define a proximity metric - Will not generate instance [Ribeiro et al. KDD] 100
  • 99. Perturbating intermediate Vector Representation [Alvarez-Melis and Jaakkola, 2017]) • Explain predictions of any black-box structured input – structured output model • Around a specific input-output pair • An explanation consists of groups of input-output tokens that are causally related. • The dependencies (input-output) are inferred by querying the blackbox model with perturbed inputs 101
  • 100. Perturbating intermediate Vector Representation [Alvarez-Melis and Jaakkola, 2017]) 102
  • 101. Explainability-Aware: Rule Induction Architecture Works on top of existing, black-box models for link prediction, via a decoder architecture (i.e., from embeddings back to graph) P. Pezeshkpour et al. Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications. NAACL 2019. Knowledge graph facts: wasBornIn (s, o) Optimization problem in Rd: find the most influential vector zs’r’ whose removal would lead to removal of wasBornIn (s, o) Decode back the vector zs’r’ to actual KG: isLocatedIn (s’, o) Rules of length 2 are then extracted by identifying frequent patterns isAffiliatedTo (a, c) ∧ isLocatedIn (c, b) → wasBornIn (a,b) 103
  • 102. Part II – (b) Visualization Techniques 104
  • 103. What’s next XAI model Mathematical justifications End user Visualizing justifications 1. Explanation Generation 2. Explanation Presentation UX engineers AI engineers Visualization Techniques 105
  • 104. Visualization Techniques • Presenting raw explanations (mathematical justifications) to target user • Ideally, done by UX/UI engineers • Main visualization techniques: – Saliency – Raw declarative representations – Natural language explanation – Raw examples 106
  • 105. Visualization – Saliency has been primarily used to visualize the importance scores of different types of elements in XAI learning systems Definition • One of the most widely used visualization techniques • Not just in NLP, but also highly used in computer vision • Saliency is popular because it presents visually perceptive explanations • Can be easily understood by different types of users A computer vision example 107
  • 106. Visualization – Saliency Example Input-output alignment [Bahdanau et al., 2015] 108
  • 107. Visualization – Saliency Example [Schwarzenberg et al., 2019] [Wallace et al., 2018] [Harbecke et al, 2018] 109 [Antognini et al, 2021]
  • 108. Visualization – Raw Examples • Use existing knowledge to explain a new instance Explaining by presenting one or more instances (usually from the training data, sometimes generated) that are semantically similar to the instance that needs to be explained Definition Jaguar is a large spotted predator of tropical America similar to the leopard. [Panchenko et al., 2016] Word sense disambiguation Raw examples 110
  • 109. Visualization – Raw Examples [Croce et al, 2018] Classification Sentences taken from the labeled data Explain with one instance (basic model) Explain with > 1 instance (multiplicative model) Explain with both positive and negative instance (contrastive model) 111
  • 110. Visualization – Natural language explanation The explanation is verbalized in human-comprehensible language Definition • Can be generated by using sophisticated deep learning approaches – [Rajani et al., 2019] • Can be generated by simple template-based approaches – [Croce et al, 2018] – raw examples and natural language explanations – Many declarative representations can be extended to natural language explanation 112
  • 111. Visualization – Natural language explanation [Rajani et al., 2019] A language model trained with human generated explanations for commonsense reasoning Open-ended human explanations Explanation generated by CAGE 113
  • 112. Visualization – Natural language explanation [Croce et al, 2018] Classification Sentences taken from the labeled data Explain with one instance (basic model) Explain with > 1 instance (multiplicative model) Explain with both positive and negative instance (contrastive model) 114
  • 113. Visualization – Natural language explanation [Croce et al, 2018] Classification Template-based explanatory model 115
  • 114. Visualization – Declarative Representations • Trees: [Jiang et al., ACL 2019] – Different paths in the tree explain the possible reasoning paths that lead to the answer • Rules: [Pezeshkpour et al, NAACL 2019] – Logic rules to explain link prediction (e.g., isConnected is often predicted because of transitivity rule, while hasChild is often predicted because of spouse rule) [Sen et al, EMNLP 2020] – Linguistic rules for sentence classification: • More than just visualization/explanation • Advanced HCI system for human-machine co- creation of the final model, built on top of rules (Next) isConnected(a,c) ∧ isConnected (c,b) ⟹ isConnected(a,b) isMarriedTo (a,c) ∧ hasChild (c,b) ⟹ hasChild(a,b) ∃ verb ∈ 𝑆: verb.theme ∈ ThemeDict and verb.voice = passive and verb.tense = future Machine-generated rules Human feedback 116
  • 115. Linguistic Rules: Visualization and Customization by the Expert Passive-Voice Possibility-ModalClass Passive-Voice VerbBase-MatchesDictionary-SectionTitleClues VerbBase-MatchesDictionary-SectionTitleClues Theme-MatchesDictionary-NonAgentSectionTitleClues Theme-MatchesDictionary-NonAgentSectionTitleClues Manner-MatchesDictionary-MannerContextClues Manner-MatchesDictionary-MannerContextClues VerbBase-MatchesDictionary-SectionTitleClues NounPhrase-MatchesDictionary-SectionTitleClues NounPhrase-MatchesDictionary-SectionTitleClues Label being assigned Various ways of selecting/ranking rules Rule-specific performance metrics Y. Yang, E. Kandogan, Y. Li, W. S. Lasecki, P. Sen. HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop. ACL, System Demonstration, 2019. Sentence classification rules learned by [Sen et al, Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification. EMNLP 2020] 117
  • 116. Visualization of examples per rule Passive-Voice Present-Tense Negative-Polarity Possibility-ModalClass Imperative-Mood Imperative-Mood VerbBase-MatchesDictionary-SectionTitleClues Manner-MatchesDictionary-MannerContextClues Manner-MatchesDictionary-MannerContextClues VerbBase-MatchesDictionary-VerbBases VerbBase-MatchesDictionary-VerbBases Linguistic Rules: Visualization and Customization by the Expert Y. Yang, E. Kandogan, Y. Li, W. S. Lasecki, P. Sen. HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop. ACL, System Demonstration, 2019. 118
  • 117. Playground mode allows to add/drop predicates Future-Tense Passive-Voice Future-Tense Passive-Voice NounPhrase-MatchesDictionary-SectionTitleClues NounPhrase-MatchesDictionary-SectionTitleClues Linguistic Rules: Visualization and Customization by the Expert Human-machine co-created models generalize better to unseen data • humans instill their expertise by extrapolating from what has been learned by automated algorithms Y. Yang, E. Kandogan, Y. Li, W. S. Lasecki, P. Sen. HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop. ACL, System Demonstration, 2019. 119
  • 118. 120 Paradigms for Communicating Explanations • Static vs. interactive – In an interactive system, explanations can naturally be incorporated as part of the human-AI feedback loop • Examples of systems with interactive explanations – HEIDL (seen before): human-machine co-creation of linguistic models, with explanations based on rules – SystemER – interactive entity resolution, also with explanations based on rules – [Chen et al, IJCAI 2020] • Multi-turn explainable conversation, leads to better recommendations – [Gao et al, IntelliSys 2021] • Proactively explains the AI system to the user ∃ verb ∈ 𝑆: verb.theme ∈ ThemeDict and verb.voice = passive and verb.tense = future Machine-generated rules Human feedback HEIDL
  • 119. SystemER: Interactive Learning of Explainable Models corporation Zip City address Trader Joe’s 95118 San Jose 5353 Almaden Express Way company Zip City address Trader Joe’s 95110 San Jose 635 Coleman Ave Rule Learning Algorithm R.corporation = S.company and R.city = S.city Candidate rule learning R.corporation = S.company and R.city = S.city and R.zip = S.zip corporation Zip City address KFC 90036 LA 5925 W 3rd St. Not a match refinement company Zip City address KFC 90004 LA 340 N Western Ave Qian et al, SystemER: A human-in-the-loop System for Explainable Entity Resolution. VLDB’17 Demo (also full paper in CIKM’17) Iteratively learns an ER model via active learning from few labels. • Rules: explainable representation for intermediate states and final model • Rules: used as mechanism to generate and explain the examples brought back for labeling Large datasets to match Example selection Domain expert labeling 121
  • 120. U: D1 x D2 Matches (R1) + + + + + + + + + + + + + + + + + + + + + - - - - - SystemER: Example Selection (Likely False Positives) To refine R1 so that it becomes high-precision, need to find examples from the matches of R1 that are likely to be false positives Negative examples will enhance precision Candidate rule R1 122
  • 121. U: D1 x D2 Matches (R1) To refine R1 so that it becomes high-precision, need to find examples from the matches of R1 that are likely to be false positives + + + + + + + + + + + + + + + + + + + + + Negative examples will enhance precision - - - - - SystemER: Example Selection (Likely False Negatives) + + Rule-Minus1 + + Rule-Minus2 + Rule-Minus3 Rule-Minus heuristic: explore beyond the current R1 to find what it misses (likely false negatives) R1: i.firstName = t.first AND i.city = t.city AND i.lastName = t.last i.firstName = t.first AND i.city = t.city AND i.lastName = t.last Rule-Minus1 i.firstName = t.first AND i.city = t.city AND i.lastName = t.last Rule-Minus3 i.firstName = t.first AND i.city = t.city AND i.lastName = t.last Rule-Minus2 Candidate rule R1 New positive examples will enhance recall At each step, examples are explained/visualized based on the predicates they satisfy or not satisfy 123
  • 122. 124 Explainable Conversational Recommendation • Multi-turn user vs. model conversation, based on (recommendation + explanation). • Enables continuous improvement of both recommendation and explanation quality. • Suitable for exploration, where users are wandering for interesting items. Chen et al, Towards Explainable Conversational Recommendation. IJCAI 2020. Explanations are the foundation for user feedback: • Users inform the system whether they like, or dislike features mentioned in the explanations • The features in the explanations may also trigger users to suggest related features they like. (-) comedy (-) anime (+) action movie
  • 123. 125 Chat-XAI Design Principles Gao et al, Chat-XAI: A New Chatbot to Explain Artificial Intelligence, IntelliSys 2021 • Key premise: many users do not understand what the AI system can do, and often assume the AI is perfect. • Proactive interactions → better communication between AI and human.
  • 124. Outline of Part II • Literature review methodology • Categorization of different types of explanation – Local vs. Global – Self-explaining vs. Post-hoc processing • Generating and presenting explanations – Explainability techniques • Common operations to enable explainability – Visualization techniques – Communication Paradigm • Other insights (XNLP website) – Relationships among explainability and visualization techniques • Evaluation of Explanations 126
  • 125. XNLP • We built an interactive website for exploring the domain • It’s self-contained • It provides different ways for exploring the domain – Cluster View (group papers based on explainability and visualization) – Tree view (categorize papers in a tree like structure) – Citation graphs (show the evolution of the field and also show influential works) – List view (list the set of papers in a table) – Search view (support keyword search and facet search) • IUI’2021 demo 127
  • 126. XNLP - various ways to explore the literature 128 Link to the website: https://xainlp2020.github.io/xainlp/home
  • 127. Outline of Part II • Literature review methodology • Categorization of different types of explanation – Local vs. Global – Self-explaining vs. Post-hoc processing • Generating and presenting explanations – Explainability techniques • Common operations to enable explainability – Visualization techniques – Communication Paradigm • Other insights (XNLP website) – Relationships among explainability and visualization techniques • Evaluation of Explanations 129
  • 128. Evaluation Techniques Informal Examination Explaining How Well Human Evaluation Comparison to Ground Truth 60% 25% 15% [Xie et al., 2017] [Ling et al., 2017] [Sydorova et al., 2019]
  • 129. Evaluation – Comparison to ground truth • Idea: compare generated explanations to ground truth explanations • Metrics used – Precision/Recall/F1 (Carton et al.,2018) – BLEU (Ling et al., 2017; Rajani et al., 2019b) • Benefit – A quantitative way to measure explainability • Pitfalls – Quality of the ground truth data – Alternative valid explanations may exit • Some attempts to avoid these issues – Having multiple annotators (with inter-annotator agreement) – Evaluating at different granularities (e.g., token-wise vs. phrase-wise) 131
  • 130. Evaluation – Human Evaluation • Idea: ask humans to evaluate the effectiveness of the generated explanations. • Benefit – Avoiding the assumption that there is only one good explanation that could serve as ground truth • Important aspects – It is important to have multiple annotators (with inter-annotator agreement, avoid subjectivity) • Observations from our literature review – Single-human evaluation (Mullenbach et al., 2018) – Multiple-human evaluation (Sydorova et al., 2019) – Rating the explanations of a single approach (Dong et al., 2019) – Comparing explanations of multiple techniques (Sydorova et al., 2019) • Very few well-established human evaluation strategies. 132
  • 131. Human Evaluation – Trust Evaluation (Sydorova et al., 2019) • Borrowed the trust evaluation idea used in CV (Selvaraju et al. 2016) • A comparison approach – given two models, find out which one produces more intuitive explanations Model A Model B examples Examples that the two models predict the same labels Explanation 1 Explanation 2 Blind evaluation 133
  • 132. Human Evaluation – Trust Evaluation (Sydorova et al., 2019) 134
  • 133. Quality of Explanation – Predictive Process Coverage • The predictive process starts from input to final prediction • Not all XAI approaches cover the whole process • Ideally, the explanation covers the whole process or most of the process • However, many of the approaches cover only a small part of the process • The end user has to fill up the gaps 135
  • 134. Quality of Explanation – Predictive Process Coverage • Take MathQA (Amini et al. 2019) as an example Induced mathematical operations 136
  • 135. Quality of Explanation – Predictive Process Coverage • Take MathQA (Amini et al. 2019) as an example 137
  • 136. Fidelity and faithfulness • Often an issue arises in surrogate model • Surrogate model is flexible since its model-agnostic • Local fidelity vs. global fidelity • Logic fidelity • Surrogate model may use completely different reasoning mechanism • Neglected in the past, now an emerging topic • General guidelines for evaluating faithfulness and fidelity • “Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?” Jacovi and Goldberg. ACL 2020 • Use fidelity to measure the goodness of an explanation approach 138 Evaluating Explanation Methods for Neural Machine Translation. Li et. al, ACL 2020 [Ribeiro et al. KDD]
  • 137. 139 Faithfulness/Fidelity • [Jacovi and Goldberg, 2020] provides a in-depth discussion about faithfulness • Guidelines for evaluating faithfulness: – Be explicit in what you evaluate (Plausibility vs. faithfulness) – Should not involve human-judgement on the quality of the interpretation. – Should not involve human-provided gold labels – Do not trust “inherent interpretability” claims – Evaluation of IUI systems should not rely on user performance. • This work aims to pave the future avenue for faithfulness evaluation research. “Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?” Jacovi and Goldberg. ACL 2020
  • 138. Evaluate fidelity in Neural Machine Translation Evaluating Explanation Methods for Neural Machine Translation. Li et. al, ACL 2020 Image source: https://developer.nvidia.com/blog/introduction-neural-machine-translation-gpus-part-3/ For a target word predicted, the explanation is usually given by extracting the most relevant words. Different methods may highlight/extract different relevant words, how to measure the quality of the extracted/highlighted words?
  • 139. Fidelity-based evaluation metric • Construct proxy models for extracted words produced by different explainable approaches • Compare these proxy models with the original NMT model • The explanation method that leads to the the proxy model that agrees with the original NMT model the best wins NMT model Explanation Approach 1 Explanation Approach 2 Word set 1 Word set 2 Proxy model selection Proxy model 1 Proxy model 2 NMT model NMT model vs. vs. The one with lower disagreement rate wins Explanation operations tested • Attention • Gradient-based (first-derivative saliency) • Prediction Difference Proxy model considered • MLP • RNN • Self-attention
  • 140. PART III – Explainability & Case Study Anbang Xu
  • 141. Evaluating explanations User-dependent measures2 Comprehensability, User satisfaction Task Oriented Measures2 Task performance Impact on AI interaction - Trust (calibration) in model Task or AI system satisfaction Inherent “goodness” measure2 Fidelity/faithfulness, Stability, Compactness Explainable AI in Industry – KDD ‘19 Comprehensability1 Actionabilty1 Reusability1 Accuracy1 Completeness1 Succintness1 Introduction to Explainable AI – CHI 2021 1. Gade, K., Geyik, S. C., Kenthapadi, K., Mithal, V., & Taly, A. (2019, July). Explainable AI in industry. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 3203-3204). 2. Liao, Q. V., Singh, M., Zhang, Y., & Bellamy, R. (2021, May). Introduction to explainable ai. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-3). 3. Schmidt, P., & Biessmann, F. (2019). Quantifying interpretability and trust in machine learning systems. arXiv preprint arXiv:1901.08558. Trust metric based on the quality of interpretability methods3 1 2 3
  • 142. Human evaluation of explainability techniques 1 . Zhu, Y., Xian, Y., Fu, Z., de Melo, G., & Zhang, Y. (2021, June). Faithfully Explainable Recommendation via Neural Logic Reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 3083-3090). 2 Doogan, C., & Buntine, W. (2021, June). Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 3824-3848). 3. Wu, H., Chen, W., Xu, S., & Xu, B. (2021, June). Counterfactual Supporting Facts Extraction for Explainable Medical Record Based Diagnosis with Graph Network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 1942-1955). Prior evaluation with a limited number of users: • KG-enhanced explanation paths that connect user to recommendation1 faithfulness • Evaluating topic interpretability2 • Text extraction to support medical diagnosis3 comprehensiveness, trustworthiness
  • 144. 14 6 The role of explanations - Improve trust Decision making with machine assistance e.g. deception detection1 Aim for human-AI complementary performance2 – Does the whole exceed sum of its part? – Slow down to reduce anchor effects – Foster appropriate reliance Full human agency Only explanations Predicted labels with accuracy Predicted labels and explanation Full automation Spectrum of machine assistance 1. Lai, V., & Tan, C. (2019, January). On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency (pp. 29-38). 2 .Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., ... & Weld, D. (2021, May). Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-16).
  • 145. 14 7 Human-centered perspective on XAI • Explanations in social sciences1 Contrastive, Selected, Social. • Leveraging human reasoning processes to craft explanations2 Map explainability techniques with cognitive patterns. 1. Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1-38. 2. Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019, May). Designing theory-driven user-centric explainable AI. In Proceedings of the 2019 CHI conference on human factors in computing systems (pp. 1-15).
  • 146. 14 8 Human-centered perspective on XAI • Explanations embedded within a socio-technical system • Explainability scenarios of possible uses • Human-AI interaction guidelines Ehsan, U., & Riedl, M. O. (2020, July). Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach. In International Conference on Human-Computer Interaction (pp. 449-466). Springer, Cham. Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., ... & Eckersley, P. (2020, January). Explainable machine learning in deployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (pp. 648-657). Wolf, C. T. (2019, March). Explainability scenarios: towards scenario-based XAI design. In Proceedings of the 24th International Conference on Intelligent User Interfaces (pp. 252-257). Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., ... & Teevan, J. (2019, May). Guidelines for human-AI interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-13).
  • 147. 14 9 Broadening XAI Who • Designing for multi-users systems3 • Creator-consumer gap w/r/t AI backgrounds2 • Leveraging questions from designers to design explanations1 1. Liao, Q. V., Gruen, D., & Miller, S. (2020, April). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-15). 2. Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I., Muller, M., & Riedl, M. O. (2021). The Who in Explainable AI: How AI Background Shapes Perceptions of AI Explanations. arXiv preprint arXiv:2107.13509. 3. Jacobs, M., He, J., F. Pradier, M., Lam, B., Ahn, A. C., McCoy, T. H., ... & Gajos, K. Z. (2021, May). Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-14).
  • 148. Broadening XAI 15 0 Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021, May). Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-16). Who
  • 149. 15 1 Broadening XAI When Onboarding1 AI lifecycle as a lens2,3 1. Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019). " Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making. Proceedings of the ACM on Human-computer Interaction, 3(CSCW), 1-24. 2. Dhanorkar, S., Wolf, C. T., Qian, K., Xu, A., Popa, L., & Li, Y. (2021, June). Who needs to know what, when?: Broadening the Explainable AI (XAI) Design Space by Looking at Explanations Across the AI Lifecycle. In Designing Interactive Systems Conference 2021 (pp. 1591-1602). 3.Hong, S. R., Hullman, J., & Bertini, E. (2020). Human factors in model interpretability: Industry practices, challenges, and needs. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW1), 1-26. 4. Anik, A. I., & Bunt, A. (2021, May). Data-Centric Explanations: Explaining Training Data of Machine Learning Systems to Promote Transparency. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-13). Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021, May). Expanding explainability: Towards social transparency in ai systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-19). What – Model outputs – Inner model working – Data-centric explanations4 – Visibility into context of explanations5
  • 150. 15 2 Human-Centered Approach to Explainabilty Human-Centered approach to Explainability AI lifecycle as a lens: User Study + Case Study Dhanorkar, S., Wolf, C. T., Qian, K., Xu, A., Popa, L., & Li, Y. (2021, June). Who needs to know what, when?: Broadening the Explainable AI (XAI) Design Space by Looking at Explanations Across the AI Lifecycle. In Designing Interactive Systems Conference 2021 (pp. 1591-1602).
  • 151. 15 3 Explainability in Practice RQ2: What is the nature of explanations and explainability concerns in industrial AI projects? RQ1: Over the course of a model’s lifecycle, who seeks and provides explanations? When do explanation needs arise?
  • 152. Domains Drug safety, key opinion leader mining, information extraction from business communications e.g., emails, contract document understanding, HR, chatbots, public health monitoring …. Interview Participants AI Engineers -7 Data Scientists - 13 Designers – 2 Information Vis Specialists - 3 Product Managers - 3 Technical Strategists - 5 *some individuals were overlapping in two roles
  • 153. 15 5 Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Business Users Product Managers Data Scientists Domain Experts Business Users Data Providers AI Engineers Data Scientists AI Engineers Data Scientists Product Managers Model Validators Business Users Designers Software Engineers Product Managers AI Engineers Business Users AI Operations Product Managers Model Validators
  • 154. 15 6 AI Lifecycle Touchpoints Initial Model building Model validation during proof-of-concept Model in-production Audience (Whom does the AI model interface with) Model Developers Data Scientists Product Managers Domain Experts Business Users Business IT Operations Model Developers Data Scientists | Technical Strategists Product Managers | Design teams BusinessUsers Business IT Operations Explainability Motivations (Information needs) - Peeking inside models to understand their inner workings - Improving model design (e.g., how should the model be retrained, re- tuned) - Selecting the right model - Characteristics of data (proprietary, public, training data) - Understanding model design - Ensuring ethical model development - Expectation mismatch - Augmenting business workflow and business actionability
  • 155. 15 7 Explanations during model development Understanding AI models inner workings “Explanations can be useful to NLP researchers explore and come up with hypothesis about why their models are working well… If you know the layer and the head then you have all the information you need to remove its influence... by looking, you could say, oh this head at this layer is only causing adverse effects, kill it and retrain or you could tweak it perhaps in such a way to minimize bias (I-19, Data scientist) “Low-level details like hyper-parameters would be discussed for debugging or brainstorming. (I-12, Data Scientist)
  • 156. 15 8 Explanations during model validation Details about data over which model is built “Domain experts want to know more about the public medical dataset that the model is trained on to gauge if it can adapt well to their proprietary data (I-4, Data Scientist) Model design at a high level “Initially we presented everything in the typical AI way (i.e., showing the diagram of the model). We even put equations but realized this will not work... After a few weeks, we started to only show examples and describing how the model works at a high level with these examples (I-4)
  • 157. 15 9 Explanations during model validation “How do we know the model is safe to use? ... Users will ask questions about regulatory or compliance-related factors: Does the model identify this particular part of the GDPR law? (I-16, Product Manager) Ethical Considerations
  • 158. 16 0 Explanations during model in-production Expectation mismatch “Data quality issues might arise resulting from expectation mismatch, but the list of recommendations must at least be as good as the manual process ... If they [the clients] come back with data quality issues ... we need to [provide] an explanation of what happened (I-5, Technical Strategist) “Model mistakes versus decision disagreements” (I-28, Technical Strategist) Explanation in service of business actionability “Is the feature it is pointing to something I can change in my business? If not, how does knowing that help me? (I-22, AI Engineer)
  • 159. Explainability in HEIDL HEIDL (Human-in-the loop linguistic Expressions wIth Deep Learning) Explainability takes on two dimensions here: models are fully explainable users involved In model co-creation with immediate feedback HEIDL video
  • 160. 16 2 Design Implications of AI Explainability Balancing external stakeholders needs with their AI knowledge “We have to balance the information you are sharing about the AI underpinnings, it can overwhelm the user (I-21, UX Designer) Group collaboration persona describing distinct types of members “Loss of control making the support and maintenance of explainable models hard (I-8, Data Scientist) Simplicity versus complexity tradeoff “The design space for (explaining) models to end users is in a way more constrained ... you have to assume that you have to put in a very, very very shallow learning curve (I-27 HCI Researcher) Privacy “Explanatory features can reveal identities (e.g., easily inferring employee, department, etc.) (I- 24, HCI researcher)
  • 161. 16 3 Future Work - User-Centered Design of AI Explainability Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Image source: https://www.slideshare.net/agaszostek/history-and-future-of-human-computer-interaction-hci-and-interaction-design Business Users Product Managers AI Engineers Data Scientists End-Users
  • 162. 16 4 Future Work - User-Centered Design of AI Explainability Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Image source: https://www.slideshare.net/agaszostek/history-and-future-of-human-computer-interaction-hci-and-interaction-design Business Users Product Managers AI Engineers Data Scientists End-Users
  • 163. 16 5 Future Work - User-Centered Design of AI Explainability Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Image source: https://www.slideshare.net/agaszostek/history-and-future-of-human-computer-interaction-hci-and-interaction-design Business Users Product Managers AI Engineers Data Scientists End-Users
  • 164. PART IV – Open Challenges & Concluding Remarks Marina Danilevsky Yunyao Li Lucian Popa Kun Qian Anbang Xu
  • 165. Explainability for NLP is an Emerging Field • AI has become an important part of our daily lives • Explainability is becoming increasingly important • Explainability for NLP is still at its early stage
  • 166. Challenge 1: Standardized Terminology Interpretability explainability transparency Directly interpretable Self-explaining Understandability Comprehensibility Black-box-ness White-box Gray-box … … … 168
  • 167. Challenge 2: What counts as explainable? 169 Attention weights indicate most predictive tokens Attention mechanisms are noisy – even for intermediate input components’ importance Attention weights are uncorrelated with gradient-based feature importance measures Different attention distributions can yield equivalent predictions May be hard to say tokens are “responsible for” model output – should not treat attention as decision justification Attention can give a plausible reconstruction though with no guarantee of faithfulness A model can be forced to reduce attention on predictive tokens, corrupting the attention-based explanation Depends on what you’re looking for: plausible or faithful explanations?
  • 168. Challenge 3: Stakeholder-Oriented Explainability Business Understanding Data Acquisition & Preparation Model Development Model Validation Deployment Monitor & Optimize Business Users Product Managers AI Engineers Data Scientists End-Users 170
  • 169. Challenge 3: Stakeholder-Oriented Explainability Statement of Work Company A Company B … Time is of essence of this Agreement. …. label = Risky Contains($sentence, $time) = False AI engineer Statement of Work Company A Company B … Time is of essence of this Agreement. …. … Risky. Need to define time frame to make this clause meaningful. Potential risk: If no time specified, courts will apply a "reasonable" time to perform. What is a "reasonable time" is a question of fact - so it will up to a jury. legal professional Different stakeholders have different expectations and skills 171
  • 170. Challenge 4: Explainability AND Accuracy Current Focus Holy Grail Explainability Accuracy H H L L Explain Blackbox ML Models Fully Explainable Models Source: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature. Machine Intelligence. Surrogate models: • OK to explain a black-box model, but fidelity suffers Model design geared towards explainability • Can be both explainable and high-accuracy! 172
  • 171. Challenge 5: Appropriate Trust-Oriented Explainability • Engender appropriate trust – Help human trust the AI, when it is right – Expose the AI’s errors, when it is wrong • Maximize AI and explanation’s helpfulness • Take nature of the task into account 173 Credit: Dan Weld. Keynote talk “Optimizing Human-AI Teams”. DaSH-LA @NAACL’2021 Source: “Does the whole exceed its parts? The effect of AI explanations on complementary team performance. CHI’2021c
  • 172. Challenge 6: Evaluation Methodology & Metrics • Need more well-established evaluation methodologies • Need more fine-grain evaluation metrics – Coverage of the prediction – Fidelity • Take into account the stakeholders as well! 174
  • 173. Path Forward • Explainability research for NLP is an interdisciplinary topic – Computational Linguistics – Machine learning – Human computer interaction • NLP community and HCI community can work closer in the future – Explainability research is essentially user-oriented – but most of the papers we reviewed did not have any form of user study – Get the low-hanging fruit from the HCI community This tutorial is on a hot area in NLP and takes an innovative approach by weaving in not only core technical research but also user-oriented research. I believe this tutorial has the potential to draw a reasonably sized crowd. – anonymous tutorial proposal reviewer “Furthermore, I really appreciate the range of perspectives (social sciences, HCI, and NLP) represented among the presenters of this tutorial.” – anonymous tutorial proposal reviewer NLP people appreciate the idea of bringing NLP and HCI people together 175
  • 174. Thank You • Project website: https://xainlp.github.io/ – Tutorial website: https://xainlp.github.io/kddtutorial/ • Questions and discussion 176