A Unified Framework for Model Explanation Ian Covert, University of Washington Explainable AI is becoming increasingly important, but the field is evolving rapidly and requires better organizing principles to remain manageable for researchers and practitioners. In this talk, Ian will discuss a new paper that unifies a large portion of the literature using a simple idea: simulating feature removal. The new class of "removal-based explanations" describes 20+ existing methods (e.g., LIME, SHAP) and reveals underlying links with psychology, game theory and information theory. Practical examples will be presented and available on the Qu.Academy site Reference: Explaining by Removing: A Unified Framework for Model Explanation Ian Covert, Scott Lundberg, Su-In Lee https://arxiv.org/abs/2011.14878

1. Qu Speaker Series
Explainable AI Workshop
Explaining by Removing: A Unified Framework for Model Explanation
Ian Covert
University of Washington
2020 Copyright QuantUniversity LLC.
Hosted By:
Sri Krishnamurthy, CFA, CAP
sri@quantuniversity.com
www.qu.academy
12/16/2020
Qu.Academy
https://quspeakerseries18.splashthat.com/

2. 2
QuantUniversity
• Boston-based Data Science, Quant
Finance and Machine Learning
training and consulting advisory
• Trained more than 1000 students in
Quantitative methods, Data Science
and Big Data Technologies using
MATLAB, Python and R
• Building a platform for AI
and Machine Learning Exploration
and Experimentation

3. 3
For registration information, go to
https://QuFallSchool.splashthat.com

4. 4
https://Quwinterschool.splashthat.com

5. 5
Next Week

6. 6

7. 7
Demos, slides and video available on QuAcademy
Go to www.qu.academy
7

8. Explaining by Removing: A Unified
Framework for Model Explanation
Ian Covert

9. Ian Covert, Scott Lundberg, Su-In Lee. “Explaining by Removing: A Unified
Framework for Model Explanation.” arXiv preprint:2011.14878
Our paper

10. 1. Motivation
2. A unified framework
3. Demonstration
4. Why feature removal?
5. Are there “right” choices?
Contents

11. > Black-box models increasingly popular
> Explainable AI increasingly important
> Which methods should we rely on?
Motivation
𝑓

12. > SHAP, LIME, SAGE, Occlusion, DeepLift, SmoothGrad, Integrated
Gradients, GradCAM, CXPlain, L2X, INVASE, Meaningful
Perturbations, Extremal Perturbations, RISE, TCAV, Guided Backprop,
Excitation Backprop, IME, QII, PredDiff, MIR, Permutation Tests, LRP,
FIDO-CA, Masking Model, Expected Gradients, LossSHAP, Shapley
Effects, MP2-G, Saliency Maps, PDPs, ICEs, TreeSHAP, …
Example methods

13. > Researchers have made significant progress
> The field is fragmented
> Growing very fast
> Lacking discussion of underlying principles
Is the field in a good place?

14. Monolithic algorithms
à interchangeable
choices
Goals of this talk
A unifying theory
that describes 20+
methods
One key idea about
how to explain ML
models

15. 1. Motivation
2. A unified framework
3. Demonstration
4. Why feature removal?
5. Are there “right” choices?
Contents

16. > Many methods implicitly simulate feature removal
> Non-trivial operation, different approaches
> Further differences in generating final “explanation”
Explaining by removing

17. A simple framework

18. A simple framework

19. A simple framework

20. A simple framework

21. > The model 𝒇 requires a specific set of features
𝒇(𝒙) for 𝒙 ∈ 𝓧
> Require a subset function 𝑭 that accepts a feature subset
𝑭(𝒙 𝑺) for 𝒙 ∈ 𝒳 and 𝑺 ⊆ 𝟏, 𝟐, … , 𝒅
1. Feature removal

22. > Select a target quantity to explain
§ Individual prediction: 𝒗 𝒙 𝑺 = 𝑭 𝒙 𝑺
§ Prediction loss: 𝒗 𝒙𝒚 𝑺 = − ℓ 𝑭 𝒙 𝑺 , 𝒚
§ Dataset loss: 𝒗 𝑺 = − 𝔼 ℓ 𝑭 𝑿 𝑺 , 𝒀
2. Model behavior
𝑓

23. > Every method has an underlying set function 𝒗 𝑺
3. Summary technique

24. > Every method has an underlying set function 𝒗 𝑺
3. Summary technique

25. > Every method has an underlying set function 𝒗 𝑺
3. Summary technique

26. > Every method has an underlying set function 𝒗 𝑺
> Explanations provide a concise summary
3. Summary technique

27. > Feature attribution: output scores 𝒂 𝟏, 𝒂 𝟐, … , 𝒂 𝒅
Summary types

28. > Feature attribution: output scores 𝒂 𝟏, 𝒂 𝟐, … , 𝒂 𝒅
> E.g., remove individual
Summary types
−

29. > Feature attribution: output scores 𝒂 𝟏, 𝒂 𝟐, … , 𝒂 𝒅
> E.g., remove individual, include individual
Summary types
−

30. > Feature attribution: output scores 𝒂 𝟏, 𝒂 𝟐, … , 𝒂 𝒅
> E.g., remove individual, include individual
> Feature selection: output influential set 𝑺∗ ⊆ 𝟏, 𝟐, … , 𝒅
Summary types

31. Example methods

32. > Removal: default values
> Behavior: individual prediction
> Summary: fit linear model
Example: LIME (2016)

33. > Removal: marginalize out
> Behavior: individual prediction (same as LIME)
> Summary: Shapley value
Example: SHAP (2017)

34. > Removal: marginalize out (same as SHAP)
> Behavior: dataset loss
> Summary: Shapley value (same as SHAP)
Example: SAGE (2020)

35. > Removal: marginalize out (same as SAGE)
> Behavior: dataset loss (same as SAGE)
> Summary: remove individual
Example: permutation test (2001)
−

36. > 20+ existing methods
> Local and global
> Feature attribution and
feature selection
A unifying framework

37. A unifying framework
Remove
individual
Include
individual
Mean when
included
Shapley value
Linear
model
High value
subset
Low value
subset
Partitioned
subsets
Zeros
Occlusion
CXPlain
RISE MM
Default values
LIME
(images)
Extend pixels MIR
Blurring EP MP
Generative
model
FIDO-CA
Marginalize
(replacement
distribution)
LIME
(tabular)
Marginalize
(uniform)
IME (2010)
Marginalize
(marginals
product)
QII
Marginalize
(marginal)
Permutation
test
SHAP
KernelSHAP
Marginalize
(conditional)
PredDiff
Conditional
perm. test
SHAP SAGE
LossSHAP
Shapley Effects
Tree
distribution
TreeSHAP
Missingness
during training
L2X
INVASE
Separate
models
Feature
ablation
Univariate
predictors
IME (2009)
Shapley Net
Effects
Summary technique
Featureremoval
∎ Prediction ∎ Prediction loss ∎ Mean prediction loss ∎ Dataset loss ∎ Dataset loss (output)Model behavior
Feature attribution Feature selection
> Method “space”
> Neighboring methods
> Unique methods, new
methods

38. 1. Motivation
2. A unified framework
3. Demonstration
4. Why feature removal?
5. Are there “right” choices?
Contents

39. > GitHub: https://github.com/iancovert/removal-explanations
> Simple implementation of many methods
> Focus on interchangeable choices
Demonstration

40. Summary
> A new class of methods based on feature removal
> Each method is specified by three choices
> Framework offers a great degree of flexibility
Removal-based explanations

41. 1. Is feature removal a smart approach to model explanation?
2. Are there “right” choices for each dimension?
Key questions

42. 1. Motivation
2. A unified framework
3. Demonstration
4. Why feature removal?
5. Are there “right” choices?
Contents

43. > Intuitive to many research groups
> Feature removal is a form of counterfactual reasoning
> Undo act of observing information (rather than changing what was
observed)
> Removal is anchored in psychology (subtractive counterfactual) and
philosophy (method of difference)
Why feature removal?

44. > Counterfactuals change aspects of a situation
(observation of feature values)
> Can understand models by changing inputs
> Often complicated to explore
> Feature removal gives a more practical way to
explore and summarize functions
Counterfactual reasoning

45. 1. Motivation
2. A unified framework
3. Demonstration
4. Why feature removal?
5. Are there “right” choices?
Contents

46. “Right” choices?
> Methods determined by three choices
> Every method has something to offer
> Conceptual and computational trade-offs

47. > Marginalizing out features with their conditional
distribution
> Difficult to implement, but many approximations
> Yields information-theoretic explanations
Feature removal strategy

48. > Intuitively (Chest X-Ray)
> How should a doctor interpret this?
> Mathematically:
𝑭 𝒙 𝑺 = 𝔼 𝒇 𝑿 𝑿 𝑺 = 𝒙 𝑺
= ∫ 𝒑 𝒙(𝑺 𝒙 𝑺 𝒇(𝒙 𝑺, 𝒙(𝑺)
Conditional distribution removal

49. Information-theoretic explanations

50. > Any choice provides valuable information
> Range of perspectives
> Depends on use-case
Model behavior

51. > Precedent in cooperative game theory
> Many potential desirable properties
> Shapley value satisfies most
𝝓𝒊 𝒗 =
𝟏
𝒅
<
𝑺⊆𝑫-
𝒅 − 𝟏
𝑺
.𝟏
𝒗 𝑺 ∪ 𝒊 − 𝒗 𝑺
Summary technique

52. Game-theoretic explanations

53. 1. Motivation
2. A unified framework
3. Why feature removal?
4. Demonstration
5. Are there “right” choices?
6. Conclusion + questions
Contents

54. > Presented a new perspective for understanding explainability tools
> Developed rigorous foundations
> Aim to inform practitioners, guide researchers
Concluding thoughts

55. Questions

56. 56
Instructions for the Lab:
1. Go to https://academy.qusandbox.com/#/register and register using the code:
"QUFALLSCHOOL"

57. Thank you!
Sri Krishnamurthy, CFA, CAP
Founder and CEO
QuantUniversity LLC.
srikrishnamurthy
www.QuantUniversity.com
Contact
Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be
distributed or used in any other publication without the prior written consent of QuantUniversity LLC.
57