The document discusses explainable AI (XAI) and making machine learning and deep learning models more interpretable. It covers the necessity and principles of XAI, popular model-agnostic XAI methods for ML and DL models, frameworks like LIME, SHAP, ELI5 and SKATER, and research questions around evolving XAI to be understandable by non-experts. The key topics covered are model-agnostic XAI, surrogate models, influence methods, visualizations and evaluating descriptive accuracy of explanations.

1. Explainable AI – Making ML and DL models more interpretable
Explainable AI – Making ML and DL models more interpretable

2. About Me
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
2
Aditya
Bhattacharya
I am currently working as the Lead AI/ML Engineer at West Pharmaceutical
Services with the responsibility of leading and managing a global AI team and
creating AI products and platforms at West. I am well seasoned in Data Science,
Machine Learning, IoT and Software Development. and has established the AI
Centre of Excellence and worked towards democratizing AI practice for West
Pharmaceuticals and Microsoft. In the Data Science domain, Computer Vision,
Time-Series Analysis, Natural Language Processing and Speech analysis are my
forte.
Apart from my day job, I am an AI Researcher at an NGO called MUST Research,
and I am one of the faculty members for the MUST Research Academy :
https://must.co.in/acad
Website : https://aditya-bhattacharya.net/
LinkedIn: https://www.linkedin.com/in/aditya-bhattacharya-b59155b6/

3. Key Topics 1. Necessity and Principles of Explainable AI
2. Model Agnostic XAI for ML models
3. Model Agnostic XAI for DL models.
4. Popular frameworks for XAI
5. Research Questions to consider
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 3

4. Necessity and Principles of
Explainable AI

5. 5
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e

6. 6
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e

7. 7
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e

8. 8
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
XAI
Trace model
prediction from
logic of math to
nature of data
Understand the
reasoning behind
each model
predictions
Understand the
model using which
AI decision
making is based
Traceable
AI
Reasonable
AI
Understand
able
AI

9. 9
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e

10. 1 0
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e

11. Model
Agnostic
Results
Visualizations
Influence
Methods
Example
Based
Methods
Knowledge
Extractions
1 1
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Using Surrogate models like linear
models or decision trees to explain
complex models
Estimates the
importance or relevant
features.
Extracting statistical
information from input
and the output
Select instances of the datasets that
explains the behaviour of the model

12. 1 2
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Problem, Data,
Audience
Post Hoc
Analysis
Model
Predictive
Accuracy
Descriptive
Accuracy
Iterative
Explainability

13. Model Agnostic XAI for ML models

14. 1 4
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e

15. 1 5
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e

16. 1 6
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Explainer
Surrogate Models
Predictions
Blackbox ML Model

17. 1 7
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e

18. E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 1 8
Prediction: Deny Loan
Loan Application
Suggestion: Increase your salary by 50K & pay your credit card bills on time for next 3 months
Predictive
Model
Loan Applicant
Counterfactual Generation
Algorithm

19. Model Agnostic XAI for DL models
1 9

20. 2 0
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Understanding flow of information through gradient flow between the
layers of Deep Neural Network model using the following approaches:
1. Saliency Maps
2. Guide Backpropagation
3. Gradient Class Activation Methods
• Layer GRAD CAM
• Layer Conductance using GRAD CAM
• Layer Activation using GRAD CAM
Saliency Maps Guided Backprop GRAD CAM Layer Conductance Layer Activation

21. Can such explainability
methods be applied for
complex models?
2 1
Explain ab l e AI: Making ML and DL models more interpr et a b l e

22. 2 2
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Image Captioning
using Attention based
Encoder-Decoder
Architecture

23. 2 3
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
[Kim et. al., 2018]
Zebra
(0.97)
How important is the notion of “stripes” for this prediction?
Testing with Concept Activation Vectors (TCAV) is an interpretability method to understand what signals
your neural networks models uses for prediction.
https://github.com/tensorflow/tcav
Pattern representation plays
a key role in decision making
from both images and text.

24. 2 4
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
[Tan et. al., 2019]
Model
Predictions
Label 1
Label 1
.
Label 2
.
v1, v2.
v11,
v12
.
Data
Explainer
Interpretable Mimic Learning – Compressing information from Deep Networks to Shallow Network

25. 2 5
Explainable AI: Making ML and DL models more interpretable
What features need to be changed and by how much to flip a model’s prediction?
[Goyal et. al., 2019]

26. Popular frameworks for XAI
Explain ab l e AI: Making ML and DL models more interpr et a b l e

27. Popular frameworks for XAI
2 7
LIME
Local Interpretable
Model-agnostic
Explanations is
interpretability
framework that
works on
structured data,
text and image
classifiers.
SHAP
SHAP (SHapley
Additive
exPlanations) is a
game theoretic
approach to
explain the output
of any machine
learning model.
ELI5
Explain like I am 5
is another popular
framework that
helps to debug
machine learning
classifiers and
explain their
predictions.
SKATER
Skater is a unified
framework for XAI
for all forms of
models both
globally(inference
on the basis of a
complete data set)
and
locally(inference
about an individual
prediction).
TCAV
Testing with
Concept Activation
Vectors (TCAV) is a
new interpretability
method to
understand what
signals your neural
networks models
uses for prediction.
Explainable AI: Making ML and DL models more interpretable

28. 2 8
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
• Behind the workings of LIME lies the assumption that every complex model is linear on a local scale. LIME tries
to fit a simple model around a single observation that will mimic how the global model behaves at that
locality.
• Create the perturbed data and predict the output on the perturbed data
• Create discretized features and find the Euclidean distance of perturbed data to the original observation
• Convert distance to similarity score and select the top n features for the model
• Create a linear model and explain the prediction

29. 2 9
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
The lime package is on PyPI. `pip install lime`

30. 3 0
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
There is a high-speed exact algorithm for tree ensemble methods (Tree SHAP arXiv paper). Fast C++
implementations are supported for XGBoost, LightGBM, CatBoost, and scikit-learn tree models!
• SHAP assigns each feature an importance
value for a particular prediction.
• Its novel components include: the
identification of a new class of additive
feature importance measures, and theoretical
results showing there is a unique solution in
this class with a set of desirable properties.
• Typically, SHAP values try to explain the
output of a model (function) as a sum of the
effects of each feature being introduced into
a conditional expectation. Importantly, for
non-linear functions the order in which
features are introduced matters.
SHAP can be installed from PyPI

31. 3 1
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
The following figure from the KDD 18 paper, Consistent Individualized Feature
Attribution for Tree Ensembles summarizes this in a nice way!
SHAP Summary Plot
SHAP Dependence Plots

32. 3 2
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Available from pypi. pip install eli5
Check docs for more.

33. 3 3
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
SKATER provides an unified framework for both Global and Local Interpretation.
Feature Importance Partial Dependency Plots
LIME integration for explanability
Project Link:

34. 3 4
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
Testing with Concept Activation Vectors (TCAV)
is a new interpretability method to understand
what signals your neural networks models uses
for prediction.
What's special about TCAV compared to
other methods?
TCAV instead shows importance of high
level concepts (e.g., color, gender, race)
for a prediction class - this is how humans
communicate!
TCAV gives an explanation that is generally true for a class of interest, beyond one image (global
explanation).
For example, for a given class, we can show how much race or gender was important for classifications in
InceptionV3. Even though neither race nor gender labels were part of the training input!
pip install tcav https://github.com/tensorflow/tcav

35. E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
3 5
The Concept Activation Vectors (CAVs) provide an interpretation of a neural net’s internal
state in terms of human-friendly concepts. TCAV uses directional derivatives to quantify the
degree to which a user-defined idea is vital to a classification result–for example, how sensitive
a prediction of “zebra” is to the presence of stripes.
TCAV essentially learns ‘concepts’ from examples. For instance, TCAV needs a couple of
examples of ‘female’, and something ‘not female’ to learn a “gender” concept. The goal of
TCAV is to determine how much a concept (e.g., gender, race) was necessary for a prediction
in a trained model even if the concept was not part of the training.

36. Research question to consider …

37. All these frameworks are great
and can bring interpretability
to a great extent, but can non-
expert consumers of AI
models interpret these
interpretability methods?
3 7
Explain ab l e AI: Making ML and DL models more interpr et a b l e

38. Summary
• Why is Explainable AI (XAI) important?
• Commonly used Model Agnostic XAI for ML models
• Commonly used Model Agnostic XAI for DL models.
• Popular frameworks for XAI
• Can we evolve XAI and extend explainability to non-expert
users?
3 8
Explain ab l e AI: Making ML and DL models more interpr et a b l e

39. Thank you
Aditya Bhattacharya
https://aditya-bhattacharya.net/
aditya.bhattacharya2016@gmail.com
https://www.linkedin.com/in/aditya-bhattacharya-b59155b6/
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 3 9