1. Measuring ROI from data science initiatives is challenging for many organizations as the outcomes are often not clearly defined, quantified, or attributed to the initiatives. Breaking the chain from data to insights to actions to outcomes is common. 2. A framework is presented for quantifying the value of data science initiatives using 5 steps - define success metrics, measure the metrics, attribute outcomes to causal factors, calculate net costs and benefits to determine breakeven, and benchmark results. 3. The framework is applied to a case study of a beverage manufacturer that used analytics to optimize plant costs. Key metrics like cost savings, employee productivity, and process efficiency were defined and attribution methods like A/B testing were used

1. How to Measure
ROI from Your
Data Science
Initiatives
Ganes Kesari
Webinar, July 2021

2. Was success declared a bit too early?

3. Why is ROI so
difficult?
How to compute
returns?
How can you do it
right?

4. 4
INTRODUCTION
Ganes Kesari
Co-founder & Chief Decision Scientist
“Simplify Data Science for all”
100+ Clients
Help start, apply and
adopt Data Science
@kesaritweets
/gkesari

5. 5
Why is ROI so difficult?

6. State of the Industry: Data & Analytics investments vs returns
Reference: ESIThoughtLab report – Driving ROI from AI
We hear more about the money invested.. ..than about the returns generated
On average, firms invested over
$38 M in data & analytics
Over 79% of these firms report
negative or no ROI

7. Reason #1: Data often doesn’t lead to outcomes
Reference: ESIThoughtLab report – Driving ROI from AI
Data Insights Recommendations Actions Outcomes
This chain is very often broken…
Model stats are
a poor cousin of
insights
Most
dashboards
don’t prescribe
actions
The actions
don’t connect
to a workflow
Data sits idle and
untapped
“ 80% of projects don’t deliver outcomes in the industry
Outcomes
often go
unmeasured

8. Reason #2: The outcomes are often not quantified as benefits
Higher
Efficiency
Better
decisions
Improved
branding
“ One measurement is worth a thousand expert opinions.
- Grace Hopper

9. Reason #3: The benefits may not have been caused by your project
Disney+ launched in November 2019 had a stretch
goal to acquire 90 million subscribers by 2024.
It beat this goal in 14 months.
Was this due to a brilliant marketing strategy, or
was it just a pandemic windfall?

10. The 5 steps to compute ROI

11. What’s your data really worth?
Data has a lot of intrinsic value…
…or indirectly, the value will be
wasted
…but unless you monetize it
directly..

12. 12
Define Success
What are your target
outcomes?
Tally Results
What’s your benchmark
performance?
Reckon
Breakeven
What are your net costs
& benefits?
Attribute
Outcomes
What led to the results
observed?
Measure Value
How will you measure
your outcomes?
Success
S
Measure
M
Attribution
A
Reckon
R
Tally
T
The SMART framework to quantify value from data & analytics initiatives
S M A R T

13. 13
Beverage manufacturer uses analytics
to optimize plant cost
Challenge
“Drink It” is a leading global manufacturer of soft
drinks. Over the past year, the company has
observed a rise in bottling costs across regions.
Approach
• Statistical diagnostic analytics to identify
recommendations for improvement
• Command centre with predictive capabilities
to run scenario-based simulations

14. 14
1. Define Success: What are your target outcomes?
Organizational
Financial
Innovation
Stakeholder
Customer
Pick the right mix of outcome categories…

15. 15
1. Define Success: What are your target outcomes?
Finance Innovation Customer Organization
Portfolio mix
Stakeholder
Revenue
Margin
Cash Flow
Employees
Investors
Customer Exp.
Business value Brand Equity
Ops. Perf.
ESG
Organization
Financial
Innovation
Stakeholder
Customer
Innov. culture
..and identify the attributes that your initiative must influence

16. 16
DRINK IT: Definition of success outcomes
Strategic Objectives Users Outcomes
• Optimize the global
manufacturing cost
• Improve resource and
employee utilization
• Streamline
manufacturing process
• Plant managers
• Regional
Leadership
• Financial
• Cost saving
• Employee
• Enablement
• Utilization
• Organization
• Ops. Performance
• ESG
“Drink it”

17. 17
2. Measure Value: How will you quantify your outcomes?
Finance Innovation Customer Organization
Stakeholder
Organization
Financial
Innovation
Stakeholder
Customer
• Revenue growth
• Gross margin
• Operating
expense
• Receivable​s/
payables
• New products
launched
• Revenue from
new products
• Employee
innovation index
• Attrition rate​
• Employee
satisfaction​
• Stock Price​
• Return on equity
• CSAT score,​
• Customer Lifetime
Value​
• Acquisition Cost​
• Share of wallet​
• Time to Market​
• Process cycle
time​
• Brand Salience​
• Reduction of
carbon footprint

18. 18
DRINK IT : Measuring the success metrics through FISCO
Finance
Employee
Organization
Cost saving
Enablement
Utilization
Ops. Perf.
ESG
• Manufacturing cost per case
• Production line cost per plant
• People empowerment
• Employee productivity
• Manufacturing process cycle time
• Asset utilization
• Reduction of carbon footprint
Outcome
Categories
Success
attributes
Success
metrics
“Drink it”

19. 19
3. Attribute Outcomes: What led to the results observed?
Causal factor 1 Causal factor 2
Causal factor 3 Causal factor 4
Outcomes
Identify all factors that could influence your target outcomes

20. 20
3. Attribute Outcomes: What led to the results observed?
A/B Testing DoWhy Causal Testing
Reference: Microsoft GitHub report – DoWhy | An end-to-end library for causal inference
A B
10% attrition 7% attrition

21. 21
DRINK IT: Causal diagram representation
Manufacturing process Market factors
Material cost Employee efficiency
Manufacturing
plant
cost
Process efficiency
Transportation cost
Labour cost
Command Tower insights
Employee productivity
Seasonality
Competitors
Staffing levels
Demand
Material cost
“Drink it”
A/B Testing helped isolate benefits delivered by analytics

22. 22
4. Reckon Breakeven: What are your net costs and benefits?
Gain Heads
Cost Heads
• Revenue,
• Margin,
• Cash flow
• Building capabilities
(Software & Hardware)
• Effort
• Time
• Opportunity cost
• Innovation
• Employees, Investors
• Customer experience,
Business Value
• Operations, Brand
Equity, ESG
Balance your qualitative benefits with quantitative measures

23. 23
Cost Heads: Account for obvious and non-intuitive expenditure
People
Efforts
Technology
Infrastructure
Capability
Building
Opportunity
Cost
People
Effort
Technology
Infrastructure
Capability
Building
Opportunity
Cost
Technology team, business
team, change management
Hardware, software,
infrastructure costs
Data initiatives, Technology
capabilities
Return on best foregone
option - Return on chosen
option

24. 24
DRINK IT: Cost & Gain heads analysis
Cost Heads
People
Effort
Implementation + Training
+ Support + Change mgmt
Technology
Infrastructure
Software + Hardware +
Maintenance + Other
infrastructure
Capability Building
Analytics delivery process
+ Workflow integration
Opportunity Cost
Projected benefits from
lean process improvement
Total cost: $4.25 M
Gain Heads
Finance Cost saving
Asset utilization cost saving
+ Material cost decrease
Employee
Enablement
2.1 points Employee
satisfaction score increase
Utilization
8% increase in employee
productivity
Organizati
on
Process
Efficiency
6% Improvement in process
cycle time
ESG
5% Carbon footprint
reduction
Financial savings: $4 M + Qualitative benefits
“Drink it”

25. 25
5. Tally Results: What’s your benchmark performance?
Average payback (years) across industries
Reference: ESIThoughtLab report – Driving ROI from AI
Compare ROI across the firm, against competitors, and the industry
“ The average payback on data investments is ~17 months
Average payback (years) by org D&A maturity
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Beginner Implementer Advancer Leader

26. 26
Payback in 1.2 years
(Manufacturing industry
average: 1.32 years)
ROI: $ 4.00 M
Investment: $ 4.25 M
Benefits and Cost
Summary
Breakeven
Benchmark
DRINK IT: Benchmark ROI across industries and track progress
“Drink it”

27. 27
How can you do it right?

28. 28
Applying the learnings to your organization
For how many of your
projects is the outcome
crystal clear?
Is your breakeven better
than your industry
average?
Is there a KPI or a metric
that signifies it?
Are you confident that
your project improved the
metric?
Do you have a handle
on all your costs?
01
02
03 05
04

29. We regularly share our thought leadership through articles, talks and webinars
29
We regularly speak in events and conduct webinars to share knowledge on getting value from data
Featured Talks
Featured Publications
When and how to
build out your data
science team
How to beat
resistance to AI
projects: 3 steps for
executives
How NGOs can
leverage AI without
breaking the bank
Why data leaders
must play offense
during COVID-19
We frequently publish articles in leading magazines to share insights with executives and practitioners
Webinar: How to
Build Successful Data
Science Teams
Webinar: How to structure
your Analytics teams for
the Best Outcomes
Webinar: The best
way to Choose your
Data Science Projects
Panel: Role of AI
Strategy and Culture
in Org Transformation
The 5 roles that every
data science team
must hire
Whiteboard Series:
Executive insights in
under 5 minutes

30. 30
Thank You!
@kesaritweets
/gkesari www.gramener.com
Reach out for a free discovery session: https://gramener.com/data-advisory-workshop
Ganes Kesari

31. 31
Planning starts early but realization happens at later stage
• Clarity of Vision &
Strategy Alignment
what the organization
intends to achieve and
Alignment with the
corporate vision and
long-term goals.
• Initiatives Planning &
Project Prioritization
How the initiatives are
planned and governed
to ensure business
value.
• Data Analytics & Data
Consumption
Generation &
presentation of
actionable insights
from data(dashboards,
visualization, data
stories).
• Adoption & ROI
Adoption of the data
initiatives.
Quantification, and
tracking of the value
generated from data
science initiatives.
• Data Literacy
Ability of the users to
read, write, and
communicate with data
P L A N N I N G
V I S I O N E X E C U T I O N
R E A L I Z E
V A L U E
D A T A
C U L T U R E
Insights & Recommendation Action
Most of the organizations leave their data initiatives at the action stage which not only makes them
suffer in quantifying ROI but also makes their data initiatives less successful
Reference: Gramener toolkit
Data

32. Poll #1
32
Which of these is the biggest
challenge for your organization?
Here’s a short & simple poll to help you
reflect.

33. Poll #2
33
Have you ever quantified ROI
for your data initiatives?
Here’s a short & simple poll to help you
reflect.

34. Poll #3
34
Which of these 5 steps do
you need help with?
Here’s a short & simple poll to help you
reflect.