I review three frameworks for analytic operations that are designed to improve the value obtained when deploying analytic models into products, services and internal operations.

1. Some Frameworks to Improve Analytic Operations
Robert L. Grossman
Analytic Strategy Partners LLC
& University of Chicago
June 6, 2019

2. 1. Case Study: IBM Oncology*
*This section is adapted from Robert L. Grossman, The Strategy and Practice of Analytics, 2020, to appear.

3. On Aug 11, 2018, the Wall Street Journal reported: “In many cases, [Watson] didn’t
add much value. In some cases, Watson wasn’t accurate. Watson can be tripped up by
a lack of data in rare or recurring cancers, and treatments are evolving faster than
Watson’s human trainers can update the system.”

4. IBM Watson for Oncology
• Based upon the success of IBM Watson winning Jeopardy in 2011, IBM branded a new type
of computing called cognitive computing and launched a new Division of IBM that thus far
has spent over $15B.
• The technology is based upon natural language processing and includes a framework for
answering questions.
• Watson Oncology is critically dependent upon access to cancer data, expert oncologists, and
a smart team of software engineers working for several years.
• Watson Oncology business model: does Watson replace the oncologist, double check the
oncologist, augment the oncologist for difficult cases? IBM charges $200 - $1000 / patient.
• The Wall Street Journal reports: “In many cases, the tools didn’t add much value. In some
cases, Watson wasn’t accurate. Watson can be tripped up by a lack of data in rare or
recurring cancers, and treatments are evolving faster than Watson’s human trainers can
update the system.”
Source: Daniela Hernandez and Ted Greenwald, IBM Has a Watson Dilemma, Wall Street Journal, August 11, 2018.

5. Framework: Develop, Deploy and Extract (DDE)
Develop the model Deploy the model Extract value from the Model
• Build oncology models
using IBM’s cognitive
computing technology.
• Develop models to
determine what type
and sub-type of cancer
does the patient have?
• Is the model
integrated into an
existing system or
stood up as a
separate system?
• Does the model
make a diagnosis
(Dx), recommend a
treatment, or both?
• Does the model
replace the
oncologist making
the diagnosis (Dx)?
• Does the model
double check the
oncologist?
• Does the model
handle difficult
cases?

6. 2. What Are Analytic Operations?
This section is adapted from Robert L. Grossman, Developing an AI Strategy: a Primer, Open Data Press, 2020, available online at analyticstrategy.com

7. Analytic
strategy
Analytic algorithms & models Analytic operations
“Amateurs talk about tactics,
but professionals study
logistics."
- Gen. Robert H. Barrow, USMC
(Commandant of the Marine
Corps)
Amateurs talk about
analytic models but
professionals study
analytic operations and
analytic infrastructure.
Analytic Infrastructure
The Analytic Diamond
This section is adapted from Robert L. Grossman, Developing an AI Strategy:
a Primer, Open Data Press, 2020, available online at analyticstrategy.com

8. Analytic
Diamond
Analytic algorithms
& models
Analytic operations
Deploying analytic models into operations
Analytic Infrastructure
The Analytic Chasm

9. Analytic algorithms
and models
Analytic operations
1. How do you deploy the model
into operational systems?
2. How do you quickly detect
drift and update a deployed
model?
3. What actions are associated
with a model and what value
do they provide?
4. Are there “hooks” that can
increase the value of the
actions?
Deploy the model
Extract value
Analytic infrastructure

10. Analytic algorithms
and models
5. Are there segments in which
more specialized actions can
increase the value?
6. How do you measure and report
the value generated to the
product/service owner and
other stakeholders?
7. How do you provide the required
data and model security?
8. How do you provide the required
privacy and compliance?
Analytic infrastructure
Analytic operations
Extract value
Protect

11. Get the data, set up
the infrastructure,
put in place the
compliance and
security, etc.
Analyze &
model the
data.
Deploy the solution with
the model in a manner that
provides value to the
organization.
Time
Effort
Get the data Build a
model
Deploy the model

12. 3. Scores, Actions and Measures—
The SAM Framework*
This section is adapted from Robert L. Grossman, Developing an AI Strategy: a Primer, Open Data Press, 2020, available online at analyticstrategy.com

13. Scores vs Actions
Score Action
Credit Likelihood to default Do you offer a prescreened card? If so,
what is the credit line and interest rate?
Response Likelihood to response
to an offer
Which ad from an inventory do I offer
and how many impressions?
Hospital
readmission
Likelihood to be
readmitted to a hospital
within 90 days
Delay release from hospital; follow up
after release; etc.
Election
model
Likelihood of support
(which candidate do you
support). Likelihood of
voting.
If support is high, ask for $ or to
volunteer. If likelihood is below a
threshold, help them make a plan to get
out to vote (GOTV)

14. SAM Framework
Models
Scores
Actions
Measures Dashboards
Model
Consumer
• Measures quantify
business value
generated by
actions
• Measures should
align with analytic
strategy.
Model
Producer
Analytic
operations
Develop
model

15. Presidential Campaigns
• Goal: Win 270 electoral votes
• Three models
o Support: Prob that a vote is for your candidate?
o Persuasion: Can someone be persuaded
to vote for Candidate?
o Turnout: Will someone vote?
• Actions
o Send email, knock on their door,
o Ask for dollars, ask them to volunteer
(if they are likely to vote for your candidate)
o Help them make a plan to get out and vote
o Have them talk to their Facebook friends (build FB apps)
o Target actions around specific events in specific states

16. Framework: Segment by Score, Action by Cell (SSAC)
Low High
Low
High
Support for
the candidate
Likelihood of
turnout
Persuade
Ignore
actions associated
with each cell
segment by scores
Help with
GOTV plans
Ask for $, get
to volunteer

17. 4. Ways to Deploy Models*
*This section is adapted from Robert L. Grossman, The Strategy and Practice of Analytics, 2020, to appear.

18. Exploratory Data Analysis
Get and
clean the data
Build model in
dev/modeling environment
Initial deployment
Use champion-challenger
methodology to improve
model
Analytic modeling
Analytic operations
Deploy
model
Retire model and deploy
improved model
Select analytic
problem &
approach
Scale up
deployment
ModelDev
AnalyticOps
Perf.
data
Data Scientists
Enterprise IT
Life cycle of a model
*Source: Robert L. Grossman, The Strategy and Practice of Analytics, 2020, to appear.

19. The Five Main Approaches (E3RW)
1. Embed analytics in databases
2. Scoring Engines (import/export
models)
3. Encapsulate models using
containers (and virtual machines)
4. Read a table of parameters
5. Wrap algo code or analytic system
(and perhaps create a service)
Approaches (E3RW)
• Use languages for analytics, such as
PMML and PFA & analytic engines
• Use languages for workflows, such
as CWL & workflow engines
• Use containers and container-
orchestration systems for
automating software deployment
and scale out, such as Docker &
Kubernetes
Techniques

20. 5. Summary

21. Summary
• It’s important to understand the differences between the data
scientists building the model, the enterprise IT team deploying the
model, and the product team ensuring the model provides value.
• We introduced three frameworks:
o Develop, Deploy and Extract (DDE) Framework
o The Scores-Actions-Measures (SAM) Framework
o The Segment by Score and Action by Cells (SSAC) Framework
• AnalyticOps: i) Deploying models and workflows with Analytic Engines
(PMML & PFA); ii) Analytic Containers with Software Deployment
Automation Environments (e.g. Kubernetes).
• We discussed the IBM Watson for Oncology case study.

22. Develop, Deploy and Extract (DDE) Framework
Scores
Actions
Measures of
the Responses
Estimate the value provided
by the model
Find “hooks” and strategies to
increase the value
Gain consensus on the
value and communicate it
Train the model
Validate the model
Package the model
Develop the model Deploy the model Extract value from the Model

23. Questions
Additional information about some of the topics discussed here can be found in my book: Developing an AI
Strategy: a Primer, Open Data Press, 2020, available online at analyticstrategy.com

24. For more information:
analyticstrategy.com
Robert L. Grossman
rgrossman.com
@BobGrossman
Linkedin: robertgrossman
ctds.uchicago.edu

25. Abstract: There is a lot of information and best practices available so data scientists
can build analytic models, but much less about how analytic models can best be
integrated into a company's products, services or operations, which we call analytic
operations. We describe three frameworks so that a company or organization can
improve its analytic operations and explain the frameworks using case studies.
About RLG: Robert L. Grossman is a Partner at Analytic Strategy Partners LLC, which
he founded in 2016. From 2002-2015, he was the Founder and Managing Partner at
Open Data Group, which built and deployed predictive models over big data for
Fortune 500 companies. He is also the Frederick H. Rawson Distinguished Service
Professor of Medicine and Computer Science and the Jim and Karen Frank Director
of the Center for Translational Data Science (CTDS) at the University of Chicago.