Solve User Problems: Data Architecture for Humans

Let’s solve user problems
(data architecture for humans)
March, 2021
Mark Madsen - @markmadsen - https://www.linkedin.com/in/markmadsen/

Where I am in my career
(and number of
mistakes I make)

© Third Nature Inc.
This talk will not be about best practices
Best practice in the early
market is usual a euphemism
for “workaround”
What the innovator did may
not be right, it may just be
not wrong
What the analyst firms call
best practice is often better
described as survival bias

There’s a difference between having no past and actively ignoring it

Copyright Third Nature, Inc.
A HISTORY OF REINVENTION

"Those who cannot remember
the past are condemned to
repeat it.”
George Santayana
If there’s one lesson we can take from history, It’s that nobody
learns any lessons from history.

Online
Realtime
For decision making
Today we’ll call it streaming

Technology patterns
New “Data Bases” ™
Storage virtualization
Separation of storage and
compute

Technology patterns
compute
What year was it?

Technology patterns
compute
Welcome to 1975

BETTER
is
New
Our core beliefs in software are based
on this. Progress is not a promise.

BETTER ?
is
New
This is fundamentally a belief in leading
with technology to solve problems…

Technology Adoption
Some people can’t resist getting the
next new thing because it’s new.
Many IT organizations are like this,
promoting a solution and hunting for
the problem that matches it.
Better to ask “What is the problem
for which this technology is the
answer?”

The solution to a puppy problem is not to add more puppies

Marketing and case studies: what people say vs reality
Beware the case study,
unless they talk about it
from first hand experience,
in production, and say what
did not work. Which should
be most of it. Most cases and
vendor testimonials are:
▪ Aspirational
▪ Immature
▪ Apply to 10 companies
world wide

Design tip:
Be skeptical about
anything you hear
regarding new data
platform technology
• Optimism
• Ignorance
• Lacking info on
“what it does
poorly”, which you
know very well about
your existing vendors

Be skeptical because technology
has a tendency to solve a problem
with a problem.
Solve scalability with brute
force parallelism. Now you
have an availability problem.
Solve that with redundant
copies. Now you have a
consistency problem…

THE EVOLUTION OF
ORGANIZATIONAL DATA USE

History: This is how BI was done through the 80s
First there were files and reporting programs.
Application files feed through a data processing pipeline to generate an
output file. The file is used by a report formatter for print/screen. Files are
largely single-purpose use.
Every report is a program written by a developer.
Data pipeline
code

History: This is how BI ended the 80s
The inevitable situation was...
Data pipeline
code

History: This is how we started the 90s
Collect data in a database. Queries replaced a LOT of application code
because much was just joins. We learned about “dead code”
SQL
SQL
SQL
SQL
SQL

Pragmatism and Data
Lessons learned during the ad-
hoc SQL era of the DW market:
When the technology is awkward for
the users, the users will stop trying
to use it.
Even “simple” schemas weren’t
enough for anyone other than
analysts and their Brio…
Led to the evolution of
metadata-driven SQL-
generating BI tools, ETL tools.

BI evolved to hiding query generation for end users
With more regular schema models, in particular dimensional models that
didn’t contain cyclic join paths, it was possible to automate SQL generation
via semantic mapping layers created by analysts.
We developed data pipeline building tools (aka ETL).
Query via business terms made BI usable by non-technical people.
ETL
SQL
Life got much easier…for a while

Today’s model: Lake + data engineers, looks familiar…
The Lake with data pipelines to files or Hive tables is
exactly the same pattern as the COBOL batch
Pipeline
code
We already know that people don’t scale. Don’t do this

DESIGN AND COMPLEXITY TODAY

"Always design a thing by considering it in its next larger
context - a chair in a room, a room in a house, a house in
an environment, an environment in a city plan." – Eliel
Saarinen

Order Entry
Order
Database
Customer
Service
Integration
Program
Inventory
Database
Distribution
Integration
Program
Receivables
Database
Accounts
Receivable
Data
Warehouse
Analysts &
users
This is the simplistic view people have of IT, if they see
even this level of detail

Real complexity is based on communication, which is data flows
Internal 3rd party & custom applications, event streams, logs, external & SaaS
applications, 3rd party datasets… – this is the reality

Monthly
Production plans
Weekly pre-
orders for
bulk cheese
Availability
confirmation
and location
In store system
Store
Stock
Management
Store EPOS
data
Category
Supervis
or
Stock
adjustments/
order
interventions
Order
adjustment
Stock/order
interventions
*
*
Orders
(based on 6
day
forecast)
Dallas
Distrib Centre
WMS
Picking/load
teams
Pos/Pick
lists/Load
sheets
Confirmed
Deliveries/
Confirmed
picks +
loads
Farmers
Milk intake/
silos Cheese plant
Plant
Processor
In-house Cheese
store
Contract Cheese
store
Processor
Packing plant
Processor
National
Distribution
Centre
Retailer
RDC
Retailer Stores
(550)
Retailer HQ
Consolidated
Demand
Ordering
Processor NDC
Customer
Services
Daily order -
SKU/Depot/
Vol
Sent @ 12.30-13:00
Delivery
orders
Processor HQ
Sales
Team/
Account
Manager
Processor HQ
Forecasting
Team
Processor HQ
Bulk Planning
Team
Cheese plant
Planner/Stock
office
Processor HQ
Milk
Purchasing
Team
Cheese plant
Transport
Manager
Actual
daily
delivery
figures
Daily
collection
planning
Weekly order for delivery to
Packing plant
Daily &
weekly Call-
off
Daily Call-off
15/day
22 pallet loads 15/day
A80
Shortages/
Allocation
instructions
Annual
Buying plan
Milk Availability
Forecast
Annual
prediction
of milk
production
Shortages/
Allocation
instructions
Daily milk
intake
Weekly milk
shortages
shortages
Spot mkt or
Processor
ingredients
Packing plant
Planning
Team
Processor HQ
JBA Invoicing
and
Sales Monitor
FGI and Last 5
weeks sales
Expedite
Changes
to existing
forecast -
exceptions
Retailer HQ
Retailer Buyer
Meeting
every 6
weeks
Packing plant
Cheese
ordering
10 day stock
plan
On line
stock info
7 day order
plan for bulk
cheese
Arrange
daily
delivery
schedule
Emergency
call-off
Daily
optimisation
of loads
Service
Monitor
Despatch and
delivery
confirmations
Processor NDC
Transport
Planning
Transport
Plan
Processor NDC
Inventory
Monitoring
Stock and
delivery
monitoring
Processor NDC
Warehouse
management
syatem
Operation
Instructions
Key
Shaded Boxes = Product flow system
Un-shaded boxes = Information flow system
Retailer
Cheese Processor
Farms
Schedule
weekly &
Daily
10 Day
plan(wed) and
daily plan
15/day
Changes
to existing
forecast -
exceptions
Stock
availability
Monthly
review
Annual
f/cast
Source: IGD Food Chain Centre, February 2008
All companies operate in the context of an industry. The external data
interchanges and market signals are today as important as the internal
data, for both strategic and operational decision making.
Gray = companies in value chain
Red = information flows and systems

Monthly
Production plans
Weekly pre-
orders for
bulk cheese
Availability
confirmation
and location
In store system
Store
Stock
Management
Store EPOS
data
Category
Supervis
or
Stock
adjustments/
order
interventions
Order
adjustment
Stock/order
interventions
*
*
Orders
(based on 6
day
forecast)
Dallas
Distrib Centre
WMS
Picking/load
teams
Pos/Pick
lists/Load
sheets
Confirmed
Deliveries/
Confirmed
picks +
loads
Farmers
Milk intake/
silos Cheese plant
Plant
Processor
In-house Cheese
store
Contract Cheese
store
Processor
Packing plant
Processor
National
Distribution
Centre
Retailer
RDC
Retailer Stores
(550)
Retailer HQ
Consolidated
Demand
Ordering
Processor NDC
Customer
Services
Daily order -
SKU/Depot/
Vol
Sent @ 12.30-13:00
Delivery
orders
Processor HQ
Sales
Team/
Account
Manager
Processor HQ
Forecasting
Team
Processor HQ
Bulk Planning
Team
Cheese plant
Planner/Stock
office
Processor HQ
Milk
Purchasing
Team
Cheese plant
Transport
Manager
Actual
daily
delivery
figures
Daily
collection
planning
Weekly order for delivery to
Packing plant
Daily &
weekly Call-
off
Daily Call-off
15/day
22 pallet loads 15/day
A80
Shortages/
Allocation
instructions
Annual
Buying plan
Milk Availability
Forecast
Annual
prediction
of milk
production
Shortages/
Allocation
instructions
Daily milk
intake
Weekly milk
shortages
shortages
Spot mkt or
Processor
ingredients
Packing plant
Planning
Team
Processor HQ
JBA Invoicing
and
Sales Monitor
FGI and Last 5
weeks sales
Expedite
Changes
to existing
forecast -
exceptions
Retailer HQ
Retailer Buyer
Meeting
every 6
weeks
Packing plant
Cheese
ordering
10 day stock
plan
On line
stock info
7 day order
plan for bulk
cheese
Arrange
daily
delivery
schedule
Emergency
call-off
Daily
optimisation
of loads
Service
Monitor
Despatch and
delivery
confirmations
Processor NDC
Transport
Planning
Transport
Plan
Processor NDC
Inventory
Monitoring
Stock and
delivery
monitoring
Processor NDC
Warehouse
management
syatem
Operation
Instructions
Key
Shaded Boxes = Product flow system
Un-shaded boxes = Information flow system
Retailer
Cheese Processor
Farms
Schedule
weekly &
Daily
10 Day
plan(wed) and
daily plan
15/day
Changes
to existing
forecast -
exceptions
Stock
availability
Monthly
review
Annual
f/cast
The real data context of the
organization that is assembled
by the data platforms is
subsets of all of these systems.
The complexity of a DW is a
function of the complexity of
the organization and all the
integration points.
There’s more to it than just the
systems and technologies…
Data
Warehouse

Data is transformed, cleaned, integrated, and new data is
derived. This adds a level of temporal and semantic
complexity to data management, and it’s always hidden.
Machine learning won’t “solve” data integration. It will
help in some areas, mostly with augmenting simpler tasks.
Data flows – the dark matter of your architecture diagrams

Data Complexity, one application
Meanwhile, people complain when a data model looks like this

This is a map of one
organization’s analytic data,
showing the dataset
complexity inherent in a mid-
sized organization.
Different views of
data complexity

Data complexity is not just
based on the number of
datasets, or the number of
tables.
It is based on the number of
connections. This is an order
of magnitude higher than
number of objects.
Organizational complexity
drives communication
complexity drives data
complexity.
Different views of
data complexity

This view is only
showing connections
between objects in
data sets based on
data relationships. All
these connections are
joins you must take
care with in a well
managed platform.

Different views of
data complexity
A reverse gravity view, showing
the mass of reused / replicated
information at the center and
the nodes where large
interchanges occur.
These different views show
how complex an organization’s
data really is, rather than the
abstract list of sources and
terabytes stored.
This is why managing data is a
difficult job

Different views –
data and use
The value of data is tied to its
use. This shows relationships
between people and data used.
This and the prior diagram
show an important point: 70%
of the data is used and reused
constantly. 30% of the data is
used by one or a few people,
often new data with
undetermined value.
This information can be used to
determine where and how you
should spend your limited
resources and money.

Connections and uses of data are scale free networks
The connections in the data have an exponential distribution. Each new copy
of data (or derived set, subset, aggregate) adds N-1 possible connections.
You can’t manage all the data. Which data do you spend time on?
Nodes (tables)
Number of
connections
Used often: 70% of
the connections tie to
a small set of data,
the core of reuse.
Centrally manage this
Used seldom: 10% of the
connections go to a large
number of objects (new, low
value, narrow). Locally
curate this

The reality of data availability is that it can only be a subset
The rest of the data
is still here…
There will always be more data
available than ability to analyze
it. Some judgement must be
applied to sort the more from
the less important

Loosely
managed
data
User
managed
data
In an expanded ecosystem of data, curation processes are
needed to address quality, definition and structure
Closely
managed
data
High quality,
well-known
Directional
quality
Unknown / low
quality
Curation is directly attached to your data architecture...

Data curation is an undeveloped practice
The problem with so many sources,
types, formats and latencies of data
is that it is now impossible to create
in advance one model for all of it.
Data modeling is about the inside of
a dataset. Curation is about the
entire dataset.
It isn’t development. It’s about:
creating, labeling, organizing, finding,
navigating, archiving.
Data curation, rather than data
modeling, is becoming the most
important data management
practice.

The real purpose of this work is not to help IT be more
productive. IT exists to help users be more productive
Starting with technology is like getting excited by a new chisel.

Today’s market “solution”:
Replace the data
warehouse with the data
lake and self-service*
*Picture for illustrative purposes
only, no warranty express or
implied, actual system you
receive may vary.
By a lot.

Today’s market solution:
the Data Lake to replace
the Data Warehouse
Data hoarding is not a data
management strategy

The solution to one technology problem is another technology
Buy a catalog!
Just add more
technology to solve
your non-technical
problem.
Now that you know
what data is there -
how do you find it?
How do you get it?

Practices need to catch up to technologies
A catalog is a
useful, necessary
component.
It is useless
without
organizing
principles and
practices.
AKA data
curation and
data architecture

So who maintains the catalog now?
IT is already
viewed as a
bottleneck.
Many
organizations do
not have full-
time data
administrators,
and the DW
team is already
overtaxed.

Why not let the user drive?

Have you ever looked at user generated taxonomies?
Users also have a job to do and won’t welcome more administrative work

Developers
think of self-
service as
data access
– the user
must be
self-reliant

Users think
of self-
service in
terms of a
finished data
product

<Problem>
creates
<Opportunity for new technology solution>
creates
<Different problem>
repeat
Seems
familiar…

WHAT CAN WE DO ABOUT THESE
PROBLEMS?

Value is not in the product, it’s in the practice
The poor carpenter blames his tools

You are a designer. You need to think like one.
“Everyone designs who devises
courses of action aimed at changing
existing situations into preferred
ones.” ~ Herbert Simon

We seldom think
systemically. It’s time to
start (again)

Often we got here because of bad policy, not technology – people would
rather work around their data teams than work with them on data initiatives
http://akvkbi.blogspot.com/2017/06/dwh-development-related-survey-results.html

Design tip: any time you deny a behavior or a request, ask yourself “how
will they do this on their own? What do they do instead?”
Bad policy causes more problems than bad technology

Shape the architecture for the people, don’t shape try to shape people

Data should be governed by policy, e.g. zoning
http://welcometocup.org/file_columns/0000/0530/cup-whatiszoning-guidebook.pdf

We need to do today what we were doing 30 years ago
We spent time then to understand the users, what they wanted,
the needs, and found ways to justify the work to meet those needs.
We don’t
do enough
of this
We over-
emphasize
this

Technology is a tool
Tools enable you to build things
You build things for people
So start here

The primary focus should be on goals, specifically of the users
“The engineer, and more generally
the designer, is concerned with how
things ought to be - how they ought
to be in order to attain goals, and to
function.” ~ Herbert Simon

What organizations say they want
Time to value
Ability to do new things more
easily, aka innovate
• with data
• with technology
Efficiency, aka reduce costs

What organizations say sounds like “do more with less”
Time to value
Ability to do new things more
easily, aka innovate
• with data
• with technology
Efficiency, aka reduce costs
I end up with more questions:
• TTV for whom?
• TTV for what?
• New thing for whom?
• One time or recurring (cost
or TTV)?
• TTV as latency of
throughput?
• Local cost or global cost?
• More efficient vs less
flexible?
This is manager-speak – we need to talk about users

The questions for the
data ecosystem from an
architect’s perspective
What people?
What goals?
What uses?
What time frames?

Mapping User Need: don’t work on assumptions
Work-as-Prescribed
Work-as-Imagined
Work-as-Done
Work-as-Disclosed
We are oriented here. But people encounter obstacles to their work. They
create solutions. The solutions become part of the work, how the work is
done. Work as-done diverges from the official definition of the work as-
prescribed or as-disclosed. Most tech startups have no real idea about the
work – they believe they are solving technical problems and the user is IT.
We need to
focus here
https://safetydifferently.com/the-varieties-of-human-work/

Starting with technology is starting in the solution space, not the problem space
https://indiyoung.com/about-problem-space/

Analysis and data science workflows are generally poorly understood
An analyst trying to answer a question has highs and lows along their workflow.
The environment is defined by independent, often mismatched tools, some fit for
purpose and others not, with no single product capable of meeting their needs.
Each usage model has several of these maps tied to different roles
Where is
data? Can I
access new
data?
Why does IT
have to be
involved?
Green = solved
Yellow = gap, poss opportunity
Red = obstacle, opportunity
Why can’t I
store data
I’m working
on?
How do I link
new data to
existing
data?
How do I
share
information
with others?

User goals: more than accessing the data
Explore and
Understand
Inform
and
Explain
Convince
and
Decice
Deliver
Process

The real design criteria:
context and point of use
Information use is diverse and varies
based on context:
▪ Get a quick answer
▪ Solve a one-off problem
▪ Analyze causes
▪ Do experiments
▪ Make repetitive decisions
▪ Use data in routine processes
▪ Make complex decisions
▪ Choose a course of action
▪ Convince others to take action
One size doesn’t fit all.

Data architecture requires understanding data use so we
can build the right infrastructure
Monitor
Analyze
Exceptions
Analyze
Causes
Decide Act
No problem No idea Do nothing
Understanding the details of uses, workflows, tasks, and
activities allows us to look at the higher organizational level again

This is part of a larger system. Feedback loops exist and
operate at different frequencies.
Collect
new data
Monitor
Analyze
Exceptions
Analyze
Causes
Decide Act
Act on the process
Act within the process

Data platforms are the most complex in the organization, far
more complex than any web application or ERP system.

Manage your data
(or it will manage you)
Data management is where
developers are weakest.
Modern engineering practices
are where data management is
weakest.
Users care about their tasks.
You need to bridge these
groups and practices in the
organization if you want to do
meaningful work with data.

Mark spent most of the past 25 years working in
the analytics field, starting in AI at the University
of Pittsburgh and autonomous robotics at
Carnegie Mellon University before moving into
technology management. Today he is a Fellow in
the Technology & Innovation Office at Teradata.
Previously, he was president of Third Nature, an
advisory firm focused on services for analytics and
technology strategy, and product design.
Mark is an award-winning author, architect and
CTO who has received awards for his work from
the American Productivity & Quality Center,
Smithsonian Institute, and industry associations.
He is an international speaker, and chairs several
conferences and program committees. You can
find him on LinkedIn at
https://www.linkedin.com/in/markmadsen
About the Presenter

Further Reading
Thinking in Systems, Donella Meadows
An Introduction to Systems Thinking, Gerald Weinberg
Contextual Design, Beyer & Holtzblatt
Badass: making Users Awesome, Kathy Sierra
Information Design, Jacobsen
Data: A Guide to Humans, Phil Harvey
In Search of Certainty, Burgess
https://indiyoung.com/about-problem-space/
http://welcometocup.org/file_columns/0000/0530/cup-whatiszoning-guidebook.pdf

CC Image Attributions
Thanks to the people who supplied the creative commons licensed images used in this presentation:
well town hall.jpg - http://flickr.com/photos/tuinkabouter/1135560976/
seattle library 1 - http://www.flickr.com/photos/thomashawk/2671536366/
chicken_head2.jpg - http://www.flickr.com/photos/coycholla/4901760905
egg_face1.jpg - http://www.flickr.com/photos/sally_monster/3228248457
indonesian angry mask phone - Erik De Castro Reuters.jpg

Solve User Problems: Data Architecture for Humans

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Solve User Problems: Data Architecture for Humans

Ähnlich wie Solve User Problems: Data Architecture for Humans (20)

Mehr von mark madsen

Mehr von mark madsen (14)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Solve User Problems: Data Architecture for Humans