Data Catalog as the Platform for Data Intelligence

The Data Catalog as the Platform
for Data Intelligence
Satyen Sangani
August 18, 2020
Twitter: @satyx

2 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Topics
• The drive for data culture
• The search for (data) intelligence
• The state of the data catalog
• The data catalog as a platform for data intelligence

3 | Alation Confidential & Proprietary.The Catalog is the Platform™
The World is Changing Fast...

...And Your Organization is Changing Too.
Used to have too little
data, now too much
Capture tribal knowledge,
enable collaboration
Comply with changing
laws and regulations
Data Explosion
• 63 ZB in 2021 (volume)
• 300+ DBMSs (variety)
• IoT and Edge (velocity)
Evolving Laws
• Privacy: GDPR, CCPA, HIPAA
• Risk: BCBS 239
• Industry-specific: FedRAMP
Changing Workforce
• Turnover, restructuring
• New SMEs, lost stewards
• Remote work (WFH)

In a Changing World, Everyone is Making Decisions
People are constantly changing
Information doesn’t flow fast enough
Expertise travels even more slowly than information

How Do You Want Them to Make Decisions?
Simon Says? Their Gut?
Based on data
and evidence?

Characteristics of a Data Culture
Hypothesis/
Test Oriented
Knowledge Systematically
Collected/Documented
Methodological
Transparency
Distributed Evidence Based Non-Hierarchical

Listen like you’re wrong.
Measure ourselves through customer impact.
Move the ball.
Build for the long term.
Alation’s Values Intended to Drive a Data Culture

Data Culture Is an Imperative
The Benefits
are Compelling
• Insights-driven
businesses on track to
earn $1.8T by 2021
• 7x more likely to
increase revenue as
result of big data
• 2.8x more likely to have
double-digit growth
The Stakes are High
& Mistakes Costly
• Fines of 20M Euros or
4% of annual revenue
(GDPR)
• $5M fines for misleading
financial reports (SOX)
• Incorrect decisions
made on bad data or
bad models
Data Culture is a Priority
Forrester: Insights-Driven Businesses Set the Pace
for Global Growth. October 2018
2018 Gartner Fourth Annual Chief Data Officer Study

The Alation Vision
To empower a curious and rational world
Data Culture
Data
Literacy
Enable proper
interpretation & analysis
Data
Governance
Take responsibility
& authority
Data Search
& Discovery
Find &
understand

Topics

Data Intelligence
Intelligence (as defined by Google):
in •tel •li•gence
/inˈteləjəns/
The ability to acquire and apply knowledge and skills.
Data Intelligence (as defined by me):
da•ta in •tel •li•gence
/ˈdādə/ /inˈteləjəns/
The organizational ability to acquire and apply
knowledge, skills, and the resulting competitive
advantage by leveraging data & analytics.

We’ve Been Searching for a While...

The “Truth” Depends on Your Point of View

Academia Gets This

Intelligence Comes From
The ability to test
multiple scenarios
Documented prose,
not just reports
Traceable claims,
data and reports
Having the right info
at the right time
Links to
evidence
Open
Access

You don’t need a single source of truth...
...you need a single system of reference.

Topics

What is a Data Catalog?
• A repository of metadata on information sources across
an organization
- Search & discovery
- Data governance & curation
- Collaboration & analysis
• Catalogs a broad range of information assets
- Data sets, tables, articles, reports, queries, visualizations, conversations
• Includes common functionality such as:
Answers these
core questions:
How to find information?
Can it be used?
Should it be used?
How should it be used?
Business Glossary Lineage Catalog Pages Search

Alation Commercialized the Data Catalog
“Alation started the ML Data Catalog trend.”
-Forrester Wave, Machine Learning Data Catalogs, 2018
• Founded 8 years ago in 2012
• Born out of my experience
at Oracle
• Envisioned a collaborative catalog
• Born in the era of “big” data

But, Metadata Management and Data Catalogs
Have Been Around for Years.
CatalogMetadata Management
Used by: Information Suppliers
Model: System of Record
Inspiration: Chart of Accounts/
Inventory Management
Intelligence: Transactional: Based on
Human Assertion
Coverage: Systems with “physical” data
Used by: Information Consumers
Model: System of Engagement
Inspiration: Collaborative/Social
Intelligence: Automated: Based on Social
Behaviors
Coverage: add. Reports, Algorithms,
ETLs, Streams

Living Catalogs Enable You to Quickly Self-Serve
is a trusted catalog that
helps you quickly find a
professional.
is trusted catalog that helps
you quickly learn about
products sold on the web.
is a trusted catalog
that helps you
quickly surf the web.
You used to have a
rolodex and a network.
Experts in Data Management New OLED TV What is PageRank?
You used to go
to a store.
You used to get help
from a librarian.

Amazon Reinvented the Supply Chain
They did this because they understood demand

Topics

Early Uses of Data Catalogs Were Search/Discovery
Data
Scientist
Business
Analyst

The Importance of Machine Learning
Plot the dots
Connect
Make Connections
Compute
Draw Conclusions
Conclude
Sue
Bob
Query #2
Query #3
Query #1
Top Users: Bob and Sue
Popularity: II

Collaborative
data catalog
Virtuous Cycle of Adoption
(More) people
use data catalog
Data catalog
becomes more
valuable

The “Truth” Depends on Your Point of View

The Data Catalog Is that Source of Reference
FIND | UNDERSTAND | TRUST

The Catalog as Platform for Data Intelligence

“Alation helps the business uphold our
governance standards so anybody coming in
can get their data quickly and efficiently,
and make the right decisions.”
AMY KEELTY
Information Strategy & Governance Director
American Family Insurance
Data Governance Naturally Followed

Data Governance is Evolving from Centralized
to Distributed Responsibility
Guided Community Driven
Social Collaboration
Centralized
Command & Control
Centralized
Command & Control
Guided Community Driven
Social Collaboration

A Different, Value-Driven Approach
People-First
Exercised at
Point of Use Collaborative Intelligent
Connects policy to
action in users’
day-to-day activities
Guides people’s
behavior by discovering
and formalizing their
relationship with data
Automation powered
by AI and ML
Crowdsourcing and
community-driven
Iterative,
Agile
Incremental, not a
big-bang approach
Active Data Governance

The Accelerating Pace of Cloud Data Migration
These systems are increasingly headed to the cloud,
and by 2022, 75% of all databases will be deployed
or migrated to a cloud platform, the report said.”
1990s:
• Spreadsheets
2005:
• Data Lakes
2000s:
• Data Warehouse
2010:
• Cloud Hosted Data Warehouses
• Cloud Managed Data Lakes
”

What Data is Most-Used?
1. In this example, of the 145
tables, only 13 have ever been
queried by Database users
2. Reduce storage & compute
costs through migrating what
users care about
3. Move related Datasets by
understanding commonly
joined tables

Catalogs Provide Intelligence...
The ability to test
multiple scenarios
Documented prose,
not just reports
Traceable claims,
data and reports
Having the right info
at the right time
Links to
evidence
Open
Access

Where Do We Go Next?
More intelligence about:
Semantics Third Party Data
How data can be
merged/joined
Algorithms

Scale Data Discovery & Automate Data Tagging
Delivers CCPA &
GDPR templates
to tag data sets
It’s hard to identify
and classify private
or sensitive data
Solution
Automate data discovery
and sensitive data tagging
Gathers technical
metadata across
enterprise data
sources
1
2
3 Understands
unique individuals’
data across disparate
data sources

Accelerate Governance & Stewardship of Sensitive Data
Classifies sensitive
data and correlates
it to the related
datasets
We can’t easily flag
relevant policies when
users interact with data
Solution
Accelerate governance
of sensitive data
Ingests metadata
and is the single
catalog to find
trusted data
1
2
3 Groups the data
and uses a template
to assign policies to
that group to scale
policy notification
across dataset

Ensure Ongoing Compliance
Scales & automates
discovery and tagging
by connecting to,
profiling and tagging
relational, non relational
and unstructured data
It’s hard to maintain
as business and
regulations change
Solution
Single point of reference
for policy & usage
guidelines at the point
of consumption
Surfaces metadata and
usage patterns for insights
from various data sources
1
2
3 Single interface for users to
find policies along with data
to break down silos

How Alation and BigID Address the Challenges
Use - Data workers
access data with the
associated policies
5
Classify - BigID profiles
data and classifies data into
a personal and sensitive
category with associated
policy templates
2
Tag – Alation imports the data
classification from BigID and
tags Alation catalog pages to
associate the metadata with
the correct policies
31
Ingest - Alation ingests
metadata from all
enterprise data sources
for stewards to tag for
contextual classification
Scale - BigID uses Alation
documentation to
automate data association
across data landscape
4

Satyen Sangani
@satyx

Data Catalog as the Platform for Data Intelligence

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Data Catalog as the Platform for Data Intelligence

Ähnlich wie Data Catalog as the Platform for Data Intelligence (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Data Catalog as the Platform for Data Intelligence