DataEd Slides: Leveraging Data Management Technologies

Copyright 2020 by Data Blueprint Slide # 1Peter Aiken, PhD
(Unlocking Business Value)
Peter Aiken, PhD
Leveraging
Data
Management
Technologies
• DAMA International President 2009-2013 / 2018
• DAMA International Achievement Award 2001
(with Dr. E. F. "Ted" Codd
• DAMA International Community Award 2005
• I've been doing this a long time
• My work is recognized as useful
• Associate Professor of IS (vcu.edu)
• Founder, Data Blueprint (datablueprint.com)
• DAMA International (dama.org)
• CDO Society (iscdo.org)
• 11 books and dozens of articles
• Experienced w/ 500+ data
management practices worldwide
• Multi-year immersions
– US DoD (DISA/Army/Marines/DLA)
– Nokia
– Deutsche Bank
– Wells Fargo
– Walmart … PETER AIKEN WITH JUANITA BILLINGS
FOREWORD BY JOHN BOTTEGA
MONETIZING
DATA MANAGEMENT
Unlocking the Value in Your Organization’s
Most Important Asset.
2Copyright 2020 by Data Blueprint Slide #
Peter Aiken, Ph.D.

Trifacta Overview
© 2020 Trifacta | Proprietary and Confidential 1
Contact Info | Trifacta.com

© 2020 Trifacta | Proprietary and Confidential
USE CASES
Data Onboarding
Data Science/ML
Reporting & Analytics
DATA PLATFORMS
Databases
Log Files
Spreadsheets
IoT Sensors
Apps
“It’s impossible to overstress this: 80% of the work in any data project is in cleaning the data.”
— DJ Patil, Former Chief Data Scientist of
the United States
DATA PIPELINE
Discovering
Structuring
CleansingEnriching
Validating
The 80% Problem Is Well Understood

https://www.trifacta.com/blog/introducing-data-school/

Thank You
Contact Info | Trifacta.com
© 2020 Trifacta | Proprietary and Confidential 7

Copyright 2020 by Data Blueprint Slide # X
By the end of this session, you should have a better
understanding of data management technologies in
terms of:
• Technology Considerations
• Data Technology Architecture
• CASE Tools
• Repositories
• Profiling/Discovery Tools
• Data Quality Engineering Tools
• Data Life Cycle
• Other Technologies:
– Servers, EII Technologies, Portals, Conversion Tools
Approaching Data Management Technologies
Blind Persons and the Elephant
http://www.dailymirror.lk/print/opinion/editorial-we-need-to-become-channels-of-peace/172-27164
It is like a fan!
It is like a snake!
It is like a wall!
It is like a rope!
It is like a tree!

Unrefined
data management
definition
Sources
Uses
Data Management
More refined
data management
definition
Sources
ReuseData Management➜ ➜

Data Governance
Data Assets/Ethical Framework
Sources
➜ Use
➜Reuse
Better still data management definition
➜
Standard data
Data supply
Data literacy
Making a Better Data Sandwich
Data literacy
Standard data
Data supply

Standard data
Data supply
Data literacy
Quality engineering/
architecture work products
do not happen accidentally!
Standard data
Data supply
Data literacy
This cannot happen without engineering and architecture!

Technologies by themselves, are a One Legged Stool
Success Requires a 3-Legged Stool
People
Process
Technology

People
Process
Technology

Supply/demand for data talent
https://www.logianalytics.com/bi-trends/3-keys-understanding-data/
Growth of Data vs. Growth of Data Analysts
• Stored data accumulating at
28% annual growth rate
• Data analysts in workforce
growing at 5.7% growth rate
R. Buckminster Fuller

https://en.wikipedia.org/wiki/Moore%27s_law#/media/File:Moore%27s_Law_Transistor_Count_1971-2016.png
Postpone technology investments
as long as possible
The hardest part of
requirements is not
doing design
Vendor Hype
• CIOs/CDOs feel pressure
• Vendor/project promise auditing
• No understanding of hype curve

Who wrote this … ?
• In considering any new subject,
• there is frequently a tendency
first to overrate what we find to
be already interesting or
remarkable, and
• secondly - by a sort of natural
reaction - to undervalue the true
state of the case.
– Lady Augusta Ada King,
(1815 – 1852)
Countess of Lovelace
– (aka) Ada Lovelace,
daughter of Lord Byron
– Publisher of the first
computing program

http://www.gartner.com/technology/research/methodologies/hype-cycle.jsp
Technology Trigger: A potential technology breakthrough kicks things off. Early proof-of-concept stories and media interest
trigger significant publicity. Often no usable products exist and commercial viability is unproven.
Trough of Disillusionment: Interest wanes as experiments and implementations fail to deliver. Producers of the
technology shake out or fail. Investments continue only if the surviving providers improve their products to the
satisfaction of early adopters.
Peak of Inflated Expectations: Early publicity produces a number of
success stories—often accompanied by scores of failures. Some
companies take action; many do not.
Slope of Enlightenment: More instances of how the technology can benefit the
enterprise start to crystallize and become more widely understood. Second- and third-
generation products appear from technology providers. More enterprises fund pilots;
conservative companies remain cautious.
Plateau of Productivity: Mainstream adoption starts to
take off. Criteria for assessing provider viability are more
clearly defined. The technology’s broad market
applicability and relevance are clearly paying off.
Gartner Five-phase Hype Cycle
Hype Cycle for Data Management

Hype Cycle for Information Governance and Master Data Management
Hype Cycle for Analytics and Business Intelligence

terms of:
• CASE Tools
• Repositories
• Data Life Cycle
Data Management Technologies
• Managing data technology should follow the same
principles and standards for managing any
technology
• Leading reference model for technology
management is the Information Technology
Infrastructure Library (ITIL):
http://www.itil-officialsite.com/home/home.asp
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International

Understanding Data Technology Requirements
Need to understand:
• How the technology works
• How it provides value in the
context of a particular business
• Requirements of a data technology before determining what
technical solution to choose for a particular situation
Suggested questions:
• What problem does this data technology mean to solve?
• What sets this data technology apart from others?
• Are there specific hardware/software/operating systems/storage/
network/connectivity requirements?
• Does this technology include data security functionality?
Defining Data Technology Architecture
• Data technology is part of the overall technology architecture
• It is also often considered part of the enterprise’s data architecture
• Data technology architecture addresses 3 questions:
– What technologies are
standard/required/preferred/acceptable?
– Which technologies apply to which
purposes and circumstances?
– In a distributed environment, which
technologies exist where, and
how does data move from one node to another?

Definitions
• as opposed to
mobile device
management
• MDM is a discipline
or strategy
– "… where the business
and the IT organization
work together to
ensure the uniformity,
accuracy,
semantic persistence,
stewardship and
accountability
of the enterprise's
official, shared master
data."
• Sold as technology-
based solution
Master Data Architecture
• Technology-first
approaches often de-
emphasize the
people and process
components
• Successful MDM also
requires
– Governance/quality
– Process architecture

Tools and Methods Are Required!
terms of:
• CASE Tools
• Repositories
• Data Life Cycle

Computer-aided software engineering (CASE)
is the scientific application of a set of tools and
methods to a software system which is meant
to result in high-quality, defect-free, and
maintainable software products. It also refers
to methods for the development of information
systems together with automated tools that
can be used in the software development
process. Computer Aided Software/Systems Engineering Tools
• Scientific application of a set of tools and methods to a software system which is
meant to result in high-quality, defect free, and maintainable software products
• Refers to methods for the development of information systems together with
automated tools that can be used in the software development process
• CASE functions include analysis, design, and programming
Source: http://en.wikipedia.org/wiki/
CASE Tools
CASE-based Support
http://www.visible.com

CASE-based Support
CASE-based Support

CASE Tool Evolution
• Microsoft
– Excel
– Powerpoint
– Visio
• ERwin ER/Studio
• Rational Rose
• Open source
– It is never free. Even open-sourced
technology requires care and feeding
• List of CASE Tools
– http://www.unl.csi.cuny.edu/faqs/software-enginering/tools.html
Typical expense concept
[adapted from Huff "Elements of a Realistic
CASE Tool Adoption Budget" © 1992
Communications of the ACM]
• Sample budget for
implementing a $2500/seat
CASE technology can be $2.5
million over a 5-year period
$187K =
$2500/seat
× 75 seats
$360K = training
$500K = workstations
$150K= assessment costs
$910K = total initial investment
$150K = in-house support
$ 55K = hardware and software maintenance
$ 60K = ongoing training and misc.
$265K = annual additional investment
× 5 years
$1325K investment over 5 years
$187K =
$2500/seat
× 75 seats
$360K = training
× 5 years
$187K =
$2500/seat
× 75 seats
$360K = training
× 5 years

This includes:
• Senders
– flows from the CASE effort that
can inform the re-architecting
effort.
• Receivers
– flows from the project that can
inform the CASE effort.
• Senders and receivers
– some elements, such as
restructuring and reengineering,
are both senders and receivers.
CASE Tool: "Taxonomy"
A variety of
CASE-based
methods and
technologies can
access and
update the
metadata
metadata
Integration
Additional metadata uses
accessible via: web; portal;
XML; RDBMS
Everything must "fit" into one
CASE technology
Changing Model of CASE Tool Usage
Limited access
from outside
the CASE
technology
environment
CASE
tool-specific
methods
and
technologies
Limited additional
metadata use

terms of:
• CASE Tools
• Repositories
• Data Life Cycle
The Biggest Challenges to Data Management Practice

One Eighth of the Data Management Spend
88%
12%
Metadata
• Metadata management
is still a nascent
discipline that only
represents 12% of the
time spent in data
management
Repositories have been difficult to "sell"
21 September 1999
Michael Blechar, Lisa Wallace
Management Summary
Most executive and IS managers view an IT metadata repository as an
esoteric technology that is not directly related to the business.
However, as will be seen, an IT metadata repository can substantially
help IS organizations support the applications, which in turn support
the business. An IT metadata repository is a pre-built system and
reference database where the IS organizations can track and manage
the information about the applications and databases they build and
maintain; think of it as the inventory and change impact reporting
system for IS. These repositories track metadata such as the
descriptions of jobs, programs, modules, screens, data and
databases, and the interrelationships between them. Metadata differs
from the actual data being described. Metadata is information about
data. For example, the metadata descriptions in the repository tell one
that the field "customer number" appears in Databases A, B and F ...
[From gartner.com]

What tools do you use?
45%
23%
13%
9%
7%
2%
1% 1% 1% 1%
None HomeGrown Other CA Platinum Rochade Universal
Repository
DesignBank DWGuide InfoManager Interface
Metadata
Tool
• Many build their own
Repository Technologies in Use
Number Responding=181
• Almost 50% doesn't use
• The "traditional" players are low
numbers
Magic Quadrant for Metadata Management Solutions
https://www.gartner.com/document/3894971?ref=solrAll&refval=219836558&qid=de595a5685b6f86db0ec6

IBM's AD/Cycle Information Model
https://wiscorp.com/kwf_diagram.html

• "The repository" does not have to be an integrated solution
– it must be an easily integrateable solution
• Repository functionality (does not equal a) repository
– metadata must easily evolve to repository solution
• Multiple repositories are not necessarily bad
– as interim solutions, Excel has been working quite well
• Minimal functionality includes
• ability to create, read, update, delete, and evolve metadata items
• Remember the 1st law of data management
– In order to manage metadata, you need metadata repository functions
Implementing Metadata Repository Functionality
terms of:
• CASE Tools
• Repositories
• Data Life Cycle

Time Spent by Data Management Teams Across Disciplines
Data Discovery Technologies
• Data analysis software technologies deliver up to 10X
productivity over manual approaches
• Based on a powerful computing technology that allows data
engineers to quickly form candidate hypotheses with respect to
the existing data structures
• Hypotheses are then presented to the SMEs (both business and
technical) who confirm, refine, or deny them
• Allows existing data structures to be inferred at rate that is an
order of magnitude more effective than previous manual
approaches
• Pioneers include Evoke->CSI, Metagenix->Ascential->IBM,
Sypherlink
Profiling
Discovery
Analysis

How has this been done in the past?
Old
• Manually
• Brute force
• Repository dependent
• Quality indifferent
• Not repeatable
New
• Semi-automated
• Engineered
• Repository
independent
• Integrated quality
• Repeatable
• Currency
• Accuracy
Select an Attribute to
get a list of values
Double-click a value to
see rows with that value

Comparing Weekly Progress
Monday
Morning:
Model
preparation
Afternoon:
Model refinement/
validation session
Tuesday
Morning:
Model refinement/
validation session
Afternoon:
Model refinement/
validation session
Wednesday
Morning:
Model
preparation
Afternoon:
Model refinement/
validation session
Thursday
Morning:
Model refinement/
validation session
Afternoon:
Model refinement/
validation session
Friday
Morning:
Model
preparation
Afternoon:
Model refinement/
validation session
Monday
Morning:
Model
preparation
Afternoon:
Model
preparation
Tuesday
Morning:
Model
preparation
Afternoon:
Model refinement/
validation session
Wednesday
Morning:
Model
preparation
Afternoon:
Model
preparation
Thursday
Morning:
Model
preparation
Afternoon:
Model refinement/
validation session
Friday
Morning:
Model
preparation
Afternoon:
Model
preparation
Reactive
Proactive
Trifacta/Data Wrangling

terms of:
• CASE Tools
• Repositories
• Data Life Cycle
Data Quality Engineering Tools
• 4 categories of activities:
– Analysis
– Cleansing
– Enhancement
– Monitoring
• Principal tools:
– Data Profiling
– Parsing and Standardization
– Data Transformation
– Identity Resolution and Matching
– Enhancement
– Reporting

DQ Tools
1. Data Profiling
– Need to be able to distinguish between good and bad data before making any
improvements
– Data profiling is a set of algorithms for 2 purposes:
• Statistical analysis and assessment of the data quality values within a data set
• Exploring relationships that exist between value collections within and across data sets
DQ Tools
2. Parsing & Standardization
– Data parsing tools enable the definition of
patterns that feed into a rules engine used
to distinguish between valid and invalid
data values
– Actions are triggered upon matching a
specific pattern
– When an invalid pattern is recognized, the
application may attempt to transform the
invalid value into one that meets
expectations
https://www.youtube.com/watch?v=r9UhJxFT5rk

DQ Tool Pattern
DQ Tools
4. Identify Resolution & Matching
– Basic approaches to matching:
– Deterministic
• Relies on defined patterns and rules for
assigning weights and scores to determine
similarity
– Predictable
• Only as good as anticipations of the rules
developers
– Probabilistic
• Uses statistical techniques to assess
probabilities that pairs of records represent
the same entity
– Not reliant on rules
• Refined based on experience -> matchers
can improve precision as more data is
analyzed

DQ Tools
5. Enhancement
– A method for adding value to information by accumulating additional information
about a base set of entities and then merging all the sets of information to
provide a focused view
Examples:
– Time/date stamps
– Auditing information
– Contextual information
– Geographic information
– Demographic information
– Psychographic information
DQ Tools
6. Reporting
• Good reporting supports:
– Inspection and monitoring of conformance to data quality expectations
– Monitoring performance of data stewards conforming to data quality SLAs
– Workflow processing for data quality incidents
– Manual oversight of data cleansing and correction
• Associate report results w/:
– Data quality measurement
– Metrics
– Activity

terms of:
• CASE Tools
• Repositories
• Data Life Cycle
Data acquisition activities Data usage activitiesData storage
Traditional Data Life Cycle

Metadata
Creation
Data
Assessment
MetadataRefinement
DataRefinement
Data
Manipulation
DataCreation
Data
Utilization
Metadata
Structuring
Data Storage
DataLifeCycleModel
Metadata Data
Dimension Focus/Phase: Refinement Creation Structuring Creation Manipulation Refinement Utilization Assessment
Data
Architecture
Quality
Data architecture quality
is the focus of metadata
creation & refinement
efforts.
↵ ↵ ↵
Data Model
Quality
Data model quality is the
focus of metadata
refinement & structuring
efforts
↵ ↵ ↵
Data Value
Quality
Data value quality is the
focus of the data
creation, manipulation,
and refinements phases.
↵ ↵ ↵ ↵
Data
Representation
Quality
Data representation
quality is the focus of
data utilization phase.
↵ ↵
Dimensions Related to Phases
• Data architecture quality is the focus of metadata creation and refinement efforts.
• Data model quality is the focus of metadata structuring efforts
• Data value quality is the focus of the data creation, manipulation, and refinements phases.
• Data architecture and model quality are the focus of metadata refinement efforts.
• Data representation quality is the focus of data utilization and assessment phase.

terms of:
• CASE Tools
• Repositories
• Data Life Cycle
Other Technologies
Data Integration Definition:
• Pulling together and reconciling dispersed data for analytic
purposes that organizations have maintained in multiple,
heterogeneous systems. Data needs to be accessed and
extracted, moved and loaded, validated and cleaned,
standardized and transformed.
• Other tools include:
– Servers
– EII technologies
– Portals
– Conversion tools
Source: http://www.information-management.com

Portal Options
[Adapted from Terry Lanham Designing Innovative Enterprise Portals and Implementing Them Into Your Content Strategies Lockheed
Martin’s Compelling Case Study Web Content II: Leveraging Best-of-Breed Content Strategies - San Francisco, CA 23 January 2001]
Legacy Systems Transformed Into Web-services Accessed Through a Portal
Organizational Portal
Saturday, April 6, 2019 - All systems operational!
Organizational News
• Organizational Early News • Industry News
• Press Releases • Newsletters
Organizational IT
• Service Desk
• Settings
Email
• 320 new msgs, 14,572 total
• Send quick email
Organizational Essentials
• Knowledge network
• Employee assistance
• IT procurement
• Organizational media design
• Organizational merchandise
Search
Go
Stocks
Full Portfolio
XYZ
YYZ
ZZZ
Market Update
50
29.5
45.25
As of:
Saturday, April 6, 2019
Get Quote
Reporting
Regional
• Northeast
• Northwest
• Southeast
• Southwest
• Midnorth
• Midsouth
State
• Alabama
• Arkansas
• Georgia
• Mississippi
• Vermont
• Virginia
Legacy
Application 1
Legacy
Application 2
Legacy
Application 3
Legacy
Application 4
Legacy
Application 5
Web
Service 1.1
Web
Service 1.2
Web
Service 1.3
Web
Service 2.1
Web
Service 2.2
Web
Service 3.1
Web
Service 3.2
Web
Service 4.1
Web
Service 4.2
Web
Service 5.1
Web
Service 5.2
Web
Service 5.3

Top Tier Demo
Portals as a Data Quality Tool

Defining Spaces
• ETL Extract Transform, Load
– delivers aggregated data to a
new database
• EAI Enterprise Application Integration
– connects applications to other applications in a
predictable manner using
pre-established connections
• EII Enterprise Information Integration
– between ETL and EAI - delivers tailored views of
information to users at the time that it is required
Meta-Matrix Virtual-Integration Example

By the end of this session, you should have a better understanding
of data management technologies and their use as part of a people
process & technology 3-legged stool in terms of:
• CASE Tools
• Repositories
• Data Life Cycle
Gartner Key Findings
• Data assets continue to drive strategic cloud service providers’
offerings
• Machine learning is increasingly popular–key uses:
– Data integration tools,
– Database management systems,
– Data quality tools and
– Metadata management solutions
• Increasing use of cloud for production applications requiring that
database in the cloud
• Organizations applying a combination of data warehouses, data
lakes and data hubs can achieve greater flexibility to support a
range of use cases compared to those applying only one.

IT Business
Data
Perceived State of Data
Data
Desired To Be State of Data
IT Business

The Real State of Data
Data
IT Business
It isn't possible to go digital
Digital

aBy just spelling 'data'
Dat
It requires more work
Data
a

Lady Ada Augusta King Rule
https://people.well.com/user/adatoole/bio.htm
Recent Technology Realization
GarbageIn➜
GarbageOut!Recent

GI➜GO!
Perfect
Model
Garbage
Data
Garbage
Results
Data
Warehouse
Machine
Learning
Business
Intelligence
Block Chain
AI
MDM
Analytics
Technology
Data
Governance
GI➜GO!
Perfect
Model
Quality
Data
Garbage
Results
Data
Warehouse
Machine
Learning
Business
Intelligence
Block Chain
AI
MDM
Analytics
Technology
Data
Governance

Quality In ➜ Quality Out!
Perfect
Model
Quality
Data
Good
Results
Data
Warehouse
Machine
Learning
Business
Intelligence
Block Chain
AI
MDM
Analytics
Technology
Data
Governance
More Data Management Tools

More Data Management Tools
Gartner Recommendations
Leaders seeking to advance their strategies and deliver effective solutions
must:
• Validate both product capabilities and data availability
• Leverage automation to free up scarce specialist resources
• Update policies and governance
standards to address cloud and
database platform as a service
(dbPaaS) before purchasing
• Favor providers with a clear
roadmap for ML enabling better
business outcomes or service levels
• Don’t expect a single piece of
infrastructure to meet all of your
needs. Plan the core by
assessing use case type,
processing flexibility and
semantic requirements

+ =
Questions?
It’s your turn!
Use the chat feature or
Twitter (#dataed) to submit
your questions now!
Upcoming Events
May Webinar
Data Management Best Practices
May 12, 2020 @ 2:00 PM ET
June Webinar
Approaching Data Governance
Strategically
June 9, 2020 @ 2:00 PM ET
Sign up for webinars at:
www.datablueprint.com/webinar-schedule
Brought to you by:

10124 W. Broad Street, Suite C
Glen Allen, Virginia 23060
804.521.4056
Copyright 2020 by Data Blueprint Slide # 95

DataEd Slides: Leveraging Data Management Technologies

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie DataEd Slides: Leveraging Data Management Technologies

Ähnlich wie DataEd Slides: Leveraging Data Management Technologies (20)

Mehr von DATAVERSITY

Mehr von DATAVERSITY (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (17)

DataEd Slides: Leveraging Data Management Technologies