Bridging Data Modeling and Data Integration with an Agile Approach

Bridging
DATAMODELING
and
DATAINTEGRATION
Pavel Najvar
Chief Technology Evangelist
CloverETL

DATA INTEGRATION SOFTWARE
designed for IT staff to rapidly
implement, manage, and automate
data workflows that take care of
converting data from sources to
targets, solve data quality issues,
perform complex data movements
between applications, and even
facilitate the continuous exchange
of data among systems.
❏ RAPID DESIGN
❏ TRANSFORMATION POWER (VISUAL & CODE)
❏ ORCHESTRATION
❏ AUTOMATION
❏ PUBLISHING DATA

AN INTERESTING CHALLENGE
How to quickly transition from models
to production data flows?

Developing and operating a data architecture are
often two separate activities (... teams,
technologies).

technologies).
MODELING TOOLS
Great at modeling, but bad at execution.
❏ Easy to see data relationships and
transformations
❏ Usually very weak ability to execute the models,
no automation, monitoring, …
❏ Usually no ability to consume data from queues,
files, remote locations, web services, …

technologies).
MODELING TOOLS
transformations
CloverETL
Optimized for execution & automation
❏ Provides data design tools as well, but is oriented
towards runtime, not modeling use cases
❏ Easily connects to variety of data sources (not
focused on just DBMS)
❏ Provides monitoring and automation tools
needed in production environments

MODELING TOOLS
transformations
CloverETL
❏ Provides data design tools as well, but is oriented
towards runtime, not modeling use cases
❏ Easily connects to variety of data sources (not
focused on just DBMS)
❏ Provides monitoring and automation tools
needed in production environments
Blend developing and operating a data architecture
much closer with a bridge.

MODELING TOOLS
CloverETL
Blend developing and operating a data architecture
much closer with a bridge.

Blend Developing and operating a data architecture with a bridge.
MODELING TOOLS
CloverETL
SELECT
S0.orderId AS orderId,
S0.customerId AS customerId,
S0.orderDatetime AS orderDatetime,
S0.orderState AS orderState,
S1.lineItemCount AS lineItemCount,
S1.totalValue AS orderValue
FROM
SCHEMA.EXTENDED_ORDER S0,
SCHEMA.LINE_ITEM_SUMMARY S1
WHERE
S0.orderId = S1.orderId;

CloverETL
Benefits from transitioning to the realm of Data Integration:
❏ Connectivity outside just DBs
❏ Queues, APIs, NoSQL, files
❏ Repeatability, Automation and Monitoring
❏ Reliably operating the data architecture
❏ Publish data to any target
❏ Database, file, Queue, API, ...

❏ Connectivity outside just DBs
❏ Queues, APIs, NoSQL, files
❏ Repeatability, Automation and Monitoring
❏ Reliably operating the data architecture
❏ Publish data to any target
❏ Database, file, Queue, API, ...
www.cloveretl.com
THANK YOU

Agile & Data Modeling -
How Can They Work Together?
Donna Burbank
Global Data Strategy Ltd.
Lessons in Data Modeling DATAVERSITY Series
October 26th, 2017

Global Data Strategy, Ltd. 2017
Donna Burbank
Donna is a recognised industry expert in
information management with over 20
years of experience in data strategy,
information management, data
modeling, metadata management, and
enterprise architecture. Her background
is multi-faceted across consulting,
product development, product
management, brand strategy, marketing,
and business leadership.
She is currently the Managing Director at
Global Data Strategy, Ltd., an
international information management
consulting company that specializes in
the alignment of business drivers with
data-centric technology. In past roles,
she has served in key brand strategy and
product management roles at CA
Technologies and Embarcadero
Technologies for several of the leading
data management products.
As an active contributor to the data
management community, she is a long
time DAMA International member, Past
President and Advisor to the DAMA
Rocky Mountain chapter, and was
recently awarded the Excellence in Data
Management Award from DAMA
International in 2016.
She was on the review committee for
the Object Management Group’s
Information Management Metamodel
(IMM) and the Business Process
Modeling Notation (BPMN). Donna is
also an analyst at the Boulder BI Train
Trust (BBBT) where she provides advices
and gains insight on the latest BI and
Analytics software in the market.
She has worked with dozens of Fortune
500 companies worldwide in the
Americas, Europe, Asia, and Africa and
speaks regularly at industry
conferences. She has co-authored two
books: Data Modeling for the
Business and Data Modeling Made
Simple with ERwin Data Modeler and is a
regular contributor to industry
publications.
She can be reached at
donna.burbank@globaldatastrategy.com
Donna is based in Boulder, Colorado,
USA.
2Follow on Twitter @donnaburbank

DATAVERSITY Lessons in Data Modeling Series
• January - on demand How Data Modeling Fits Into an Overall Enterprise Architecture
• February - on demand Data Modeling and Business Intelligence
• March - on demand Conceptual Data Modeling – How to Get the Attention of Business Users
• April - on demand The Evolving Role of the Data Architect – What does it mean for your Career?
• May - on demand Data Modeling & Metadata Management
• June - on demand Self-Service Data Analysis, Data Wrangling, Data Munging, and Data Modeling
• July - on demand Data Modeling & Metadata for Graph Databases
• August - on demand Data Modeling & Data Integration
• Sept 28 - on demand Data Modeling & Master Data Management (MDM)
• October 26 Agile & Data Modeling – How Can They Work Together?
• December 5 Data Modeling, Data Quality & Data Governance
3
This Year’s Line Up

Agenda
• Data Modeling and Agile – Key Definitions & Context
• The Business Value of Data Modeling in a Agile Way
• An Agile Approach to Data Modeling
• Summary & Questions
4
What we’ll cover today

Data Modeling is Hotter than ever
5
In a recent DATAVERSITY survey,
over 96% of were engaged in Data
Modeling in their organizations.
Sexiest job of the 21st Century?

What are Data Models?
6
• Data Models are another good source of both business & technical metadata
• They store structural metadata as well as business rules & definitions – in a visual, easily-
consumable, “agile” way.
Customer
Customer_ID CHAR(18) NOT NULL
First Name
Last Name
City
Date Purchased
CHAR(18)
CHAR(18)
CHAR(18)
CHAR(18)
NOT NULL
NOT NULL
NULL
NULL
Technical Metadata Business Metadata
Data Model

What is Agile?
We are uncovering better ways of developing software by doing it and helping others do it.
Through this work we have come to value:
• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
7Agile Manifesto courtesy of http://agilemanifesto.org/
The Agile Manifesto

Capital “Agile: vs. Lowercase “agile”
The Agile Manifesto:
• Individuals and interactions over processes
and tools
• Working software over comprehensive
documentation
• Customer collaboration over contract
negotiation
• Responding to change over following a plan
8
Agile:
Adjective
1. Quick and well-coordinated in movement.
2. Active
3. Marked by an ability to think quickly
• There is the “Agile” design methodology, and then there is just the plain, old meaning of “agile”.
courtesy of http://www.dictionary.com/

Agility is not a new concept
9
1947 2015

The Sages Agree
“Inch by inch, everything’s a cinch. Yard by yard, everything is hard.” - John Bytheway
10
"The journey of a thousand miles begins with a single step." - Lao Tzu (circa 500 BC)
“A stitch in time saves nine.” - proverb
“If you don’t have time to do it right, do you have time to do it again?”– numerous sources
“Just do it” - Nike
“Implement your data modling project in small, incremental steps, creating ‘quick wins’
that build to a longer-term sustainable architecture." – Donna Burbank

Find a Balance in Implementing a Data Architecture
• Find the Right Balance
• Data Modeling projects can have the reputation for being overly “academic”, long, expensive, etc.
• No architecture at all can cause chaos.
• When done correctly, data modeling improve efficiency and better align with business priorities
11
Focus on Business Value
Business Value
Too Academic, nothing
gets done
Too “Wild West”, nothing
gets done - chaos

Where is Your Organization on this Spectrum?
• Let’s do a “current state maturity assessment”
• Is your organization A, B, C, or D or E?
12
“Analysis Paralysis” or “Wild West”?
Business Value
Too Academic, nothing
gets done
Too “Wild West”, nothing
gets done - chaos
A
B
C
D
E

So How Do You Make Sense of It All?
• With the amount of data sources available and stakeholders involved, creating a
data strategy can be a daunting task.
• It’s critical to create a Data Strategy that:
• is agile and provides solid results
• manages the complexity of today’s data ecosystem
• is sustainable both architecturally & organizationally
13

Data Modeling is Part of a Larger Enterprise Landscape
14
A Successful Data Strategy Requires Many Inter-related Disciplines
“Top-Down” alignment with
business priorities
“Bottom-Up” management &
inventory of data sources
Managing the people, process,
policies & culture around data
Coordinating & integrating
disparate data sources
Leveraging & managing data for
strategic advantage

Data Models is Needed by Business Stakeholders
15
Making business decisions on accurate and well-understood data
In organizations using data models,
• 73% are using Logical Data Models
• 68% are using Conceptual Data Models

Levels of Data Models
16
Conceptual
Logical
Physical
Purpose
Communication & Definition of
Business Terms & Rules
Clarification & Detail
of Business Rules &
Data Structures
Technical
Implementation on
a Physical Database
Audience
Business Stakeholders
Data Architecture
Business Analysts
DBAs
Developers
Business Concepts
Data Entities
Physical Tables

Speak with a Wide Variety of Stakeholders
17

Stakeholder Feedback
• Determine key business issues & drivers through direct feedback.
• Many issues around data can be resolved through data modeling.
18
I didn’t know we had any
documented data
standards
$12m has been spent on
projects to clean up the data
over the past 2-3 years
What are the data structures
used in the application?
We have 15 customer
databases – with many
duplications.
There is limited ownership or
enforcement of common
practices and standards
across the projects
There was an error in reporting
products by customer & region
that was noticed by upper
management.
The application isn’t working right –
it doesn’t allows customers to have
more than one email.
Customer Support keeps
entering the wrong
Return Codes.

Tell a Story
• A core artifact of the Agile Development
Methodology is the User Story.
• A user story is a very high-level definition of a
requirement, containing just enough information for
developers to understand the effort. They strive to
be:
• Clear & Concise
• Not more than a few sentences
• Data models tell great stories.
• Clear, concise, and visual
• Can be read as a sentence
• Have the added benefit of being data and database-
centric.
19
A Data Model is a great User Story

Telling a User Story with a Data Model
20
• A small snippet of a data model can speak volumes, and highlight key business requirements.
The application isn’t working right –
it doesn’t allows customers to have
more than one email!
User Issue Current Database
Design
Proposed Design to
Resolve Issue
The system doesn’t allow us to store
more than one email for each
customer currently…
…But is this logic correct?
• Should we also track home vs work email? (e.g. Type)
• Is a customer required to have an email? Or could the
email for a customer be unknown?
Developer

Avoid “Death by Data Modeling”
21
• “We’re just going to sit in this room for a few days
until we scope out the entire enterprise data model
plastered across these three walls.
• Just about 1000 entities or so…
• First off, what is the data type for account code? …”

Case Study: International Pharmaceutical Company
• An international Pharmaceutical company was looking to make better use of its
data to streamline its Clinical Development, Commercial Processes, and R&D.
• Business alignment was a key first step
• Business sponsor had been involved in “death by modeling” efforts in the past.
• Instead, created “blueprints” of how the business runs & how data maps to that
• These were actually detailed data models, process models, & mappings – but done
in a business-focused, agile, easily-consumable way.
• Research scientists literally had data models printed on their walls – with sticky
notes and pen marks to indicate changes & feedback.
• Data-driven Efficiencies and Process Improvement were discovered in the R&D
process.
• Business stakeholders were convinced of the value of data management &
governance
• Greater understanding how data was used by and critical to key business activities
22
Business Alignment through Process & Data

The Value of Whiteboarding
It’s often helpful to “whiteboard” data models with sticky notes
Policy
Account
Employee
• Short whiteboard sessions with
key stakeholders can flesh out key
metadata definitions & scope in a
short period of time.
• And it can be fun and interactive.
• Very agile.

Break the Large Modeling Efforts into Manageable Chunks
24
Instead of creating large models all at once Break them into smaller “chunks” / sprints

An Agile Approach to Data Modeling
Align with
Business
Needs
Top-Down
Business
Design
Bottom-Up
Technical
Review
Iterate &
Refine
Publish &
Communicate
• Align with Business Priorities
• Create Subject-Area Focused Working Group
• Source Documentation from Related Efforts
• Scope Business Subject Area(s)
• Define core business entities &
relationships
• Draft entity definitions
• Reverse Engineer Physical Models
for related systems
• Align with project teams for App &
System Delivery
• Iterate refine business model based
on differing system rules
• Utilizing the Agile Sprint
approach for constant team and
business feedback for quick
results (Core Working Group)
• Fail fast for quick
correction and ultimate
solid model delivery
(Wider Enterprise)
Rapid Development, Rapid Feedback
Focus on Communication & Iteration

A little data modeling up-front prevents headaches down the road
From Data Modeling for the Business by Hoberman, Burbank, Bradley, Technics Publications, 2009
• It’s often tempting to skip data
modeling documentation because it’s
“faster”
• But…long-term, it’s ultimately longer as
errors and inconsistencies need to be
fixed as a result.
“If you don’t have time to do it right, do
you have time to do it again?”

Integrating Data Modeling Into the Agile Lifecycle
27
• Integrating Data Modeling & Metadata checkpoints & activities into the Agile development lifecycle helps
proactively manage data-related issues before and during development, rather than reactively after the fact.
• Below is a high-level overview of the types of data-related questions that can be asked by team members along
the various phases of the Agile development lifecycle.
• Are there common standards that can be reused?
• How do I publish & share my work with others?
• Are there overlaps or conflicts in data usage or design?
• Are other teams defining & using terms differently?
• How will we implement our core data requirements?
• What are our agreed definitions for core concepts (e.g.
Active Account?)
• Are there any new data
requirements?
Product
Owner
New Vision/Concept
Release Planning
Agile
Development
Sprints
Planning Day
Data
Stewards
Developers
Etc.
Developers Developers
Product
Manager

Human Metadata
• Much business metadata and the history of the business exists in employee’s heads.
• It is important to capture this metadata in an electronic format for sharing with others.
• Avoid the dreaded “I just know”
28
Avoid the dreaded “I just know”
Part Number is what used to
be called Component
Number before the
acquisition.
Business Glossary
Metadata Repository
Data Models
Etc.

Data Models can provide “Just Enough” Metadata Management
29
Metadata
Storage
Metadata
Lifecycle &
Versioning
Data Lineage
Visualization
Business Glossary Data Modeling
Metadata
Discovery &
Integration w/
Other Tools
Customizable
Metamodel
Data Modeling Tools
(e.g. Erwin, SAP
PowerDesigner, Idera
ER/Studio)
x X x X X x
Metadata Repositories (e.g.
ASG, Adaptive, DAG) X X X X X X
Data Governance Tools (e.g.
Collibra, Diaku) x x X x x
Spreadsheets x x x
• While data modeling tools are not metadata repositories, nor designed to be, they offer many features shared with these
repository solutions:
• Metadata storage, Data lineage visualization, Business Glossary, Integration with BI tools, ETL tools, etc.
• Metadata repositories have a broader range metadata sources & dedicated metadata management support.
• And Data Modeling tools, of course, have the added benefit of doing data modeling! 
• And the benefit is that much of the needed metadata is in these data models.

Summary
• Data Modeling is more important than ever
• Data models are both “Agile” and “agile”
• Align data models with critical business objectives and identify “quick wins”
• Use small “sprints” to create data models – not all at once
• Have fun! Data models are for the cool kids.

About Global Data Strategy, Ltd
• Global Data Strategy is an international information management consulting company that specializes
in the alignment of business drivers with data-centric technology.
• Our passion is data, and helping organizations enrich their business opportunities through data and
information.
• Our core values center around providing solutions that are:
• Business-Driven: We put the needs of your business first, before we look at any technology solution.
• Clear & Relevant: We provide clear explanations using real-world examples.
• Customized & Right-Sized: Our implementations are based on the unique needs of your organization’s
size, corporate culture, and geography.
• High Quality & Technically Precise: We pride ourselves in excellence of execution, with years of
technical expertise in the industry.
31
Data-Driven Business Transformation
Business Strategy
Aligned With
Data Strategy
Visit www.globaldatastrategy.com for more information

DATAVERSITY Lessons in Data Modeling Series
• January - on demand How Data Modeling Fits Into an Overall Enterprise Architecture
• February - on demand Data Modeling and Business Intelligence
• March - on demand Conceptual Data Modeling – How to Get the Attention of Business Users
• April - on demand The Evolving Role of the Data Architect – What does it mean for your Career?
• May - on demand Data Modeling & Metadata Management
• June - on demand Self-Service Data Analysis, Data Wrangling, Data Munging, and Data Modeling
• July - on demand Data Modeling & Metadata for Graph Databases
• August - on demand Data Modeling & Data Integration
• Sept 28 - on demand Data Modeling & Master Data Management (MDM)
• October 26 Agile & Data Modeling – How Can They Work Together?
• December 5 Data Modeling, Data Quality & Data Governance
32
This Year’s Line Up

White Paper: Trends in Data Architecture
33
Free Download
• Available for download on dataversity.net

Questions?
34
Thoughts? Ideas?

Bridging Data Modeling and Data Integration with an Agile Approach

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Bridging Data Modeling and Data Integration with an Agile Approach

Ähnlich wie Bridging Data Modeling and Data Integration with an Agile Approach (20)

Mehr von DATAVERSITY

Mehr von DATAVERSITY (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Bridging Data Modeling and Data Integration with an Agile Approach