4. IBM Software Group | WebSphere software
4
Course Roadmap
• Why we use Data warehousing
• Difference between Operational System and Data Warehouse
• Introduction to Data warehousing
• Data Warehousing Approaches
• Data Warehouse Technical Architecture
• Data Modelling concepts
• Operational Data Store
• Schema Design of Data warehouse
• Data Acquisation
• ETL Products
• Project Life Cycle
5. IBM Software Group | WebSphere software
5
Why We Need Data Warehousing ?
Better business intelligence for end-users
Reduction in time to locate, access, and analyze information
Consolidation of disparate information sources
To Store Large Volumes of Historical Detail Data from Mission Critical Applications
Strategic advantage over competitors
Faster time-to-market for products and services
Replacement of older, less-responsive decision support systems
Reduction in demand on IS to generate reports
6. IBM Software Group | WebSphere software
6
What is an Operational System?
Operational systems are just what their name implies; they are the systems that
help us run the day-to-day enterprise operations.
These are the backbone systems of any enterprise, such as order entry inventory
etc.
The classic examples are airline reservations, credit-card authorizations, and ATM
withdrawals etc.,
7. IBM Software Group | WebSphere software
7
Characteristics of Operational Systems
• Continuous availability
• Predefined access paths
• Transaction integrity
• Volume of transaction - High
• Data volume per query - Low
• Used by operational staff
• Supports day to day control operations
• Large number of users
8. IBM Software Group | WebSphere software
8
OLTP Vs Data Warehouse
Operational System Data Warehouse
Transaction Processing Query Processing
Predictable CPU Usage Random CPU Usage
Time Sensitive History Oriented
Operator View Managerial View
Normalized Efficient
Design for TP
Denormalized Design for
Query Processing
9. IBM Software Group | WebSphere software
9
OLTP Vs Warehouse
Operational System Data Warehouse
Designed for Atmocity,
Consistency, Isolation and
Durability
Designed for quite or static
database
Organized by transactions
(Order, Input, Inventory)
Organized by subject
(Customer, Product)
Relatively smaller database Large database size
Many concurrent users Relatively few concurrent
users
Volatile Data Non Volatile Data
10. IBM Software Group | WebSphere software
10
Operational System Data Warehouse
Stores all data Stores relevant data
Performance Sensitive Less Sensitive to performance
Not Flexible Flexible
Efficiency Effectiveness
11. IBM Software Group | WebSphere software
11
What is a Data Warehouse ?
Data WarehouseData Warehouse is a
Subject-Oriented
Integrated
Time-Variant
Non-volatile
WH Inmon - Regarded As Father Of Data WarehousingWH Inmon - Regarded As Father Of Data Warehousing
13. IBM Software Group | WebSphere software
13
13
Subject Oriented Analysis
Data Warehouse StorageTransactional Storage
SalesSales
CustomersCustomers
ProductsProducts
Entry
Sales Rep
Quantity Sold
Part Number
Date
Customer Name
Product Description
Unit Price
Mail Address
Process Oriented Subject Oriented
14. IBM Software Group | WebSphere software
14
14
Integration of Data
Data Warehouse StorageTransactional Storage
Appl. A - M, F
Appl. B - 1, 0
Appl. C - X, Y
Appl. A - pipeline cm.
Appl. B - pipeline inches
Appl. C - pipeline mcf
Appl. A - balance dec(13,2)
Appl. B - balance PIC 9(9)V99
Appl. C - balance float
Appl. A - bal-on-hand
Appl. B - current_balance
Appl. C - balance
Appl. A - date (Julian)
Appl. B - date (yymmdd)
Appl. C - date (absolute)
M, F
pipeline cm
balance dec(13, 2)
balance
date (Julian)
Integration
Encoding
Unit of
Attributes
Physical
Attributes
Naming
Conventions
Data
Consistency
15. IBM Software Group | WebSphere software
15
15
Load
Access
Mass Load / Access of DataRecord-by-Record Data Manipulation
Insert
Access
Insert
Change
Delete
Change
Volatile Non-Volatile
Volatility of Data
Data Warehouse StorageTransactional Storage
16. IBM Software Group | WebSphere software
16
16
Time Variant Data Analysis
Data Warehouse StorageTransactional Storage
Current Data Historical Data
0
5
10
15
20
Sales ( in lakhs
)
January February March
Year97
Sales ( Region , Year - Year 97 - 1st Qtr)
East
West
North
17. IBM Software Group | WebSphere software
Load/
Update
Consistent Points in Time
Updated constantly
Data changes according to
need, not a fixed schedule
Added to regularly, but loaded data
is rarely directly changed
Does NOT mean the Data
warehouse is never updated or
never changes!!
Constant Change
Operational systems
Database
Data warehouse
Datawarehouse- Differences from Operational
Systems
Insert
Insert
Update
Initial Load
Incremental Load
Incremental Load
Update
Delete
19. IBM Software Group | WebSphere software
19
DW Implementation Approaches
Top Down
Bottom-up
Combination of both
Choices depend on:
current infrastructure
resources
architecture
ROI
Implementation speed
20. IBM Software Group | WebSphere software
20
Heterogeneous Source Systems
Staging
Common Staging interface Layer
EDW- “Top Down”Approach
Data mart bus architecture Layer
Enterprise Datawarehouse
Source
1
Source
2
Source
3
Incremental Architected data marts
DM 1 DM 3DM 2
21. IBM Software Group | WebSphere software
21
Heterogeneous Source Systems
Staging
Common Staging interface Layer
EDW- “Bottom up”Approach
Data mart bus architecture Layer
Source
1
Source
2
Source
3
Incremental Architected data marts
DM 1 DM 3DM 2
Enterprise Datawarehouse
22. IBM Software Group | WebSphere software
22
Source System Data Staging Area Presentation Area
Services:
Transform from
source-to-Target
Maintain Conform
Dimensions
No user query
support
Data Store:
Flat files or
relational tables
Design Goals:
Staging
Throughput
integrity/
consistency
Load
Access
Ad Hoc Query Tools
Report Writers
Analytic Applications
Modeling:
Forecasting
Scoring
Data
Mining
Data Mart #1
Dimensional
Atomic AND
summery data
Business
Process Centric
Design Goals:
Easy-of -use
Query
Performance
Data Mart #2
Data Mart #.....
Data Mart Bus:
Conformed facts and dims
Extract
Extract
Extract
Data Access Tools
Independent Data Marts: Ralph Kimball’s Ideology
Ralph Kimball’ Approach
23. IBM Software Group | WebSphere software
23
•E/R Design or Flat File
•Retain History Needed
for
regular processing
•No end user access
• Dimensional
•Transaction &
Summary data
•Data Mart Single
subject area
(i.e. Fact table)
•Multiple Marts May
exist in a
Single Database
Instance
Bottom Up Approach
Staging Data Store
Data Warehouse
Data Mart Data Mart Data Mart
Data Mart Data MartData Mart
•Integrated Data
•Timely User Access
•Conformed Dimensions
•Single Process to
Build Dimension
25. IBM Software Group | WebSphere software
25
Bill Inmon’ Approach
Source
System
Data Staging
Area
Presentation
Area
“Enterprise Data
Warehouse”
Normalized
tables
Atomic Data
User query
support to
atomic data
Extract
Extract
Extract
Load
Data Mart #1
Dimensional
summery data
Departmental
Centric
Access
Access
Data Access
Tools
Data Mart #2
Data Mart #...
ETL
Dependent Data Marts: Bill Inmon’s Ideology
DWH
26. IBM Software Group | WebSphere software
26
Top Down Approach
• Raw Input Data
• E/R Model
• Subject Areas
• Transaction Level Detail
• Historical Persistency As justified- Archive
for Retrieval if Needed
• Most are dimensional
• Data Mart Design by Business
Function
• Summary Level Data
•
Data Mart Data Mart
Staging Data Store
Data
Warehouse
Data Mart
Data
Mart
Flat
File
•Integrated Data
•Timely user Access
•Single Process to build dimension
27. IBM Software Group | WebSphere software
27
DW Implementation Approaches
Top Down
More planning and design initially
Involve people from different work-
groups, departments
Data marts may be built later from
Global DW
Overall data model to be decided up-
front
Bottom Up
Can plan initially without waiting for
global infrastructure
built incrementally
can be built before or in parallel with
Global DW
Less complexity in design
28. IBM Software Group | WebSphere software
28
DW Implementation Approaches
Top Down
Consistent data definition and
enforcement of business rules across
enterprise
High cost, lengthy process, time
consuming
Works well when there is centralized IS
department responsible for all H/W and
resources
Bottom Up
Data redundancy and
inconsistency between data marts
may occur
Integration requires great planning
Less cost of H/W and other
resources
Faster pay-back
30. IBM Software Group | WebSphere software
30
Prod
Mkt
HR
Fin
Acctg
Data Sources
Transaction Data
IBM
IMS
VSAM
Oracle
Sybase
ETL Software Data Stores Data Analysis
Tools and
Applications
Users
Other Internal Data
ERP SAP
Clickstream Informix
Web Data
External Data
Demographic Harte-
Hanks
S
T
A
G
I
N
G
A
R
E
A
O
P
E
R
A
T
I
O
N
A
L
D
A
T
A
S
T
O
R
E
Ascential
Extract
Sagent
SAS
Clean/Scrub
Transform
Firstlogic
Load
DATASTAGE
Data Marts
Teradata
IBM
Data
Warehouse
Meta
Data
Finance
Marketing
Sales
Essbase
Microsoft
ANALYSTS
MANAGERS
EXECUTIVES
OPERATIONAL
PERSONNEL
CUSTOMERS/
SUPPLIERS
SQL
Cognos
SAS
Queries,Reporting,
DSS/EIS,
Data Mining
Micro Strategy
Siebel
Business
Objects
Web
Browser
31. IBM Software Group | WebSphere software
31
Benefits of DWH
To formulate effective business, marketing
and sales strategies.
To precisely target promotional activity.
To discover and penetrate new markets.
To successfully compete in the marketplace
from a position of informed strength.
To build predictive rather than retrospective models.
33. IBM Software Group | WebSphere software
33
Data Modeling
WHAT IS A DATA MODEL?
A data model is an abstraction of some aspect of the real
world (system).
WHY A DATA MODEL?
• Helps to visualize the business
• A model is a means of communication.
• Models help elicit and document requirements.
• Models reduce the cost of change.
• Model is the essence of DW architecture based on which
DW will be implemented
34. IBM Software Group | WebSphere software
34
STEPS in DATA MODELING
Problem & scope definition
Requirement Gathering
Analysis
Logical Database Design
Deciding Database
Physical Database design
Schema Generation
35. IBM Software Group | WebSphere software
35
Levels of modeling
Conceptual modeling
Describe data requirements from a
business point of view without technical
details
Logical modeling
Refine conceptual models
Data structure oriented, platform
independent
Physical modeling
Detailed specification of what is physically
implemented using specific technology
36. IBM Software Group | WebSphere software
36
Modeling Techniques
Entity-Relationship Modeling
Traditional modeling technique
Technique of choice for OLTP
Suited for corporate data warehouse
Dimensional Modeling
Analyzing business measures in the specific business context
Helps visualize very abstract business questions
End users can easily understand and navigate the data
structure
37. IBM Software Group | WebSphere software
37
Relationship
Relationship between entities - structural interaction
and association
described by a verb
Cardinality
1-1
1-M
M-M
Example : Books belong to Printed Media
Entity-Relationship Modeling - Basic Concepts
38. IBM Software Group | WebSphere software
38
Entity-Relationship Modeling - Basic Concepts
Attributes
Characteristics and properties of entities
Example :
Book Id, Description, book category are
attributes of entity “Book”
Attribute name should be unique and self-
explanatory
Primary Key, Foreign Key, Constraints are defined
on Attributes
39. IBM Software Group | WebSphere software
Review of Logical Modeling Terms & Symbols
Entities define specific groups of information
Sales Organization
Sales Org ID
Distribution Channel
Entity
40. IBM Software Group | WebSphere software
Review of Logical Modeling Terms & Symbols
One or more attribute uniquely identifies an instance of an
entity
Sales Organization
Sales Org ID
Distribution Channel
Identifier
41. IBM Software Group | WebSphere software
Review of Logical Modeling Terms & Symbols
The logical model identifies relationships between
entities
Sales Detail
Sales Record ID
Sales Rep
Sales Rep ID
Relationship
{
43. IBM Software Group | WebSphere software
Logical Data Model
Sales Detail
Sales Record ID
Customer
Customer ID
Product
Product SKU
Suppliers
Supplier ID
Manufacturing Group
Manufacturing Org ID
Factory
Factory ID
Sales Organization
Sales Org ID
Distribution Channel
Sales Rep
Sales Rep ID
Retail
Market
Product Sales Plan
Plan ID
Wholesale
Industry
48. IBM Software Group | WebSphere software
48
Dimensional Modeling
Dimensional modeling uses three basic concepts : measures,
facts, dimensions.
Is powerful in representing the requirements of the business
user in the context of database tables.
Focuses on numeric data, such as values counts, weights,
balances and occurences.
49. IBM Software Group | WebSphere software
49
What is a Facts
A fact is a collection of related data items, consisting of measures
and context data.
Each fact typically represents a business item, a business
transaction, or an event that can be used in analyzing the business
or business process.
Facts are measured, “continuously valued”, rapidly changing
information. Can be calculated and/or derived.
Granularity
The level of detail of data contained in the data warehouse
e.g. Daily item totals by product, by store
50. IBM Software Group | WebSphere software
50
Types of Facts
Additive
Able to add the facts along all the dimensions
Discrete numerical measures eg. Retail sales in $
Semi Additive
Snapshot, taken at a point in time
Measures of Intensity
Not additive along time dimension eg. Account balance, Inventory
balance
Added and divided by number of time period to get a time-average
Non Additive
Numeric measures that cannot be added across any dimensions
Intensity measure averaged across all dimensions eg. Room
temperature
Textual facts - AVOID THEM
51. IBM Software Group | WebSphere software
51
Dimensions
A dimension is a collection of members or units of the same type
of views.
Dimensions determine the contextual background for the facts.
Dimensions represent the way business people talk about the
data resulting from a business process, e.g., who, what, when,
where, why, how
52. IBM Software Group | WebSphere software
52
52
Dimensional Hierarchy
World
America AsiaEurope
USA
FL
Canada Argentina
GA VA CA WA
TampaMiami Orlando Naples
Continent Level
State Level
City Level
World Level
Country Level
ParentRelation
Dimension Member /
Business Entity
Geography Dimension
Attributes: Population, Tourist’s Place
53. IBM Software Group | WebSphere software
53
Dimensions Types
Conformed Dimension
Junk Dimension
Fast Changing Dimension
Role Playing Dimension
‘Garbage’ Dimension
Slowly Changing Dimension
Degenerated Dimension
53
54. IBM Software Group | WebSphere software
54
What is a Slowly Changing Dimension?
Although dimension tables are typically static lists, most dimension tables do change over
time.
Since these changes are smaller in magnitude compared to changes in fact tables, these
dimensions are known as slowly growing or slowly changing dimensions.
55. IBM Software Group | WebSphere software
55
Slowly Changing Dimension -Classification
Slowly changing dimensions are classified into three different
types
TYPE I
TYPE II
TYPE III
56. IBM Software Group | WebSphere software
56
Slowly Changing Dimensions Type I
Shane
Name
Shane@xyz.com1001
EmailEmp id
Shane
Name
Shane@xyz.com1001
EmailEmp id
Shane
Name
Shane@
abc.co.in
1001
EmailEmp id
Shane
Name
Shane@
abc.co.in
1001
EmailEmp id
Source
Source Target
Target
Shane@
xyz.com
57. IBM Software Group | WebSphere software
57
Slowly Changing Dimensions Type II
Shane
Name
Shane@xyz.com10
EmailEmp id
Shane@x
yz.
com
Email
Shane
Name
10
Emp id
1000
PM_PRI
MARYK
EY
0
PM_VER
SION_N
UMBER
Source Target
59. IBM Software Group | WebSphere software
59
Slowly Changing Dimensions -Versioning
Shane
Name
Shane@
abc.co.in
10
EmailEmp id
Source
Target
0Shane@
xyz.com
Shane101000
1Shane@
abc.co.in
Shane101001
EmailNameEmp idPM_PRIMA
RYKEY
PM_VERSION_NUMBER
60. IBM Software Group | WebSphere software
60
Slowly Changing Dimensions -Versioning
Shane
Name
Shane@
abc.com
10
EmailEmp id
Source
Target
1Shane@
abc.co.in
Shane101001
2Shane@
abc.com
Shane101003
0Shane@
xyz.com
Shane101000
EmailNameEmp idPM_PRIM
ARYKEY
PM_VERSION_NUM
BER
61. IBM Software Group | WebSphere software
61
Slowly Changing Dimensions Type II -
Flag
Shane
Name
Shane@xyz.com10
EmailEmp id
Shane@
xyz.
com
Email
Shane
Name
10
Emp id
1000
PM_PR
IMAR
YKEY
Y
PM_CUR
RENT_FL
AG
Source
Target
62. IBM Software Group | WebSphere software
62
Slowly Changing Dimensions - Flag Current
Shane
Name
Shane@
abc.co.in
10
EmailEmp id
Source
Target
NShane@
xyz.com
Shane101000
YShane@
abc.co.in
Shane101001
EmailNameEmp idPM_PRIMA
RYKEY
PM_CURRENT_FLAG
63. IBM Software Group | WebSphere software
63
Slowly Changing Dimensions - Flag Current
Shane
Name
Shane@
abc.com
10
EmailEmp id
Source
Target
NShane@
abc.co.in
Shane101001
YShane@
abc.com
Shane101003
NShane@
xyz.com
Shane101000
EmailNameEmp idPM_PRIMA
RYKEY
PM_CURRENT_FLAG
65. IBM Software Group | WebSphere software
65
Slowly Changing Dimensions Type II
Shane
Name
Shane@xyz.c
om
10
EmailEmp id
01/01/00
PM_BEG
IN_DAT
E
Shane@x
yz.com
Email
Shane
Name
10
Emp id
1000
PM_PRI
MARYK
EY
PM_EN
D_DATE
Source
Target
66. IBM Software Group | WebSphere software
66
Slowly Changing Dimensions -Effective Date
Shane
Name
Shane@
abc.co.in10
Email
Emp id
Source
Target
03/01/00
01/01/00
PM_BEGIN_D
ATE
03/01/00Shane@x
yz.com
Shane101000
Shane@
abc.co.in
Shane101001
EmailNameEmp idPM_PRIMAR
YKEY
PM_END_D
ATE
67. IBM Software Group | WebSphere software
67
Slowly Changing Dimensions - Effective Date
Shane
Name
Shane@
abc.com10
EmailEmp id
Source
Target
05/02/00
03/01/00
01/01/00
PM_BEGIN_D
ATE
05/02/00Shane@
abc.co.in
Shane101001
Shane@
abc.com
Shane101003
03/01/00Shane@
xyz.com
Shane101000
EmailNameEmp idPM_PRIM
ARYKEY
PM_END_DA
TE
69. IBM Software Group | WebSphere software
69
Slowly Changing Dimensions Type III
Shane
Name
Shane@xyz.c
om
10
EmailEmp id
PM_Prev_
Column
Name
Shane@xyz.
com
Email
Shane
Name
10
Emp id
1
PM_PRI
MARYKE
Y
01/01/00
PM_EFFEC
T_DATE
Source
Target
70. IBM Software Group | WebSphere software
70
Slowly Changing Dimensions Type III
Shane
Name
Shane@
abc.co.in10
EmailEmp id
Source
Target
Shane@xyz.co
m
PM_Prev_Colu
mnName
01/02/00Shane@
abc.co.in
Shane101
EmailNameEmp idPM_PRIMAR
YKEY
PM_EFFEC
T_DATE
71. IBM Software Group | WebSphere software
71
Slowly Changing Dimensions Type III
Shane
Name
Shane@
abc.com10
EmailEmp id
Source
Target
Shane@
abc.co.in
PM_Prev_Colu
mnName
01/03/00Shane@
abc.com
Shane101
EmailNameEmp idPM_PRIM
ARYKEY
PM_EFFECT_
DATE
72. IBM Software Group | WebSphere software
72
Degenerate Dimension
Dimension keys in fact table without corresponding dimension tables are
called Degenerate Dimensions
Purpose of Degenerate Dimensions
1. Generally used when each record in fact represents transaction line item
2. Useful for grouping transaction line items belonging to a single
transaction
73. IBM Software Group | WebSphere software
73
Fast Changing Dimension
A fast changing dimension is a dimension whose attribute or
attributes for a record (row) change rapidly over time.
1. Example: Age of associates, Income, Daily balance etc.
2. Technique to handle fast changing dimension: Create band
tables
74. IBM Software Group | WebSphere software
74
Role Playing Dimension
A single dimension which is expressed differently in a fact table using
views is called a role-playing dimension. This can be achieved by
creating views on dimension table.
75. IBM Software Group | WebSphere software
75
Conformed Dimension
A conformed dimension means the same thing to
each fact table to which it can be joined.
Typically, dimension tables that are referenced or are
likely to be referenced by multiple fact tables
(multiple dimensional models) are called conformed
dimensions
.
76. IBM Software Group | WebSphere software
76
Conformed Dimension Option #1
Identical dimensions with same keys, labels, definitions and Values
Sales Schema
Inventory Schema
SALES Facts
DATE KEY
PRODUCT KEY
STORE KEY
PROMO KEY
Product Desc
Brand Desc
Category Desc
PRODUCT KEY
INVENTORY
Facts
DATE KEY
PRODUCT KEY
STORE KEY
Product Desc
Brand Desc
Category Desc
PRODUCT KEY
77. IBM Software Group | WebSphere software
77
Conformed Dimension Option #2
Subset of base dimension with common labels, definitions
and values
Sales
Schema
Forecast
Schema
SALES $
DATE KEY
PRODUCT KEY
STORE KEY
PROMO KEY
Product Desc
Brand Desc
Category Desc
PRODUCT KEY
DATE KEY
Day-of-week
Week Desc
Month Desc
SALES $
MONTH KEY
BRAND KEYBrand Desc
Category Desc
BRAND KEY MONTH KEY
Month Desc
BRAND KEY Brand Desc Category Desc
12345 Cherriors Cereal
PROD KEY Prod Desc Brand Desc Category Desc
12345 Cherriors 10 Cherriors Cereal
78. IBM Software Group | WebSphere software
78
‘Garbage’ Dimension
A garbage dimension is a dimension that consists of low-cardinality columns
such as codes, indicators, and status flags.
Approach to handle Garbage dimension:
• Put the new attributes into existing dimension tables.
• Put the new attributes into the fact table.
• Create new separate dimension tables garbage dimension
• Create a separate ‘Garbage Dimension’ table
79. IBM Software Group | WebSphere software
79
Junk Dimensions
Whether to use junk dimension
5 indicators, each has 3 values -> 243 (35
) rows
5 indicators, each has 100 values -> 100 million (1005
) rows
When to insert rows in the dimension
80. IBM Software Group | WebSphere software
80
Factless Fact Tables
The two types of factless fact tables are:
Coverage tables
Event tracking tables
81. IBM Software Group | WebSphere software
81
Factless Fact Tables - Coverage Tables
Coverage tables are required when a primary fact
table is sparse
Example: Tracking products in a store that did not sell
83. IBM Software Group | WebSphere software
83
Factless Fact Tables - Event Tracking
These tables are used for tracking a event:
Example: Tracking student attendance
84. IBM Software Group | WebSphere software
84
Fact Constellation
Fact constellations: Multiple fact tables share dimension tables,viewed as
a collection of stars, therefore called galaxy schema or fact constellation
85. IBM Software Group | WebSphere software
85
What is a Data mart?
Data mart is a decentralized subset of data found either in a data warehouse
or as a standalone subset designed to support the unique business unit
requirements of a specific decision-support system.
Data marts have specific business-related purposes such as measuring the
impact of marketing promotions, or measuring and forecasting sales
performance etc,.
Data Mart
Data Mart
Enterprise
Data Warehouse
86. IBM Software Group | WebSphere software
86
Data marts - Main Features
Main Features:
Low cost
Controlled locally rather than centrally, conferring power on the user group.
Contain less information than the warehouse
Rapid response
Easily understood and navigated than an enterprise data warehouse.
Within the range of divisional or departmental budgets
88. IBM Software Group | WebSphere software
88
Datamart Advantages :
Typically single subject area and fewer dimensions
Limited feeds
Very quick time to market (30-120 days to pilot)
Quick impact on bottom line problems
Focused user needs
Limited scope
Optimum model for DW construction
Demonstrates ROI
Allows prototyping
Advantages of Datamart over Datawarehouse
89. IBM Software Group | WebSphere software
89
Data Mart disadvantages :
• Does not provide integrated view of business information.
• Uncontrolled proliferation of data marts results in redundancy
• More number of data marts complex to maintain
• Scalability issues for large number of users and increased data volume
Disadvantages of Data Mart
90. IBM Software Group | WebSphere software
90
90
Data marts
• Embedded data marts are marts that are stored within
the central DW. They can be stored relationally as files or
cubes.
• Dependent data marts are marts that are fed directly by
the DW, sometimes supplemented with other feeds, such
as external data.
• Independent data marts are marts that are fed directly
by external sources and do not use the DW.
DM - Types
93. IBM Software Group | WebSphere software
93
Why We Need Operational Data Store?
Need
To obtain a “system of record” that contains the best data that
exists in a legacy environment as a source of information
Best here implies data to be
Complete
Up to date
Accurate
In conformance with the organization’s information model
94. IBM Software Group | WebSphere software
ODS data resolves data integration issues
Data physically separated from production
environment to insulate it from the processing
demands of reporting and analysis
Access to current data facilitated.
Operational Data Store - Insulated from OLTP
Tactical
Analysis
OLTP Server
ODS
95. IBM Software Group | WebSphere software
95
Detailed data
Records of Business Events
(e.g. Orders capture)
Data from heterogeneous sources
Does not store summary data
Contains current data
Operational Data Store - Data
97. IBM Software Group | WebSphere software
97
ODS- Benefits
Integrates the data
Synchronizes the structural differences in data
High transaction performance
Serves the operational and DSS environment
Transaction level reporting on current data
Flat
files
Relational
Database
Operational
Data Store
60,5.2,”JOHN”
72,6.2,”DAVID”
Excel files
98. IBM Software Group | WebSphere software
Update schedule - Daily or less
time frequency
Detail of Data is mostly between
30 and 90 days
Addresses operational needs
Weekly or greater time frequency
Potentially infinite history
Address strategic needs
Operational Data Store- Update schedule
ODS
Data
Data warehouse
Data
100. IBM Software Group | WebSphere software
100
OLTP Vs ODS Vs DWH
Characteristic OLTP ODS Data Warehouse
Data redundancy Non-redundant
within system;
Unmanaged
redundancy among
systems
Somewhat
redundant with
operational
databases
Managed
redundancy
Data stability Dynamic Somewhat dynamic Static
Data update Field by field Field by field Controlled batch
Data usage Highly structured,
repetitive
Somewhat
structured, some
analytical
Highly
unstructured,
heuristic or
analytical
Database size Moderate Moderate Large to very large
Database
structure stability
Stable Somewhat stable Dynamic
101. IBM Software Group | WebSphere software
101
Star Schema Design
Single fact table surrounded by denormalized dimension
tables
The fact table primary key is the composite of the
foreign keys (primary keys of dimension tables)
Fact table contains transaction type information.
Many star schemas in a data mart
Easily understood by end users, more disk storage
required
103. IBM Software Group | WebSphere software
103
Snowflake Schema
Single fact table surrounded by normalized dimension
tables
Normalizes dimension table to save data storage space.
When dimensions become very very large
Less intuitive, slower performance due to joins
May want to use both approaches, especially if supporting multiple
end-user tools.
104. IBM Software Group | WebSphere software
104
Example of Snow flake schema
105. IBM Software Group | WebSphere software
105
Snowflake - Disadvantages
Normalization of dimension makes it difficult for user to
understand
Decreases the query performance because it involves more
joins
Dimension tables are normally smaller than fact tables - space
may not be a major issue to warrant snowflaking
106. IBM Software Group | WebSphere software
106
Data Acquisation
Data Extraction
Data Transformation
Data Loading
106
107. IBM Software Group | WebSphere software
107
Tool Category Products
ETL Tools ETI Extract, Informatica, IBM Visual Warehouse
Oracle Warehouse Builder
OLAP Server Oracle Express Server, Hyperion Essbase, IBM DB2
OLAP Server, Microsoft SQL Server OLAP Services,
Seagate HOLOS, SAS/MDDB
OLAP Tools Oracle Express Suite, Business Objects, Web
Intelligence, SAS, Cognos Powerplay/Impromtu,
KALIDO, MicroStrategy, Brio Query, MetaCube
Data Warehouse Oracle, Informix, Teradata, DB2/UDB, Sybase,
Microsoft SQL Server, RedBricks
Data Mining &
Analysis
SAS Enterprise Miner, IBM Intelligent Miner,
SPSS/Clementine, TCS Tools
Representative DW Tools
108. IBM Software Group | WebSphere software
108
ETL PRODUCTS
CODE BASED ETL TOOLS
GUI BASED ETL TOOLS
108
109. IBM Software Group | WebSphere software
109
CODE BASED ETL TOOLS
SAS ACCESS
SAS BASE
TERADATA ETL TOOLS
1. BTEQ
2. TPUMP
3. FAST LOAD
4. MULTI LOAD
110. IBM Software Group | WebSphere software
110
GUI BASED ETL TOOLS
Informatica
DT/Studio
Data Stage
Business Objects Data Integrator (BODI)
AbInitio
Data Junction
Oracle Warehouse Builder
Microsoft SQL Server Integration Services
IBM DB2 Ware house Center
112. IBM Software Group | WebSphere software
112
Extraction Types
Extraction
Full Extract
Periodic/
Incremental
Extract
113. IBM Software Group | WebSphere software
113
Full Extract
Source System
Full Extract
Data Mart
New data
114. IBM Software Group | WebSphere software
115
Incremental Extract
Data Mart
Source System
Incremental Extract
Existing data
Incremental
Data
115. IBM Software Group | WebSphere software
116
Incremental Extract
Data Mart
Source System
Incremental Extract
New data
Changed data
Existing data
Incremental
Data
116. IBM Software Group | WebSphere software
117
Incremental Extract
Data Mart
Source System
Incremental Extract
New data
Changed data
Existing data updated
using changed data
Incremental
Data
Incremental addition
to data mart
119. IBM Software Group | WebSphere software
120
Types of Data warehouse Loading
Target update types
Insert
Update
120. IBM Software Group | WebSphere software
Types of Data Warehouse Updates
Insert
Full Replace
Selective Replace
Update plus Retain History
Update
Point in Time
Snapshots
New Data
Changed Data
Data Warehouse
Source data Data Staging
121. IBM Software Group | WebSphere software
New Data and Point-In-Time Data Insert
Source data
New data
OR
Point-in-Time
Snapshot
(e.g.. Monthly)
New Data Added to
Existing Data
122. IBM Software Group | WebSphere software
Changed Data Insert
Source data
Changed Data Added to
Existing Data
Changed
data
123. IBM Software Group | WebSphere software
124
DataData
WareWare
househouse
DataData
WareWare
househouse
Enterprise
Data
Warehouse
InfoInfo
AccessAccess
InfoInfo
AccessAccess
Reporting tools
Web
Browsers
OLAP
Mining
ETLETLETLETL
External DataExternal Data
StorageStorage
BusinessBusiness
RequirementRequirement
Map DataMap Data
sourcessources
ReverseReverse
Engg.Engg.
MapMap
Req. toReq. to
OLTPOLTP
OLTPOLTP
SystemSystem
LogicalLogical
ModelingModeling
RefineRefine
ModelModel
Data Warehouse Life cycle
124. IBM Software Group | WebSphere software
125
Project Life Cycle
Software Requirement Specification
High level Design(HLD)
Low level Design(LLD)
Development
Unit Testing
System Integration Testing
Peer Review
User Acceptance Testing
Production
Maintenance
125
126. IBM Software Group | WebSphere software
127
• Data about data and the processes
• Metadata is stored in a data dictionary and repository.
• Insulates the data warehouse from changes in the schema of
operational systems.
• It serves to identify the contents and location of data in the
data warehouse
What is Metadata?
127. IBM Software Group | WebSphere software
128
Share resources
Users
Tools
Document system
Without meta data
Not Sustainable
Not able to fully utilize resource
Why Do You Need Meta Data?
128. IBM Software Group | WebSphere software
The Role of Meta Data in the Data Warehouse
Know what data you have and
You can trust it!
Meta Data enables data to become information, because with it you
129. IBM Software Group | WebSphere software
Meta Data Answers….
How have business definitions and terms changed over time?
How do product lines vary across organizations?
What business assumptions have been made?
How do I find the data I need?
What is the original source of the data?
How was this summarization created?
What queries are available to access the data
130. IBM Software Group | WebSphere software
131
Meta Data Process
Integrated with entire process and data flow
Populated from beginning to end
Begin population at design phase of project
Dedicated resources throughout
Build
Maintain
•Design
•Mapping
•Design
•Mapping
•Extract
•Scrub
•Transform
•Extract
•Scrub
•Transform
•Load
•Index
•Aggregation
•Load
•Index
•Aggregation
•Replication
•Data Set Distribution
•Replication
•Data Set Distribution
•Access & Analysis
•Resource Scheduling & Distribution
•Access & Analysis
•Resource Scheduling & Distribution
Meta DataMeta Data
System MonitoringSystem Monitoring
131. IBM Software Group | WebSphere software
132
Types of ETL Meta Data
.
ETL Meta data
Technical
Meta data
Operational
Meta data
132. IBM Software Group | WebSphere software
Data Warehouse Meta data
This Meta data stores descriptive information about the physical
implementation details of data warehouse.
Source Meta data
This Meta data stores information about the source data and the mapping of source
data to data warehouse data
Classification of ETL Meta Data
133. IBM Software Group | WebSphere software
Transformations & Integrations.
This Meta data describes comprehensive information about the Transformation and
loading.
Processing Information
This Meta data stores information about the activities involved in the processing of data
such as scheduling and archives etc
End User Information
This Meta data records information about the user profile and security.
ETL Meta Data
134. IBM Software Group | WebSphere software
135
ETL -Planning for the Movement
The following may be helpful for planning the movement
Develop a ETL plan
Specifications
Implementation