This document provides an overview of data warehousing concepts including:
- The key differences between operational systems and data warehouses in terms of design, usage, and data characteristics.
- The benefits of implementing a data warehouse for business intelligence and decision making.
- Common data warehousing architectures and approaches including top-down, bottom-up, and hybrid approaches.
- Fundamental data modeling techniques for data warehouses including entity-relationship modeling and dimensional modeling.
4. IBM Software Group | WebSphere software
4
Course Roadmap
âĸ Why we use Data warehousing
âĸ Difference between Operational System and Data Warehouse
âĸ Introduction to Data warehousing
âĸ Data Warehousing Approaches
âĸ Data Warehouse Technical Architecture
âĸ Data Modelling concepts
âĸ Operational Data Store
âĸ Schema Design of Data warehouse
âĸ Data Acquisation
âĸ ETL Products
âĸ Project Life Cycle
5. IBM Software Group | WebSphere software
5
Why We Need Data Warehousing ?
ī§ Better business intelligence for end-users
ī§ Reduction in time to locate, access, and analyze information
ī§ Consolidation of disparate information sources
ī§ To Store Large Volumes of Historical Detail Data from Mission Critical Applications
ī§ Strategic advantage over competitors
ī§ Faster time-to-market for products and services
ī§ Replacement of older, less-responsive decision support systems
ī§ Reduction in demand on IS to generate reports
6. IBM Software Group | WebSphere software
6
What is an Operational System?
ī§ Operational systems are just what their name implies; they are the systems that
help us run the day-to-day enterprise operations.
ī§ These are the backbone systems of any enterprise, such as order entry inventory
etc.
ī§ The classic examples are airline reservations, credit-card authorizations, and ATM
withdrawals etc.,
7. IBM Software Group | WebSphere software
7
Characteristics of Operational Systems
âĸ Continuous availability
âĸ Predefined access paths
âĸ Transaction integrity
âĸ Volume of transaction - High
âĸ Data volume per query - Low
âĸ Used by operational staff
âĸ Supports day to day control operations
âĸ Large number of users
8. IBM Software Group | WebSphere software
8
OLTP Vs Data Warehouse
Operational System Data Warehouse
Transaction Processing Query Processing
Predictable CPU Usage Random CPU Usage
Time Sensitive History Oriented
Operator View Managerial View
Normalized Efficient
Design for TP
Denormalized Design for
Query Processing
9. IBM Software Group | WebSphere software
9
OLTP Vs Warehouse
Operational System Data Warehouse
Designed for Atmocity,
Consistency, Isolation and
Durability
Designed for quite or static
database
Organized by transactions
(Order, Input, Inventory)
Organized by subject
(Customer, Product)
Relatively smaller database Large database size
Many concurrent users Relatively few concurrent
users
Volatile Data Non Volatile Data
10. IBM Software Group | WebSphere software
10
Operational System Data Warehouse
Stores all data Stores relevant data
Performance Sensitive Less Sensitive to performance
Not Flexible Flexible
Efficiency Effectiveness
11. IBM Software Group | WebSphere software
11
What is a Data Warehouse ?
ī§ Data WarehouseData Warehouse is a
ī§ Subject-Oriented
ī§ Integrated
ī§ Time-Variant
ī§ Non-volatile
WH Inmon - Regarded As Father Of Data WarehousingWH Inmon - Regarded As Father Of Data Warehousing
13. IBM Software Group | WebSphere software
13
13
Subject Oriented Analysis
Data Warehouse StorageTransactional Storage
SalesSales
CustomersCustomers
ProductsProducts
Entry
Sales Rep
Quantity Sold
Part Number
Date
Customer Name
Product Description
Unit Price
Mail Address
Process Oriented Subject Oriented
14. IBM Software Group | WebSphere software
14
14
Integration of Data
Data Warehouse StorageTransactional Storage
Appl. A - M, F
Appl. B - 1, 0
Appl. C - X, Y
Appl. A - pipeline cm.
Appl. B - pipeline inches
Appl. C - pipeline mcf
Appl. A - balance dec(13,2)
Appl. B - balance PIC 9(9)V99
Appl. C - balance float
Appl. A - bal-on-hand
Appl. B - current_balance
Appl. C - balance
Appl. A - date (Julian)
Appl. B - date (yymmdd)
Appl. C - date (absolute)
M, F
pipeline cm
balance dec(13, 2)
balance
date (Julian)
Integration
Encoding
Unit of
Attributes
Physical
Attributes
Naming
Conventions
Data
Consistency
15. IBM Software Group | WebSphere software
15
15
Load
Access
Mass Load / Access of DataRecord-by-Record Data Manipulation
Insert
Access
Insert
Change
Delete
Change
Volatile Non-Volatile
Volatility of Data
Data Warehouse StorageTransactional Storage
16. IBM Software Group | WebSphere software
16
16
Time Variant Data Analysis
Data Warehouse StorageTransactional Storage
Current Data Historical Data
0
5
10
15
20
Sales ( in lakhs
)
January February March
Year97
Sales ( Region , Year - Year 97 - 1st Qtr)
East
West
North
17. IBM Software Group | WebSphere software
Load/
Update
Consistent Points in Time
Updated constantly
Data changes according to
need, not a fixed schedule
Added to regularly, but loaded data
is rarely directly changed
Does NOT mean the Data
warehouse is never updated or
never changes!!
Constant Change
Operational systems
Database
Data warehouse
Datawarehouse- Differences from Operational
Systems
Insert
Insert
Update
Initial Load
Incremental Load
Incremental Load
Update
Delete
19. IBM Software Group | WebSphere software
19
DW Implementation Approaches
ī§ Top Down
ī§ Bottom-up
ī§ Combination of both
ī§ Choices depend on:
ī´current infrastructure
ī´resources
ī´architecture
ī´ROI
ī´Implementation speed
20. IBM Software Group | WebSphere software
20
Heterogeneous Source Systems
Staging
Common Staging interface Layer
EDW- âTop DownâApproach
Data mart bus architecture Layer
Enterprise Datawarehouse
Source
1
Source
2
Source
3
Incremental Architected data marts
DM 1 DM 3DM 2
21. IBM Software Group | WebSphere software
21
Heterogeneous Source Systems
Staging
Common Staging interface Layer
EDW- âBottom upâApproach
Data mart bus architecture Layer
Source
1
Source
2
Source
3
Incremental Architected data marts
DM 1 DM 3DM 2
Enterprise Datawarehouse
22. IBM Software Group | WebSphere software
22
Source System Data Staging Area Presentation Area
Services:
Transform from
source-to-Target
Maintain Conform
Dimensions
No user query
support
Data Store:
Flat files or
relational tables
Design Goals:
Staging
Throughput
integrity/
consistency
Load
Access
Ad Hoc Query Tools
Report Writers
Analytic Applications
Modeling:
Forecasting
Scoring
Data
Mining
Data Mart #1
Dimensional
Atomic AND
summery data
Business
Process Centric
Design Goals:
Easy-of -use
Query
Performance
Data Mart #2
Data Mart #.....
Data Mart Bus:
Conformed facts and dims
Extract
Extract
Extract
Data Access Tools
Independent Data Marts: Ralph Kimballâs Ideology
Ralph Kimballâ Approach
23. IBM Software Group | WebSphere software
23
âĸE/R Design or Flat File
âĸRetain History Needed
for
regular processing
âĸNo end user access
âĸ Dimensional
âĸTransaction &
Summary data
âĸData Mart Single
subject area
(i.e. Fact table)
âĸMultiple Marts May
exist in a
Single Database
Instance
Bottom Up Approach
Staging Data Store
Data Warehouse
Data Mart Data Mart Data Mart
Data Mart Data MartData Mart
âĸIntegrated Data
âĸTimely User Access
âĸConformed Dimensions
âĸSingle Process to
Build Dimension
25. IBM Software Group | WebSphere software
25
Bill Inmonâ Approach
Source
System
Data Staging
Area
Presentation
Area
âEnterprise Data
Warehouseâ
Normalized
tables
Atomic Data
User query
support to
atomic data
Extract
Extract
Extract
Load
Data Mart #1
Dimensional
summery data
Departmental
Centric
Access
Access
Data Access
Tools
Data Mart #2
Data Mart #...
ETL
Dependent Data Marts: Bill Inmonâs Ideology
DWH
26. IBM Software Group | WebSphere software
26
Top Down Approach
âĸ Raw Input Data
âĸ E/R Model
âĸ Subject Areas
âĸ Transaction Level Detail
âĸ Historical Persistency As justified- Archive
for Retrieval if Needed
âĸ Most are dimensional
âĸ Data Mart Design by Business
Function
âĸ Summary Level Data
âĸ
Data Mart Data Mart
Staging Data Store
Data
Warehouse
Data Mart
Data
Mart
Flat
File
âĸIntegrated Data
âĸTimely user Access
âĸSingle Process to build dimension
27. IBM Software Group | WebSphere software
27
DW Implementation Approaches
Top Down
ī§ More planning and design initially
ī§ Involve people from different work-
groups, departments
ī§ Data marts may be built later from
Global DW
ī§ Overall data model to be decided up-
front
Bottom Up
ī§ Can plan initially without waiting for
global infrastructure
ī§ built incrementally
ī§ can be built before or in parallel with
Global DW
ī§ Less complexity in design
28. IBM Software Group | WebSphere software
28
DW Implementation Approaches
Top Down
ī§ Consistent data definition and
enforcement of business rules across
enterprise
ī§ High cost, lengthy process, time
consuming
ī§ Works well when there is centralized IS
department responsible for all H/W and
resources
Bottom Up
ī§ Data redundancy and
inconsistency between data marts
may occur
ī§ Integration requires great planning
ī§ Less cost of H/W and other
resources
ī§ Faster pay-back
30. IBM Software Group | WebSphere software
30
Prod
Mkt
HR
Fin
Acctg
Data Sources
Transaction Data
IBM
IMS
VSAM
Oracle
Sybase
ETL Software Data Stores Data Analysis
Tools and
Applications
Users
Other Internal Data
ERP SAP
Clickstream Informix
Web Data
External Data
Demographic Harte-
Hanks
S
T
A
G
I
N
G
A
R
E
A
O
P
E
R
A
T
I
O
N
A
L
D
A
T
A
S
T
O
R
E
Ascential
Extract
Sagent
SAS
Clean/Scrub
Transform
Firstlogic
Load
DATASTAGE
Data Marts
Teradata
IBM
Data
Warehouse
Meta
Data
Finance
Marketing
Sales
Essbase
Microsoft
ANALYSTS
MANAGERS
EXECUTIVES
OPERATIONAL
PERSONNEL
CUSTOMERS/
SUPPLIERS
SQL
Cognos
SAS
Queries,Reporting,
DSS/EIS,
Data Mining
Micro Strategy
Siebel
Business
Objects
Web
Browser
31. IBM Software Group | WebSphere software
31
Benefits of DWH
To formulate effective business, marketing
and sales strategies.
To precisely target promotional activity.
To discover and penetrate new markets.
To successfully compete in the marketplace
from a position of informed strength.
To build predictive rather than retrospective models.
33. IBM Software Group | WebSphere software
33
Data Modeling
īļ WHAT IS A DATA MODEL?
īŧA data model is an abstraction of some aspect of the real
world (system).
īļ WHY A DATA MODEL?
âĸ Helps to visualize the business
âĸ A model is a means of communication.
âĸ Models help elicit and document requirements.
âĸ Models reduce the cost of change.
âĸ Model is the essence of DW architecture based on which
DW will be implemented
34. IBM Software Group | WebSphere software
34
STEPS in DATA MODELING
Problem & scope definition
Requirement Gathering
Analysis
Logical Database Design
Deciding Database
Physical Database design
Schema Generation
35. IBM Software Group | WebSphere software
35
Levels of modeling
ī§ Conceptual modeling
ī´Describe data requirements from a
business point of view without technical
details
ī§ Logical modeling
ī´Refine conceptual models
ī´Data structure oriented, platform
independent
ī§ Physical modeling
ī´Detailed specification of what is physically
implemented using specific technology
36. IBM Software Group | WebSphere software
36
Modeling Techniques
ī§ Entity-Relationship Modeling
ī´Traditional modeling technique
ī´Technique of choice for OLTP
ī´Suited for corporate data warehouse
ī§ Dimensional Modeling
ī´Analyzing business measures in the specific business context
ī´Helps visualize very abstract business questions
ī´End users can easily understand and navigate the data
structure
37. IBM Software Group | WebSphere software
37
ī§ Relationship
ī´Relationship between entities - structural interaction
and association
ī´described by a verb
ī´Cardinality
ī§ 1-1
ī§ 1-M
ī§ M-M
ī´Example : Books belong to Printed Media
Entity-Relationship Modeling - Basic Concepts
38. IBM Software Group | WebSphere software
38
Entity-Relationship Modeling - Basic Concepts
ī§ Attributes
ī´Characteristics and properties of entities
ī´Example :
ī§ Book Id, Description, book category are
attributes of entity âBookâ
ī´Attribute name should be unique and self-
explanatory
ī´Primary Key, Foreign Key, Constraints are defined
on Attributes
39. IBM Software Group | WebSphere software
Review of Logical Modeling Terms & Symbols
ī§ Entities define specific groups of information
Sales Organization
Sales Org ID
Distribution Channel
Entity
40. IBM Software Group | WebSphere software
Review of Logical Modeling Terms & Symbols
ī§ One or more attribute uniquely identifies an instance of an
entity
Sales Organization
Sales Org ID
Distribution Channel
Identifier
41. IBM Software Group | WebSphere software
Review of Logical Modeling Terms & Symbols
ī§ The logical model identifies relationships between
entities
Sales Detail
Sales Record ID
Sales Rep
Sales Rep ID
Relationship
{
43. IBM Software Group | WebSphere software
Logical Data Model
Sales Detail
Sales Record ID
Customer
Customer ID
Product
Product SKU
Suppliers
Supplier ID
Manufacturing Group
Manufacturing Org ID
Factory
Factory ID
Sales Organization
Sales Org ID
Distribution Channel
Sales Rep
Sales Rep ID
Retail
Market
Product Sales Plan
Plan ID
Wholesale
Industry
48. IBM Software Group | WebSphere software
48
Dimensional Modeling
ī§ Dimensional modeling uses three basic concepts : measures,
facts, dimensions.
ī§ Is powerful in representing the requirements of the business
user in the context of database tables.
ī§ Focuses on numeric data, such as values counts, weights,
balances and occurences.
49. IBM Software Group | WebSphere software
49
What is a Facts
ī§ A fact is a collection of related data items, consisting of measures
and context data.
ī§ Each fact typically represents a business item, a business
transaction, or an event that can be used in analyzing the business
or business process.
ī§ Facts are measured, âcontinuously valuedâ, rapidly changing
information. Can be calculated and/or derived.
ī§ Granularity
The level of detail of data contained in the data warehouse
e.g. Daily item totals by product, by store
50. IBM Software Group | WebSphere software
50
Types of Facts
ī§ Additive
ī´Able to add the facts along all the dimensions
ī´Discrete numerical measures eg. Retail sales in $
ī§ Semi Additive
ī´Snapshot, taken at a point in time
ī´Measures of Intensity
ī´Not additive along time dimension eg. Account balance, Inventory
balance
ī´Added and divided by number of time period to get a time-average
ī§ Non Additive
ī´Numeric measures that cannot be added across any dimensions
ī´Intensity measure averaged across all dimensions eg. Room
temperature
ī´Textual facts - AVOID THEM
51. IBM Software Group | WebSphere software
51
Dimensions
ī§ A dimension is a collection of members or units of the same type
of views.
ī§ Dimensions determine the contextual background for the facts.
ī§ Dimensions represent the way business people talk about the
data resulting from a business process, e.g., who, what, when,
where, why, how
52. IBM Software Group | WebSphere software
52
52
Dimensional Hierarchy
World
America AsiaEurope
USA
FL
Canada Argentina
GA VA CA WA
TampaMiami Orlando Naples
Continent Level
State Level
City Level
World Level
Country Level
ParentRelation
Dimension Member /
Business Entity
Geography Dimension
Attributes: Population, Touristâs Place
53. IBM Software Group | WebSphere software
53
Dimensions Types
ī§ Conformed Dimension
ī§ Junk Dimension
ī§ Fast Changing Dimension
ī§ Role Playing Dimension
ī§ âGarbageâ Dimension
ī§ Slowly Changing Dimension
ī§ Degenerated Dimension
53
54. IBM Software Group | WebSphere software
54
What is a Slowly Changing Dimension?
ī§ Although dimension tables are typically static lists, most dimension tables do change over
time.
ī§ Since these changes are smaller in magnitude compared to changes in fact tables, these
dimensions are known as slowly growing or slowly changing dimensions.
55. IBM Software Group | WebSphere software
55
Slowly Changing Dimension -Classification
Slowly changing dimensions are classified into three different
types
ī§ TYPE I
ī§ TYPE II
ī§ TYPE III
56. IBM Software Group | WebSphere software
56
Slowly Changing Dimensions Type I
Shane
Name
Shane@xyz.com1001
EmailEmp id
Shane
Name
Shane@xyz.com1001
EmailEmp id
Shane
Name
Shane@
abc.co.in
1001
EmailEmp id
Shane
Name
Shane@
abc.co.in
1001
EmailEmp id
Source
Source Target
Target
Shane@
xyz.com
57. IBM Software Group | WebSphere software
57
Slowly Changing Dimensions Type II
Shane
Name
Shane@xyz.com10
EmailEmp id
Shane@x
yz.
com
Email
Shane
Name
10
Emp id
1000
PM_PRI
MARYK
EY
0
PM_VER
SION_N
UMBER
Source Target
59. IBM Software Group | WebSphere software
59
Slowly Changing Dimensions -Versioning
Shane
Name
Shane@
abc.co.in
10
EmailEmp id
Source
Target
0Shane@
xyz.com
Shane101000
1Shane@
abc.co.in
Shane101001
EmailNameEmp idPM_PRIMA
RYKEY
PM_VERSION_NUMBER
60. IBM Software Group | WebSphere software
60
Slowly Changing Dimensions -Versioning
Shane
Name
Shane@
abc.com
10
EmailEmp id
Source
Target
1Shane@
abc.co.in
Shane101001
2Shane@
abc.com
Shane101003
0Shane@
xyz.com
Shane101000
EmailNameEmp idPM_PRIM
ARYKEY
PM_VERSION_NUM
BER
61. IBM Software Group | WebSphere software
61
Slowly Changing Dimensions Type II -
Flag
Shane
Name
Shane@xyz.com10
EmailEmp id
Shane@
xyz.
com
Email
Shane
Name
10
Emp id
1000
PM_PR
IMAR
YKEY
Y
PM_CUR
RENT_FL
AG
Source
Target
62. IBM Software Group | WebSphere software
62
Slowly Changing Dimensions - Flag Current
Shane
Name
Shane@
abc.co.in
10
EmailEmp id
Source
Target
NShane@
xyz.com
Shane101000
YShane@
abc.co.in
Shane101001
EmailNameEmp idPM_PRIMA
RYKEY
PM_CURRENT_FLAG
63. IBM Software Group | WebSphere software
63
Slowly Changing Dimensions - Flag Current
Shane
Name
Shane@
abc.com
10
EmailEmp id
Source
Target
NShane@
abc.co.in
Shane101001
YShane@
abc.com
Shane101003
NShane@
xyz.com
Shane101000
EmailNameEmp idPM_PRIMA
RYKEY
PM_CURRENT_FLAG
65. IBM Software Group | WebSphere software
65
Slowly Changing Dimensions Type II
Shane
Name
Shane@xyz.c
om
10
EmailEmp id
01/01/00
PM_BEG
IN_DAT
E
Shane@x
yz.com
Email
Shane
Name
10
Emp id
1000
PM_PRI
MARYK
EY
PM_EN
D_DATE
Source
Target
66. IBM Software Group | WebSphere software
66
Slowly Changing Dimensions -Effective Date
Shane
Name
Shane@
abc.co.in10
Email
Emp id
Source
Target
03/01/00
01/01/00
PM_BEGIN_D
ATE
03/01/00Shane@x
yz.com
Shane101000
Shane@
abc.co.in
Shane101001
EmailNameEmp idPM_PRIMAR
YKEY
PM_END_D
ATE
67. IBM Software Group | WebSphere software
67
Slowly Changing Dimensions - Effective Date
Shane
Name
Shane@
abc.com10
EmailEmp id
Source
Target
05/02/00
03/01/00
01/01/00
PM_BEGIN_D
ATE
05/02/00Shane@
abc.co.in
Shane101001
Shane@
abc.com
Shane101003
03/01/00Shane@
xyz.com
Shane101000
EmailNameEmp idPM_PRIM
ARYKEY
PM_END_DA
TE
69. IBM Software Group | WebSphere software
69
Slowly Changing Dimensions Type III
Shane
Name
Shane@xyz.c
om
10
EmailEmp id
PM_Prev_
Column
Name
Shane@xyz.
com
Email
Shane
Name
10
Emp id
1
PM_PRI
MARYKE
Y
01/01/00
PM_EFFEC
T_DATE
Source
Target
70. IBM Software Group | WebSphere software
70
Slowly Changing Dimensions Type III
Shane
Name
Shane@
abc.co.in10
EmailEmp id
Source
Target
Shane@xyz.co
m
PM_Prev_Colu
mnName
01/02/00Shane@
abc.co.in
Shane101
EmailNameEmp idPM_PRIMAR
YKEY
PM_EFFEC
T_DATE
71. IBM Software Group | WebSphere software
71
Slowly Changing Dimensions Type III
Shane
Name
Shane@
abc.com10
EmailEmp id
Source
Target
Shane@
abc.co.in
PM_Prev_Colu
mnName
01/03/00Shane@
abc.com
Shane101
EmailNameEmp idPM_PRIM
ARYKEY
PM_EFFECT_
DATE
72. IBM Software Group | WebSphere software
72
Degenerate Dimension
ī§ Dimension keys in fact table without corresponding dimension tables are
called Degenerate Dimensions
ī§ Purpose of Degenerate Dimensions
1. Generally used when each record in fact represents transaction line item
2. Useful for grouping transaction line items belonging to a single
transaction
73. IBM Software Group | WebSphere software
73
Fast Changing Dimension
A fast changing dimension is a dimension whose attribute or
attributes for a record (row) change rapidly over time.
1. Example: Age of associates, Income, Daily balance etc.
2. Technique to handle fast changing dimension: Create band
tables
74. IBM Software Group | WebSphere software
74
Role Playing Dimension
A single dimension which is expressed differently in a fact table using
views is called a role-playing dimension. This can be achieved by
creating views on dimension table.
75. IBM Software Group | WebSphere software
75
Conformed Dimension
A conformed dimension means the same thing to
each fact table to which it can be joined.
Typically, dimension tables that are referenced or are
likely to be referenced by multiple fact tables
(multiple dimensional models) are called conformed
dimensions
.
76. IBM Software Group | WebSphere software
76
Conformed Dimension Option #1
īą Identical dimensions with same keys, labels, definitions and Values
Sales Schema
Inventory Schema
SALES Facts
DATE KEY
PRODUCT KEY
STORE KEY
PROMO KEY
Product Desc
Brand Desc
Category Desc
PRODUCT KEY
INVENTORY
Facts
DATE KEY
PRODUCT KEY
STORE KEY
Product Desc
Brand Desc
Category Desc
PRODUCT KEY
77. IBM Software Group | WebSphere software
77
Conformed Dimension Option #2
īąSubset of base dimension with common labels, definitions
and values
Sales
Schema
Forecast
Schema
SALES $
DATE KEY
PRODUCT KEY
STORE KEY
PROMO KEY
Product Desc
Brand Desc
Category Desc
PRODUCT KEY
DATE KEY
Day-of-week
Week Desc
Month Desc
SALES $
MONTH KEY
BRAND KEYBrand Desc
Category Desc
BRAND KEY MONTH KEY
Month Desc
BRAND KEY Brand Desc Category Desc
12345 Cherriors Cereal
PROD KEY Prod Desc Brand Desc Category Desc
12345 Cherriors 10 Cherriors Cereal
78. IBM Software Group | WebSphere software
78
âGarbageâ Dimension
A garbage dimension is a dimension that consists of low-cardinality columns
such as codes, indicators, and status flags.
Approach to handle Garbage dimension:
âĸ Put the new attributes into existing dimension tables.
âĸ Put the new attributes into the fact table.
âĸ Create new separate dimension tables garbage dimension
âĸ Create a separate âGarbage Dimensionâ table
79. IBM Software Group | WebSphere software
79
Junk Dimensions
īĩ Whether to use junk dimension
īĩ5 indicators, each has 3 values -> 243 (35
) rows
īĩ5 indicators, each has 100 values -> 100 million (1005
) rows
īĩ When to insert rows in the dimension
80. IBM Software Group | WebSphere software
80
Factless Fact Tables
The two types of factless fact tables are:
ī§ Coverage tables
ī§ Event tracking tables
81. IBM Software Group | WebSphere software
81
Factless Fact Tables - Coverage Tables
Coverage tables are required when a primary fact
table is sparse
Example: Tracking products in a store that did not sell
83. IBM Software Group | WebSphere software
83
Factless Fact Tables - Event Tracking
These tables are used for tracking a event:
Example: Tracking student attendance
84. IBM Software Group | WebSphere software
84
Fact Constellation
ī§ Fact constellations: Multiple fact tables share dimension tables,viewed as
a collection of stars, therefore called galaxy schema or fact constellation
85. IBM Software Group | WebSphere software
85
What is a Data mart?
ī§ Data mart is a decentralized subset of data found either in a data warehouse
or as a standalone subset designed to support the unique business unit
requirements of a specific decision-support system.
ī§ Data marts have specific business-related purposes such as measuring the
impact of marketing promotions, or measuring and forecasting sales
performance etc,.
Data Mart
Data Mart
Enterprise
Data Warehouse
86. IBM Software Group | WebSphere software
86
Data marts - Main Features
Main Features:
ī§ Low cost
ī§ Controlled locally rather than centrally, conferring power on the user group.
ī§ Contain less information than the warehouse
ī§ Rapid response
ī§ Easily understood and navigated than an enterprise data warehouse.
ī§ Within the range of divisional or departmental budgets
88. IBM Software Group | WebSphere software
88
Datamart Advantages :
ī§ Typically single subject area and fewer dimensions
ī§ Limited feeds
ī§ Very quick time to market (30-120 days to pilot)
ī§ Quick impact on bottom line problems
ī§ Focused user needs
ī§ Limited scope
ī§ Optimum model for DW construction
ī§ Demonstrates ROI
ī§ Allows prototyping
Advantages of Datamart over Datawarehouse
89. IBM Software Group | WebSphere software
89
Data Mart disadvantages :
âĸ Does not provide integrated view of business information.
âĸ Uncontrolled proliferation of data marts results in redundancy
âĸ More number of data marts complex to maintain
âĸ Scalability issues for large number of users and increased data volume
Disadvantages of Data Mart
90. IBM Software Group | WebSphere software
90
90
Data marts
âĸ Embedded data marts are marts that are stored within
the central DW. They can be stored relationally as files or
cubes.
âĸ Dependent data marts are marts that are fed directly by
the DW, sometimes supplemented with other feeds, such
as external data.
âĸ Independent data marts are marts that are fed directly
by external sources and do not use the DW.
DM - Types
93. IBM Software Group | WebSphere software
93
Why We Need Operational Data Store?
Need
ī§ To obtain a âsystem of recordâ that contains the best data that
exists in a legacy environment as a source of information
ī§ Best here implies data to be
ī´Complete
ī´Up to date
ī´Accurate
ī§ In conformance with the organizationâs information model
94. IBM Software Group | WebSphere software
ī§ ODS data resolves data integration issues
ī§ Data physically separated from production
environment to insulate it from the processing
demands of reporting and analysis
ī§ Access to current data facilitated.
Operational Data Store - Insulated from OLTP
Tactical
Analysis
OLTP Server
ODS
95. IBM Software Group | WebSphere software
95
ī§ Detailed data
ī´ Records of Business Events
(e.g. Orders capture)
ī§ Data from heterogeneous sources
ī§ Does not store summary data
ī§ Contains current data
Operational Data Store - Data
97. IBM Software Group | WebSphere software
97
ODS- Benefits
ī§ Integrates the data
ī§ Synchronizes the structural differences in data
ī§ High transaction performance
ī§ Serves the operational and DSS environment
ī§ Transaction level reporting on current data
Flat
files
Relational
Database
Operational
Data Store
60,5.2,âJOHNâ
72,6.2,âDAVIDâ
Excel files
98. IBM Software Group | WebSphere software
ī§ Update schedule - Daily or less
time frequency
ī§ Detail of Data is mostly between
30 and 90 days
ī§ Addresses operational needs
ī§ Weekly or greater time frequency
ī§ Potentially infinite history
ī§ Address strategic needs
Operational Data Store- Update schedule
ODS
Data
Data warehouse
Data
100. IBM Software Group | WebSphere software
100
OLTP Vs ODS Vs DWH
Characteristic OLTP ODS Data Warehouse
Data redundancy Non-redundant
within system;
Unmanaged
redundancy among
systems
Somewhat
redundant with
operational
databases
Managed
redundancy
Data stability Dynamic Somewhat dynamic Static
Data update Field by field Field by field Controlled batch
Data usage Highly structured,
repetitive
Somewhat
structured, some
analytical
Highly
unstructured,
heuristic or
analytical
Database size Moderate Moderate Large to very large
Database
structure stability
Stable Somewhat stable Dynamic
101. IBM Software Group | WebSphere software
101
Star Schema Design
ī´Single fact table surrounded by denormalized dimension
tables
ī´The fact table primary key is the composite of the
foreign keys (primary keys of dimension tables)
ī´Fact table contains transaction type information.
ī´Many star schemas in a data mart
ī´Easily understood by end users, more disk storage
required
103. IBM Software Group | WebSphere software
103
Snowflake Schema
ī´Single fact table surrounded by normalized dimension
tables
ī´Normalizes dimension table to save data storage space.
ī´When dimensions become very very large
ī´Less intuitive, slower performance due to joins
ī§ May want to use both approaches, especially if supporting multiple
end-user tools.
104. IBM Software Group | WebSphere software
104
Example of Snow flake schema
105. IBM Software Group | WebSphere software
105
Snowflake - Disadvantages
ī§ Normalization of dimension makes it difficult for user to
understand
ī§ Decreases the query performance because it involves more
joins
ī§ Dimension tables are normally smaller than fact tables - space
may not be a major issue to warrant snowflaking
106. IBM Software Group | WebSphere software
106
Data Acquisation
ī§ Data Extraction
ī§ Data Transformation
ī§ Data Loading
106
107. IBM Software Group | WebSphere software
107
Tool Category Products
ETL Tools ETI Extract, Informatica, IBM Visual Warehouse
Oracle Warehouse Builder
OLAP Server Oracle Express Server, Hyperion Essbase, IBM DB2
OLAP Server, Microsoft SQL Server OLAP Services,
Seagate HOLOS, SAS/MDDB
OLAP Tools Oracle Express Suite, Business Objects, Web
Intelligence, SAS, Cognos Powerplay/Impromtu,
KALIDO, MicroStrategy, Brio Query, MetaCube
Data Warehouse Oracle, Informix, Teradata, DB2/UDB, Sybase,
Microsoft SQL Server, RedBricks
Data Mining &
Analysis
SAS Enterprise Miner, IBM Intelligent Miner,
SPSS/Clementine, TCS Tools
Representative DW Tools
108. IBM Software Group | WebSphere software
108
ETL PRODUCTS
ī§ CODE BASED ETL TOOLS
ī§ GUI BASED ETL TOOLS
108
109. IBM Software Group | WebSphere software
109
CODE BASED ETL TOOLS
ī§ SAS ACCESS
ī§ SAS BASE
ī§ TERADATA ETL TOOLS
1. BTEQ
2. TPUMP
3. FAST LOAD
4. MULTI LOAD
110. IBM Software Group | WebSphere software
110
GUI BASED ETL TOOLS
ī§ Informatica
ī§ DT/Studio
ī§ Data Stage
ī§ Business Objects Data Integrator (BODI)
ī§ AbInitio
ī§ Data Junction
ī§ Oracle Warehouse Builder
ī§ Microsoft SQL Server Integration Services
ī§ IBM DB2 Ware house Center
112. IBM Software Group | WebSphere software
112
Extraction Types
Extraction
Full Extract
Periodic/
Incremental
Extract
113. IBM Software Group | WebSphere software
113
Full Extract
Source System
Full Extract
Data Mart
New data
114. IBM Software Group | WebSphere software
115
Incremental Extract
Data Mart
Source System
Incremental Extract
Existing data
Incremental
Data
115. IBM Software Group | WebSphere software
116
Incremental Extract
Data Mart
Source System
Incremental Extract
New data
Changed data
Existing data
Incremental
Data
116. IBM Software Group | WebSphere software
117
Incremental Extract
Data Mart
Source System
Incremental Extract
New data
Changed data
Existing data updated
using changed data
Incremental
Data
Incremental addition
to data mart
119. IBM Software Group | WebSphere software
120
Types of Data warehouse Loading
ī§ Target update types
ī´Insert
ī´Update
120. IBM Software Group | WebSphere software
Types of Data Warehouse Updates
Insert
Full Replace
Selective Replace
Update plus Retain History
Update
Point in Time
Snapshots
New Data
Changed Data
Data Warehouse
Source data Data Staging
121. IBM Software Group | WebSphere software
New Data and Point-In-Time Data Insert
Source data
New data
OR
Point-in-Time
Snapshot
(e.g.. Monthly)
New Data Added to
Existing Data
122. IBM Software Group | WebSphere software
Changed Data Insert
Source data
Changed Data Added to
Existing Data
Changed
data
123. IBM Software Group | WebSphere software
124
DataData
WareWare
househouse
DataData
WareWare
househouse
Enterprise
Data
Warehouse
InfoInfo
AccessAccess
InfoInfo
AccessAccess
Reporting tools
Web
Browsers
OLAP
Mining
ETLETLETLETL
External DataExternal Data
StorageStorage
BusinessBusiness
RequirementRequirement
Map DataMap Data
sourcessources
ReverseReverse
Engg.Engg.
MapMap
Req. toReq. to
OLTPOLTP
OLTPOLTP
SystemSystem
LogicalLogical
ModelingModeling
RefineRefine
ModelModel
Data Warehouse Life cycle
124. IBM Software Group | WebSphere software
125
Project Life Cycle
ī§ Software Requirement Specification
ī§ High level Design(HLD)
ī§ Low level Design(LLD)
ī§ Development
ī§ Unit Testing
ī§ System Integration Testing
ī§ Peer Review
ī§ User Acceptance Testing
ī§ Production
ī§ Maintenance
125
126. IBM Software Group | WebSphere software
127
âĸ Data about data and the processes
âĸ Metadata is stored in a data dictionary and repository.
âĸ Insulates the data warehouse from changes in the schema of
operational systems.
âĸ It serves to identify the contents and location of data in the
data warehouse
What is Metadata?
127. IBM Software Group | WebSphere software
128
ī§ Share resources
ī´ Users
ī´ Tools
ī§ Document system
ī§ Without meta data
ī´ Not Sustainable
ī´ Not able to fully utilize resource
Why Do You Need Meta Data?
128. IBM Software Group | WebSphere software
The Role of Meta Data in the Data Warehouse
ī§ Know what data you have and
ī§ You can trust it!
Meta Data enables data to become information, because with it you
129. IBM Software Group | WebSphere software
Meta Data AnswersâĻ.
How have business definitions and terms changed over time?
How do product lines vary across organizations?
What business assumptions have been made?
How do I find the data I need?
What is the original source of the data?
How was this summarization created?
What queries are available to access the data
130. IBM Software Group | WebSphere software
131
Meta Data Process
ī§ Integrated with entire process and data flow
ī´ Populated from beginning to end
ī´ Begin population at design phase of project
ī´ Dedicated resources throughout
ī§ Build
ī§ Maintain
âĸDesign
âĸMapping
âĸDesign
âĸMapping
âĸExtract
âĸScrub
âĸTransform
âĸExtract
âĸScrub
âĸTransform
âĸLoad
âĸIndex
âĸAggregation
âĸLoad
âĸIndex
âĸAggregation
âĸReplication
âĸData Set Distribution
âĸReplication
âĸData Set Distribution
âĸAccess & Analysis
âĸResource Scheduling & Distribution
âĸAccess & Analysis
âĸResource Scheduling & Distribution
Meta DataMeta Data
System MonitoringSystem Monitoring
131. IBM Software Group | WebSphere software
132
Types of ETL Meta Data
.
ETL Meta data
Technical
Meta data
Operational
Meta data
132. IBM Software Group | WebSphere software
ī§ Data Warehouse Meta data
This Meta data stores descriptive information about the physical
implementation details of data warehouse.
ī§ Source Meta data
This Meta data stores information about the source data and the mapping of source
data to data warehouse data
Classification of ETL Meta Data
133. IBM Software Group | WebSphere software
ī§ Transformations & Integrations.
This Meta data describes comprehensive information about the Transformation and
loading.
ī§ Processing Information
This Meta data stores information about the activities involved in the processing of data
such as scheduling and archives etc
ī§ End User Information
This Meta data records information about the user profile and security.
ETL Meta Data
134. IBM Software Group | WebSphere software
135
ETL -Planning for the Movement
The following may be helpful for planning the movement
ī§ Develop a ETL plan
ī§ Specifications
ī§ Implementation