This document provides an introduction to data warehousing. It discusses how data warehousing evolved from OLTP systems to better support decision making and analytics. Key aspects covered include the definition of a data warehouse, why they are needed, examples of common uses, dimensional modeling concepts like star schemas and slowly changing dimensions, and the responsibilities of data warehouse managers.

1. An Introduction
to Data
Warehousing
Presented by Animesh Srivastava

2. In the beginning..
 In the early 80s the concept of RDBMS ushered in an era of improved access to
the valuable information contained deep within data
 Our need for information grew exponentially and we needed a solution for an
efficient Decision Supporting System to run the business(OLAP)
 With the growing data OLTP systems became inefficient and not optimal for
complex query processing, reporting and analytical need

3. In the beginning..
 In the early 70s Bill Inmon also famous as “The Father of Data Warehouse”
coined the term “Data Warehouse”
 According to Inmon “Data Warehouse is a collection of integrated, subject-
oriented databases designed to support a DSS (decision support system),
where each unit of data is non-volatile and relevant to some moment in
time”

4. These concerns
have existed
for more than
three decades-
 “We collect tons of data, but we can’t
access it.”
 “We need to slice and dice the data
every which way.”
 “Business people need to get at the
data easily.”
 “Just show me what is important.”
 “We spend entire meetings arguing
about who has the right numbers rather
than making decisions.”
 “We want people to use information to
support more fact-based decision
making.”

5. Why we need DWH?
 Consolidation of information
resources across multiples
platforms and even geographies at
a single premise by extracting(E),
transforming(T) and finally
loading(L)
 Improved query performance
 Foundation of Data Mining, Data
Visualization, Advanced reporting
using BI, OLAP tools

7. What is DWH
used for?
 Information and Knowledge:
 Intelligent Reporting and analytics
 Studying the trends and nature of
business over time
 Predicting the future with the help of
data
 Help business take key decisions and
make strategies
 Manage the history of transactions
happened within an organization
Examples:

8. What is DWH used
for?
Examples
 E-commerce
Providing cross sales, suggesting
different products like AMAZON
providing different features like
‘People also bought/viewed’.
Companies often use customer data
to predict or manipulate the
customer requirements considering
all the parameters like gender,
geography, age etc

9. What is DWH used
for?
Examples
 Supermarkets
One notable recent example of this was with
the US retailer Target. As part of its Data
Mining program, the company developed
rules to predict if their shoppers were likely
to be pregnant. By looking at the contents of
their customers shopping baskets, they could
spot customers who they thought were likely
to be expecting and begin targeting
promotions for nappies, cotton wool and so
on. The prediction was so accurate that
Target made the news by sending
promotional coupons to families who did not
yet realize they were pregnant!

10. What is DWH used for?
Examples
 Crime Agencies (where is crime
most likely to happen and
when?)
Crime prevention agencies use
analytics and Data Mining to spot
trends across myriads of data
helping with everything from
where to deploy police manpower.

11. What is DWH used
for?
Examples
 Service Providers-
Mobile phone and utilities companies
use Data Mining and Business
Intelligence to predict ‘churn’, the
terms they use for when a customer
leaves their company to get their
phone/gas/broadband from another
provider. They collate billing
information, customer services
interactions, website visits and other
metrics to give each customer a
probability score, then target offers
and incentives to customers whom they
perceive to be at a higher risk of
churning.

12. Responsibilities of DW/BI Managers:
 Understand the business users and determine the decisions that the business
users want to make with the help of the DW/BI system.
 Deliver high-quality, relevant, and accessible information and analytics to the
business users:
 Produce robust, presentable and meaningful data
 Continuously monitor the accuracy of the data and analyses
 Adapt to changing user profiles, requirements, and business priorities, along with
the availability of new data sources
 Sustain the DW/BI environment by updating the DW/BI system on a regular
basis and justify staffing and on-going expenditures

13. Pre-requisites before diving in…
 Types of Databases:
 OLTP
 OLAP
 What is Normalization?
 Types (1NF, 2NF, 3NF)
 Keys (Primary, Foreign, Composite, Surrogate)
 Data Modelling
 Conceptual
 Logical
 Physical
 E-R models
 Dimensional Modelling
 Star Schema
 Fact Tables
 Dimension tables
 Slowly changing dimensions (SCD – I, SCD – II, SCD – III)

14. OLTP and OLAP Databases
 Types of Databases:
 OLTP - Online transactional processing
 OLAP - Online analytical processing
 One of the most important assets of any organization is its information. This
asset is almost always used for two purposes: operational record keeping and
analytical decision making.
 Simply speaking, the operational systems (OLTP) are where you put the data
in, and the DW/BI system (OLAP) is where you get the data out.

15. OLTP vs OLAP
OLTP
 OLTPs are the original source of
the data.
 To control and run fundamental
business tasks
 Reveals a snapshot of ongoing
business processes
 Highly normalized with many
tables
 Low in space
 Volatile
OLAP
 OLAP data comes from the various
OLTP Databases
 To help with planning, problem
solving, and decision support
 Multi-dimensional views of various
kinds of business activities
 De-normalized, star schema
 High space
 Non-volatile

16. What is Normalization?
 Normalization is a database design technique which organizes tables in a
manner that reduces redundancy and dependency of data.
 It divides larger tables to smaller tables and links them using relationships.
 There are 3 prominent types of normalization but these techniques are still
evolving and we have totally 7 normalization techniques
 1NF
 2NF
 3NF

17. Types of
Normalization:
1NF
 Rules:
 Each table cell should
contain a single
value.
 Each record needs to
be unique

18. Types of
Normalization:
2NF
 Rules:
 The entity should be
in1NF form
 All attributes within the
entity should depend
solely on the unique
identifier of the entity

19. Types of Normalization:
3NF
 Product table  Brand table
Rules:
• The entity should be in 2NF form
• No column entry should be dependent on any other entry (value) other
than the key for the table

20. Types of Normalization:
3NF
 Product brand table  ER model:

21. Different types of keys -
 Primary Key - It identify each record uniquely in table. Primary key does not allow null value
in the column and keeps unique values throughout the column.
 Foreign Key - In a relationship between two tables, a primary key of one table is referred as a
foreign key in another table. Foreign key can have duplicate values in it and can also keep
null values if column is defined to accept nulls.
 Candidate Key- It can be selected as a primary key of the table. A table can have multiple
candidate keys, out of which one can be selected as a primary key.
 Unique Key - It can contain only unique values it also permits NULL values
 Alternate Key - It is a candidate key, not selected as primary key of the table.
 Composite Key (also known as compound key or concatenated key) - It is a group of two or
more columns that identifies each row of a table uniquely. Individual column of composite
key might not able to uniquely identify the record. It can be a primary key or candidate key
also.
 Super Key - It is a set of columns that uniquely identifies each row in a table. Super key may
hold some additional columns which are not strictly required to uniquely identify each row.
Primary key and candidate keys are minimal super keys or you can say subset of super keys.

23. Data Modelling
 Conceptual representation of different database tables/objects which depicts
the blueprint of the whole schema.
 There are 3 types of data modelling techniques which can be mentioned in
the below hierarchical manner:
 Conceptual
 Logical
 Physical

24. Data Modelling
(Conceptual Model)
 The main aim of this model is to
establish the entities, their attributes,
and their relationships. It has very less
details available of the actual Database
structure.
 The 3 basic tenants of Data Model are
 Entity: A real-world thing (
Constomer & Product)
 Attribute: Characteristics or
properties of an entity
 Relationship: Dependency or
association between two entities.

25. Data Modelling
(Logical Model)
 It defines the structure of the data
elements and set the relationships
between them.
 The advantage of the Logical data
model is to provide a foundation to
form the base for the Physical
model.
 At this Data Modeling level, no
primary or secondary key is
defined.

26. Data Modelling
(Physical Model)
 It describes the database specific
implementation of the data model.
 It offers an abstraction of the
database and helps generate
schema.
 This type of Data model also helps
to visualize database structure.
 It helps to model database columns
keys, constraints, indexes,
triggers, and other RDBMS
features.

27. Entity Relationship Models/Diagrams
 Entity-relationship diagrams (ER diagrams or ERDs) are drawings that
communicate the relationships between tables.
 It is a type of flowchart that illustrates how “entities” such as people, objects
or concepts relate to each other within a system.
 ER Diagrams are most often used to design or debug relational databases.
 Also known as ERDs or ER Models, they use a defined set of symbols such as
rectangles, diamonds, ovals and connecting lines to depict the
interconnectedness of entities, relationships and their attributes.

29. Dimensional Modelling
 Dimensional modeling is widely accepted as the preferred technique for
presenting analytic data because it addresses two simultaneous requirements:
 Deliver data that’s understandable to the business users.
 Deliver fast query performance.
 Both 3NF and dimensional models can be represented in ERDs because both
consist of joined relational tables; the key difference between 3NF and
dimensional models is the degree of normalization.
 Normalized 3NF structures divide data into many discrete entities, each of
which becomes a relational table. A database of sales orders might start with
a record for each order line but turn into a complex spider web diagram as a
3NF model, perhaps consisting of hundreds of normalized tables.

31. Dimensional Modelling (contd.)
 Normalized 3NF structures are immensely useful in operational processing
(OLTP) because an update or insert transaction touches the database in only
one place.
 Normalized models, however, are too complicated for BI queries. Users can’t
understand, navigate, or remember normalized models that resemble a map
of a city.
 The complexity of users’ unpredictable queries overwhelms the database
optimizers, resulting in disastrous query performance.
 Fortunately, dimensional modeling addresses the problem of overly complex
schemas in the presentation area.
 A dimensional model contains the same information as a normalized model,
but packages the data in a format that delivers user understandability, query
performance, and resilience to change.

32. Dimensional Modelling
– Star Schema
 Dimensional models
implemented in relational
database management systems
are referred to as star schemas
because of their resemblance to
a star-like structure.
 The downside of dimensional
modelling is that you pay a load
performance price for these
capabilities, especially with large
data sets.

33. Dimensional Modelling
– Fact Tables
 The fact table in a dimensional
model stores the performance
measurements resulting from an
organization’s business process
events.
 You should strive to store the low-
level measurement data resulting
from a business process in a single
dimensional model (grain ex: "balls
per innings").
 Imagine standing in the marketplace
watching products being sold and
writing down the unit quantity and
dollar sales amount for each product
in each sales transaction.

34. Dimensional Modelling – Fact Tables
(contd.)
 Types of facts additive, semi-additive, non-additive:
 Additive - Additive measures can be summed across any of the dimensions
associated with the fact table. (sales amount)
 Semi-Additive – They can be summed across some dimensions, but not all; balance
amounts are common semi-additive facts because they are additive across all
dimensions except time
 Non – Additive - Some measures are completely non-additive, such as ratios,
percentages and percentiles.
 Despite their sparsity, fact tables usually make up 90 percent or more of the
total space consumed by a dimensional model.
 Fact tables tend to be deep in terms of the number of rows, but narrow in
terms of the number of columns.

35. Dimensional Modelling
– Dimension Tables
 The dimension tables contain the textual
context associated with a business process
measurement event.
 They describe the “who, what, where, when,
how, and why” associated with the event.
 Dimension tables often have many columns or
attributes.
 Dimension tables tend to have fewer rows
than fact tables, but can be wide with many
large text columns.
 Each dimension is defined by a single primary
key , which serves as the basis for referential
integrity with any given fact table to which it
is joined.

36. Dimensional Modelling
– Dimension Tables
 Dimension attributes serve as the primary
source of query constraints, groupings, and
report labels.
 Dimension attributes are critical to making
the DW/BI system usable and understandable.
 The analytic power of the DW/BI environment
is directly proportional to the quality and
depth of the dimension attributes.
 Instead of third normal form, dimension
tables typically are highly denormalized with
flattened many-to-one relationships within a
single dimension table.
 We can almost always trade off dimension
table space for simplicity and accessibility.

37. Facts and Dimensions joined in a star
schema
 The first thing to notice about the dimensional schema is its simplicity and
symmetry.
 The charm of the design in is that it is highly recognizable to business users.
 Furthermore, the reduced number of tables and use of meaningful business
descriptors make it easy to navigate and less likely that mistakes will occur.
 The simplicity of a dimensional model also has performance benefits.
 Database optimizers process these simple schemas with fewer joins more
efficiently.
 Dimension attributes supply the report filters and labeling, whereas the fact
tables supply the report’s numeric values.

38. Facts and Dimensions
joined in a star
schema
SELECT st.district_name,
pd.brand,
SUM(sf.sales_dollars) AS "Sales
Dollars"
FROM store st,--dimension table
product pd,--dimension table
DATE dt,--dimension table
sales_facts sf --fact table
WHERE dt.month_name ="January"
AND dt.year =2013
AND st.store_key = sf.store_key
AND pd.product_key = sf.product_key
AND dt.date_key = sf.date_key
GROUP BY st.district_name, pd.brand

39. Slowly changing dimensions
 It is a dimension that stores and manages both current and historical data
over time in a data warehouse. It is considered and implemented as one of
the most critical ETL tasks in tracking the history of dimension records.
 There are three types of SCDs and you can use Warehouse Builder to define,
deploy, and load all three types of SCDs.
 Type 1 SCDs - Overwriting
 Type 2 SCDs - Creating another dimension record
 Type 3 SCDs - Creating a current value field

40. Slowly Changing
Dimensions (contd.)
 Type 1 -Overwriting the old
value. In this method no history
of dimension changes is kept in
the database. The old dimension
value is simply overwritten be
the new one.

41. Slowly Changing
Dimensions (contd.)
 Type 2 - Creating a new
additional record. In this
methodology all history of
dimension changes is kept in the
database. Changes in the
attributes are captured by
adding a new row with a new
surrogate key to the dimension
table.

42. Slowly Changing
Dimensions (contd.)
 Type 3 - Adding a new column. In
this type usually only the current
and previous value of dimension
is kept in the database. The new
value is loaded into 'current/new'
column and the old one into
'old/previous' column. Generally
speaking the history is limited to
the number of column created
for storing historical data. This is
the least commonly needed
technique.

43. Data Marts
 A data mart is focused on a single functional
area of an organization and contains a subset
of data stored in a Data Warehouse.
 A data mart is a condensed version of Data
Warehouse and is designed for use by a
specific department, unit or set of users in an
organization. E.g., Marketing, Sales, HR or
finance.
 It is often controlled by a single department
in an organization.
 Data Mart usually draws data from only a few
sources compared to a Data warehouse.
 Data marts are small in size and are more
flexible compared to a Data warehouse.

44. A glimpse of Data Warehouse
Architecture

45. Data Warehouse Architecture

46. Data Warehouse Architecture (Inmon)

47. Data Warehouse Architecture (Kimball)

48. Data Warehouse Architectures (In Detail)
 Topics:
 Independent Data Mart Architecture
 Kimball Architecture
 Inmon Architecture (Hub-and-Spoke Corporate Information Factory)
 Key differences between Inmon and Kimball architectures

49. Independent Data
Mart Architecture
 Data is deployed on a departmental
basis without concern to sharing and
integrating information across the
enterprise.
 Its less recommended but this approach
is prevalent, especially in large
organizations.
 It’s the path of least resistance for fast
development at relatively low cost, at
least in the short run.
 Multiple uncoordinated extracts from
the same operational sources and
redundant storage of analytic data are
inefficient and wasteful in the long run.

50. Kimball Architecture
 There are four separate and distinct components to consider in the DW/BI
environment:
 Operational source systems (OLTP)
 ETL system
 Data presentation area (Dimensional Model)
 Business intelligence applications.

51. Kimball Architecture (contd.)

52. Kimball Architecture (contd.)
 Operational Source Systems:
 These are the operational systems of record that capture the business’s
transactions (OLTP)
 Extract, Transformation, and Load System:
 Extraction is the first step in the process of getting data into the data warehouse
environment
 There are numerous potential transformations, such as :
 Cleansing the data. Ex converting to standard date formats or correcting the misspellings.
 Combining data from multiple sources, and de-duplicating data.
 Primary mission of the ETL system is to hand off the dimension and fact tables in
the delivery step, these subsystems are critical.

53. Kimball Architecture (contd.)
 Data presentation area:
 The dimensional model with the star schema which is the byproduct of the whole
ETL design.
 Business Intelligence Applications:
 By definition, all BI applications query the data in the DW/BI presentation area to
generate the data in more meaningful and descriptive form using various reporting
techniques like pie charts, bar graphs, cross tabs etc.

54. Hub and spoke
Corporate Information
Factory (Inmon)
 This model identifies the key subject areas,
and most importantly, the key entities the
business operates with and cares about, like
customer, product, vendor, etc.
 Detailed logical model is created for each
major entity. Ex: A logical model will be built
for Customer with all the details related to
that entity.
 The key point here is that the entity structure
is built in normalized form (3NF), data
redundancy is avoided as much as possible.
 Data marts specific for departments are built
on top of the 3NF model and the data marts
can have de-normalized data to help with
reporting/ fast querying.

55. Kimball vs Inmon Architectures
Characteristics Favours Kimball Favours Inmon
Business decision support
requirements
Tactical Strategic
Data integration
requirements
Individual business
requirements
Enterprise-wide integration
The structure of data
KPI, business performance
measures, scorecards…
Data that meet multiple and
varied information needs
and non-metric data
Persistence of data in
source systems
Source systems are quite
stable
Source systems have high
rate of change
Skill sets Small team of generalists Bigger team of specialists
Time constraint
Urgent needs for the first
data warehouse
Longer time is allowed to
meet business’ needs.
Cost to build Low start-up cost High start-up costs

56. Refrences
 The Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling,
2nd Edition Ralph Kimball, Margy Ross
 Introduction to Data Warehousing Concepts docs.oracle.com
 A Short History of Data Warehousing – DATAVERSITY www.dataversity.net
 Data integration – Wikipedia en.wikipedia.org
 Data Warehouse Design: The Good, the Bad, the Ugly blog.panoply.io
 ER Model Basic Conceptswww.tutorialspoint.com
 Oracle ATG Web Commerce - Search ERDdocs.oracle.com
 Document Entities | Oracle Magazineblogs.oracle.com

57. Thank You!