Advances in Database Querying Overview of Data Warehouses, OLAP, Data Mining and Query Processing

Advances in Database Querying Satish Bobba Sr.Informatica Developer [email_address]

Acknowledgments Pankaj Gangshettiwar Anit Jha CGI AMS

Overview ,[object Object],[object Object],[object Object],[object Object]

Data, Data everywhere yet ... ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

What is a Data Warehouse? ,[object Object],[object Object]

Why Data Warehousing? Which are our lowest/highest margin customers ? Who are my customers and what products are they buying? Which customers are most likely to go to the competition ? What impact will new products/services have on revenue and margins? What product prom- -otions have the biggest impact on revenue? What is the most effective distribution channel?

Decision Support ,[object Object],[object Object],[object Object],[object Object],[object Object]

Evolution of Decision Support ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

What are the users saying... ,[object Object],[object Object],[object Object],[object Object]

Data Warehousing -- It is a process ,[object Object],[object Object]

Traditional RDBMS used for OLTP ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

OLTP vs Data Warehouse ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Data Warehouse Architecture Relational Databases Legacy Data Purchased Data Data Warehouse Engine Optimized Loader Extraction Cleansing Analyze Query Metadata Repository

From the Data Warehouse to Data Marts Departmentally Structured Individually Structured Data Warehouse Organizationally Structured Less More History Normalized Detailed Data Information

Users have different views of Data Organizationally structured OLAP Explorers: Seek out the unknown and previously unsuspected rewards hiding in the detailed data Farmers: Harvest information from known access paths Tourists: Browse information harvested by farmers

Wal*Mart Case Study ,[object Object],[object Object],[object Object],[object Object],[object Object]

Old Retail Paradigm ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

New (Just-In-Time) Retail Paradigm ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Information as a Strategic Weapon ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Schema Design ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Star Schema ,[object Object],[object Object],T i m e p r o d c u s t c i t y f a c t date, custno, prodno, cityname, sales

Dimension Tables ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Fact Table ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Snowflake schema ,[object Object],[object Object],T i m e p r o d c u s t c i t y f a c t date, custno, prodno, cityname, ... r e g i o n

Fact Constellation ,[object Object],[object Object],[object Object],Hotels Travel Agents Promotion Room Type Customer Booking Checkout

Data Granularity in Warehouse ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Granularity in Warehouse ,[object Object],[object Object],[object Object],[object Object],[object Object]

Levels of Granularity Operational 60 days of activity account activity date amount teller location account bal account month # trans withdrawals deposits average bal amount activity date amount account bal monthly account register -- up to 10 years Not all fields need be archived Banking Example

Data Integration Across Sources Trust Credit card Savings Loans Same data different name Different data Same name Data found here nowhere else Different keys same data

Data Transformation ,[object Object],[object Object],[object Object],Sequential Legacy Relational External Operational/ Source Data Data Transformation Accessing Capturing Extracting Householding Filtering Reconciling Conditioning Loading Validating Scoring

Data Transformation Example encoding unit field appl A - balance appl B - bal appl C - currbal appl D - balcurr appl A - pipeline - cm appl B - pipeline - in appl C - pipeline - feet appl D - pipeline - yds appl A - m,f appl B - 1,0 appl C - x,y appl D - male, female Data Warehouse

Data Integrity Problems ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Data Transformation Terms ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Data Transformation Terms ,[object Object],[object Object],[object Object],[object Object]

Refresh ,[object Object],[object Object],[object Object],[object Object]

When to Refresh? ,[object Object],[object Object],[object Object],[object Object]

Refresh techniques ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

How To Detect Changes ,[object Object],[object Object],[object Object],[object Object]

Querying Data Warehouses ,[object Object],[object Object],[object Object],[object Object]

SQL Extensions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Reporting Tools ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Operational data Detailed transactional data Data warehouse Merge Clean Summarize Direct Query Reporting tools Mining tools Decision support tools Oracle SAS Relational DBMS+ e.g. Redbrick IMS Crystal reports Essbase Intelligent Miner Bombay branch Delhi branch Calcutta branch Census data OLAP GIS data

Deploying Data Warehouses ,[object Object],[object Object],[object Object]

Cultural Considerations ,[object Object],[object Object],[object Object],[object Object]

User Training ,[object Object],[object Object]

Warehouse Products ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Nature of OLAP Analysis ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Multi-dimensional Data ,[object Object],Dimensions : Product, Region, Time Hierarchical summarization paths Product Region Time Industry Country Year Category Region Quarter Product City Month week Office Day Month 1 2 3 4 7 6 5 Product Toothpaste Juice Cola Milk Cream Soap Region W S N

Conceptual Model for OLAP ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Operations ,[object Object],[object Object],[object Object],[object Object]

More Cube Operations ,[object Object],[object Object],[object Object],[object Object]

More OLAP Operations ,[object Object],[object Object],[object Object],[object Object],[object Object]

MOLAP vs ROLAP ,[object Object],[object Object]

SQL Extensions ,[object Object],[object Object],[object Object],[object Object]

OLAP: 3 Tier DSS Data Warehouse Database Layer Store atomic data in industry standard Data Warehouse. OLAP Engine Application Logic Layer Generate SQL execution plans in the OLAP engine to obtain OLAP functionality. Decision Support Client Presentation Layer Obtain multi-dimensional reports from the DSS Client.

Strengths of OLAP ,[object Object],[object Object],[object Object],[object Object],[object Object]

Brief History ,[object Object],[object Object],[object Object],[object Object],[object Object]

OLAP and Executive Information Systems ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Microsoft OLAP strategy ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Why Data Mining ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Data Mining helps extract such information

Data mining ,[object Object],[object Object],[object Object],[object Object]

Some basic operations ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Classification ,[object Object],Age Salary Profession Location Customer type Previous customers Classifier Decision rules Salary > 5 L Prof. = Exec New applicant’s data Good/ bad

Classification methods ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],Decision trees Salary < 1 M Prof = teacher Age < 30 Good Bad Bad Good

Pros and Cons of decision trees ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],More information: http://www.stat.wisc.edu/~limt/treeprogs.html

Neural network ,[object Object],Hidden nodes Output nodes x1 x2 x3 x1 x2 x3 w1 w2 w3 Basic NN unit A more typical NN

Pros and Cons of Neural Network ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Conclusion: Use neural nets only if decision trees/NN fail.

Bayesian learning ,[object Object],[object Object],[object Object],[object Object],[object Object]

Clustering ,[object Object],[object Object],[object Object],[object Object]

Association rules ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Milk, cereal Tea, milk Tea, rice, bread cereal T

Variants ,[object Object],[object Object],[object Object],[object Object]

Prevalent  Interesting ,[object Object],[object Object],[object Object],1995 Milk and cereal sell together! Milk and cereal sell together! 1998 Zzzz...

What makes a rule surprising? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Application Areas Industry Application Finance Credit Card Analysis Insurance Claims, Fraud Analysis Telecommunication Call record analysis Transport Logistics management Consumer goods promotion analysis Data Service providers Value added data Utilities Power usage analysis

Data Mining in Use ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Why Now? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Data Mining works with Warehouse Data ,[object Object],[object Object]

Mining market ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

OLAP Mining integration ,[object Object],[object Object],[object Object],[object Object],[object Object]

State of art in mining OLAP integration ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Vertical integration: Mining on the web ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Part 4: Speeding up Query Processing

Advances in Database Querying Overview of Data Warehouses, OLAP, Data Mining and Query Processing

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (17)

Ähnlich wie Advances in Database Querying Overview of Data Warehouses, OLAP, Data Mining and Query Processing

Ähnlich wie Advances in Database Querying Overview of Data Warehouses, OLAP, Data Mining and Query Processing (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Advances in Database Querying Overview of Data Warehouses, OLAP, Data Mining and Query Processing