1. Data Warehousing (DAY 2) Siwawong W. Project Manager 2010.05.25

2. Agenda Data Warehouse: DW Structure/Modeling 10:00 – 10:30 09:00 – 09:15 Registration 09:15 – 09:30 Review 1 st Day class 09:30 – 10:00 Data Warehouse: Data Warehousing 10:30 – 10:45 Break & Morning Refreshment 10:45 – 12:00 DWs and Data Marts 12:00 – 13:00 Lunch Break 13:00 – 15:00 Query Processing 15:00 – 15:15 Break 15:15 – 16:00 OLAP & DSS

3. 1 st Day Review

6. Data Warehouse: Data Warehousing

8. Data Warehouse: Architecture

9. Data Warehouse Architecture Data Warehouse Engine Optimized Loader Extraction Cleansing Analyze Query Metadata Repository Relational Databases Legacy Data Purchased Data ERP Systems

11. Loading the Warehouse Cleaning the data before it is loaded

14. Data Integration Across Sources Trust Credit card Savings Loans Same data different name Different data Same name Data found here nowhere else Different keys same data

15. Data Transformation Example Data Warehouse encoding unit field appl A - balance appl B - bal appl C - currbal appl D - balcurr appl A - pipeline - cm appl B - pipeline - in appl C - pipeline - feet appl D - pipeline - yds appl A - m,f appl B - 1,0 appl C - x,y appl D - male, female

30. Structuring/Modeling Issues

37. Vertical Partitioning Frequently accessed Rarely accessed Smaller table and so less I/O Acct. No Name Balance Date Opened Interest Rate Address Acct. No Balance Acct. No Name Date Opened Interest Rate Address

40. D/W Schema

53. Data Warehouse vs. Data Marts What comes first?

55. From the Data Warehouse to Data Marts Data Information Departmentally Structured Individually Structured Data Warehouse Organizationally Structured Less More History Normalized Detailed

57. Data Warehouse and Data Marts OLAP Data Mart Lightly summarized Departmentally structured Organizationally structured Atomic Detailed Data Warehouse Data

60. Data Mart Centric Data Marts Data Sources Data Warehouse

61. Problems with Data Mart Centric Solution If you end up creating multiple warehouses, integrating them is a problem

62. True Warehouse Data Marts Data Sources Data Warehouse

65. Query Processing

69. Bitmap Index Customer Query : select * from customer where gender = ‘F’ and vote = ‘Y’ gender (f) vote (y) result vote gender 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 M F F F F M Y Y Y N N N

78. On-Line Analytical Processing (OLAP)

87. A Visual Operation: Pivot (Rotate) 10 47 30 12 Juice Cola Milk Cream NY LA SF 3/1 3/2 3/3 3/4 Month Region Product

88. “ Slicing and Dicing” Product Sales Channel Regions Retail Direct Special Household Telecomm Video Audio India Far East Europe The Telecomm Slice

96. Relational OLAP: 3 Tier DSS Store atomic data in industry standard RDBMS. Generate SQL execution plans in the ROLAP engine to obtain OLAP functionality. Obtain multi-dimensional reports from the DSS Client. Data Warehouse ROLAP Engine Decision Support Client Database Layer Application Logic Layer Presentation Layer

99. MD-OLAP: 2 Tier DSS MDDB Engine MDDB Engine Decision Support Client Database Layer Application Logic Layer Presentation Layer Store atomic data in a proprietary data structure (MDDB), pre-calculate as many outcomes as possible, obtain OLAP functionality via proprietary algorithms running against this data. Obtain multi-dimensional reports from the DSS Client.

103. Recipe for a Successful Warehouse

110. References/External Links (1) Data Warehousing & Data Mining S. Sudarshan Krithi Ramamritham IIT Bombay (2) Data Warehousing Hu Yan e-mail: [email_address] (3) Data Warehosing Concept: MOLAP, ROLAP and HOLAP http://www.1keydata.com/datawarehousing/molap-rolap.html

111. Thank you for your attention! [email_address] www.blueballgroup.com