Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

IT Ready - DW: 1st Day

  • Loggen Sie sich ein, um Kommentare anzuzeigen.

  • Gehören Sie zu den Ersten, denen das gefällt!

IT Ready - DW: 1st Day

  1. 1. Data Warehousing (DAY 1) Siwawong W. Project Manager 2010.05.24
  2. 2. Agenda 09:00 – 09:15 Registration 09:15 – 09:30 Self-Introduction 09:30 – 10:30 Data Warehouse: Introduction 10:30 – 10:45 Break & Morning Refreshment 10:45 – 12:00 Data Warehouse: Introduction (Cont’) 12:00 – 13:00 Lunch Break 13:00 – 15:00 Review RDBMS & SQL command 15:00 – 15:15 Break 15:15 – 16:00 Case Study ~ Q/A
  4. 4. About Me <ul><li>My Name: Siwawong Wuttipongprasert </li></ul><ul><ul><li>Nick-name: Tae (You can call this name. it’s easier) </li></ul></ul><ul><li>My Background: </li></ul><ul><ul><li>B.Eng (Computer Engineering), Chiang Mai University. </li></ul></ul><ul><li>My Career Profile: </li></ul><ul><ul><li>10+ years in IT business </li></ul></ul><ul><ul><li>5+ years with Blue Ball Co., Ltd. </li></ul></ul><ul><ul><li>Role: Programmer, System Analysis, Consultant & Project Manager </li></ul></ul><ul><ul><li>Working Area: ERP, MRP, Retailing, Banking, Financial, E-Commerce, etc. </li></ul></ul><ul><ul><li>Working with multi-cultures: Japanese, German and Vietnamese </li></ul></ul><ul><li>Know Me More.. </li></ul>
  5. 5. My Company: Blue Ball Blue Ball Group is an Offshoring Company that focus totally in customer satisfaction. It takes advantage of western management combined with Asian human resources to provide high quality services Thailand (Head Office) Mexico (Special Developments) Vietnam (Offshoring Center)
  6. 6. Services from My Company Offshoring Programmers &Testers   Blue Ball will get you ready to offshore successfully. No need to rush you into offshoring without you feeling confident on how to send, organize, receive, test and accept job confidently   System Development & Embedded Solutions   Solutions that combine technological expertise and deep business understanding. We only start coding once every single detail such as milestones, scheduling, contact point, communication, issue management and critical protocols are in place Web design and E-commerce   Premium web design, CMS, e-commerce solutions and SEO services. Website maintenance and copy content creation to develop marketing campaigns that SELL for discerning companies to increase the quality and reach of their marketing campaigns
  7. 7. My Clients
  8. 8. Data Warehouse: Introduction
  9. 9. Data Warehouse: Introduction <ul><li>Data Warehousing, OLAP and data mining: </li></ul><ul><ul><li>what and why (now)? </li></ul></ul><ul><li>Relation to OLTP </li></ul><ul><li>Review RDMBS & SQL Command </li></ul><ul><li>A case study </li></ul>
  10. 10. Data Warehouse: What & Why? Problem Statements
  11. 11. A producer wants to know…. Which are our lowest/highest margin customers ? Who are my customers and what products are they buying? Which customers are most likely to go to the competition ? What impact will new products/services have on revenue and margins? What product prom- -otions have the biggest impact on revenue? What is the most effective distribution channel?
  12. 12. Data, Data everywhere, yet ... <ul><li>I can’t find the data I need </li></ul><ul><ul><li>data is scattered over the network </li></ul></ul><ul><ul><li>many versions, subtle differences </li></ul></ul><ul><li>I can’t get the data I need </li></ul><ul><ul><li>need an expert to get the data </li></ul></ul><ul><li>I can’t understand the data I found </li></ul><ul><ul><li>available data poorly documented </li></ul></ul><ul><li>I can’t use the data I found </li></ul><ul><ul><li>results are unexpected </li></ul></ul><ul><ul><li>data needs to be transformed from one form to other </li></ul></ul>
  13. 13. What is a Data Warehouse? <ul><li>A single, complete and consistent store of </li></ul><ul><li>data obtained from a variety of different </li></ul><ul><li>sources made available to end users in a </li></ul><ul><li>what they can understand and use in a </li></ul><ul><li>business context. </li></ul><ul><li>[Barry Devlin] </li></ul>
  14. 14. What are the users saying... <ul><li>Data should be integrated across the enterprise </li></ul><ul><li>Summary data has a real value to the organization </li></ul><ul><li>Historical data holds the key to understanding data over time </li></ul><ul><li>What-if capabilities are required </li></ul>
  15. 15. What is Data Warehousing? A process of transforming data into information and making it available to users in a timely enough manner to make a difference [Forrester Research, April 1996] Data Information
  16. 16. Evolution <ul><li>60’s: Batch reports </li></ul><ul><ul><li>hard to find and analyze information </li></ul></ul><ul><ul><li>inflexible and expensive, reprogram every new request </li></ul></ul><ul><li>70’s: Terminal-based DSS and EIS (executive information systems) </li></ul><ul><ul><li>still inflexible, not integrated with desktop tools </li></ul></ul><ul><li>80’s: Desktop data access and analysis tools </li></ul><ul><ul><li>query tools, spreadsheets, GUIs </li></ul></ul><ul><ul><li>easier to use, but only access operational databases </li></ul></ul><ul><li>90’s: Data warehousing with integrated OLAP engines and tools </li></ul>
  17. 17. Very Large Data Bases Terabytes -- 10^12 bytes: Petabytes -- 10^15 bytes: Exabytes -- 10^18 bytes: Zettabytes -- 10^21 bytes: Zottabytes -- 10^24 bytes: Walmart -- 24 Terabytes Intelligence Agency Videos Geographic Information Systems National Medical Records Weather images
  18. 18. Data Warehousing -- It is a process <ul><li>Technique for assembling and managing data from various sources for the purpose of answering business questions. Thus making decisions that were not previous possible </li></ul><ul><li>A decision support database maintained separately from the organization’s operational database </li></ul>
  19. 19. Data Warehouse <ul><li>A data warehouse is a </li></ul><ul><ul><li>subject-oriented: Organized based on use </li></ul></ul><ul><ul><li>Integrated: inconsistencies remove </li></ul></ul><ul><ul><li>time-varying: data are normally time-series </li></ul></ul><ul><ul><li>non-volatile: store in read-only format </li></ul></ul><ul><li>collection of data that is used primarily in organizational decision making. </li></ul><ul><ul><li> -- Bill Inmon, Building the Data Warehouse 1996 </li></ul></ul>
  20. 20. Data Warehouse: Subjected-Oriented WH is organized around the major subjects of the enterprise..rather than the major application areas.. This is reflected in the need to store decision-support data rather than application-oriented data Subject-Oriented DBWH Sales Operational DB Order Processing Application-Oriented
  21. 21. Data Warehouse: Integrated Because the source data come together from different enterprise-wide applications systems. The source data is often inconsistent using the integrated data source must be made consistent to present a unified view of the data to the users
  22. 22. Data Warehouse: time-varying The source data in the WH is only accurate and valid at some point in time or over some time interval. The time-variance of the data warehouse is also shown in the extended time that the data is held, the implicit or explicit association of time with all data, and the fact that the data represents a series of snapshots Historical data is recorded
  23. 23. Data Warehouse: Non-volatile Data is NOT update in real time but is refresh from OS on a regular basis. New data is always added as a supplement to DB, rather than replacement. The DB continually absorbs this new data, incrementally integrating it with previous data Anyone who is using the database has confidence that a query will always produce the same result no matter how often it is run
  24. 24. Explorers, Farmers and Tourists Explorers: Seek out the unknown and previously unsuspected rewards hiding in the detailed data Farmers: Harvest information from known access paths Tourists: Browse information harvested by farmers
  25. 25. Data Warehouse Architecture Data Warehouse Engine Optimized Loader Extraction Cleansing Analyze Query Metadata Repository Relational Databases Legacy Data Purchased Data ERP Systems
  26. 26. OLAP & Data Mining
  27. 27. Data Warehouse for DS & OLAP <ul><li>Putting Information technology to help the knowledge worker make faster and better decisions </li></ul><ul><ul><li>Which of my customers are most likely to go to the competition? </li></ul></ul><ul><ul><li>What product promotions have the biggest impact on revenue? </li></ul></ul><ul><ul><li>How did the share price of software companies correlate with profits over last 10 years? </li></ul></ul>
  28. 28. Decision Support (DS) <ul><li>Used to manage and control business </li></ul><ul><li>Data is historical or point-in-time </li></ul><ul><li>Optimized for inquiry rather than update </li></ul><ul><li>Use of the system is loosely defined and can be ad-hoc </li></ul><ul><li>Used by managers and end-users to understand the business and make judgements </li></ul>
  29. 29. Data Mining works with Warehouse Data Data Warehousing provides the Enterprise with a memory Data Mining provides the Enterprise with intelligence
  30. 30. We want to know ... <ul><li>Given a database of 100,000 names, which persons are the least likely to default on their credit cards? </li></ul><ul><li>Which types of transactions are likely to be fraudulent given the demographics and transactional history of a particular customer? </li></ul><ul><li>If I raise the price of my product by Rs. 2, what is the effect on my ROI? </li></ul><ul><li>If I offer only 2,500 airline miles as an incentive to purchase rather than 5,000, how many lost responses will result? </li></ul><ul><li>If I emphasize ease-of-use of the product as opposed to its technical capabilities, what will be the net effect on my revenues? </li></ul><ul><li>Which of my customers are likely to be the most loyal? </li></ul>Data Mining helps extract such information
  31. 31. Application Areas Industry Application Finance Credit Card Analysis Insurance Claims, Fraud Analysis Telecommunication Call record analysis Transport Logistics management Consumer goods promotion analysis Data Service providers Value added data Utilities Power usage analysis
  32. 32. Data Mining in Use <ul><li>The US Government uses Data Mining to track fraud </li></ul><ul><li>A Supermarket becomes an information broker </li></ul><ul><li>Basketball teams use it to track game strategy </li></ul><ul><li>Cross Selling </li></ul><ul><li>Warranty Claims Routing </li></ul><ul><li>Holding on to Good Customers </li></ul><ul><li>Weeding out Bad Customers </li></ul>
  33. 33. What makes data mining possible? <ul><li>Advances in the following areas are making data mining deployable: </li></ul><ul><ul><li>data warehousing </li></ul></ul><ul><ul><li>better and more data </li></ul></ul><ul><ul><ul><li>i.e., operational, behavioral, and demographic </li></ul></ul></ul><ul><ul><li>the emergence of easily deployed data mining tools and </li></ul></ul><ul><ul><li>the advent of new data mining techniques. </li></ul></ul><ul><ul><ul><ul><li>-- Gartner Group </li></ul></ul></ul></ul>
  34. 34. Why Separate Data Warehouse? <ul><li>Performance </li></ul><ul><ul><li>Operation DBs designed & tuned for known transaction & workloads. </li></ul></ul><ul><ul><li>Complex OLAP queries would degrade performance for operation transaction. </li></ul></ul><ul><ul><li>Special data organization, access & implementation methods needed for multidimensional views & queries. </li></ul></ul><ul><li>Function </li></ul><ul><ul><li>Missing data: Decision support requires historical data, which operation DBs do not typically maintain. </li></ul></ul><ul><ul><li>Data consolidation: Decision support requires consolidation (aggregation, summarization) of data from many heterogeneous sources: operation DBs, external sources. </li></ul></ul><ul><ul><li>Data quality: Different sources typically use inconsistent data representations, codes, and formats which have to be reconciled. </li></ul></ul>
  35. 35. What’s OLTP? <ul><li>DBMS built for O n L ine T ransaction P rocessing (OLTP) is generally regarded as unsuitable for data warehousing because each system is designed with a differing set of requirements in mind </li></ul><ul><li>example: OLTP systems are design to maximize the transaction processing capacity, while data warehouses are designed to support ad hoc query processing </li></ul>
  36. 36. What are Operational Systems? <ul><li>They are OLTP systems </li></ul><ul><li>Run mission critical applications </li></ul><ul><li>Need to work with stringent performance requirements for routine tasks </li></ul><ul><li>Used to run a business! </li></ul>
  37. 37. RDBMS used for OLTP <ul><li>Database Systems have been used traditionally for OLTP </li></ul><ul><ul><li>clerical data processing tasks </li></ul></ul><ul><ul><li>detailed, up to date data </li></ul></ul><ul><ul><li>structured repetitive tasks </li></ul></ul><ul><ul><li>read/update a few records </li></ul></ul><ul><ul><li>isolation, recovery and integrity are critical </li></ul></ul>
  38. 38. Operational Systems <ul><li>Run the business in real time </li></ul><ul><li>Based on up-to-the-second data </li></ul><ul><li>Optimized to handle large numbers of simple read/write transactions </li></ul><ul><li>Optimized for fast response to predefined transactions </li></ul><ul><li>Used by people who deal with customers, products -- clerks, salespeople etc. </li></ul><ul><li>They are increasingly used by customers </li></ul>
  39. 39. Examples of Operational Data Data Industry Usage Technology Volumes Customer File All Track Customer Detail Legacy application, flat files, main frames Small-medium Account Balance Finance Control Account Activities Legacy applications, hierarchical databases, mainframe Large Point-of- Sale data Retail Generate bills, manage stock ERP, Client/Server, relational databases Very Large Call Record Tele-Comm. Billing Legacy application, hierarchical database, mainframe Very Large Production Record Mfg. Control Production ERP, RDBMS, AS/400 Medium
  40. 40. Related to OLTP
  41. 41. Application-Orientation vs. Subject-Orientation Application-Orientation Subject-Orientation Data Warehouse Customer Vendor Product Activity Operational Database Loans Credit Card Trust Savings
  42. 42. OLTP vs. Data Warehouse <ul><li>OLTP systems are tuned for known transactions and workloads while workload is not known a priori in a data warehouse </li></ul><ul><li>Special data organization, access methods and implementation methods are needed to support data warehouse queries (typically multidimensional queries) </li></ul><ul><ul><li>e.g ., average amount spent on phone calls between 9AM-5PM in Pune during the month of December </li></ul></ul>
  43. 43. OLTP vs. Data Warehouse <ul><li>OLTP </li></ul><ul><ul><li>Application Oriented </li></ul></ul><ul><ul><li>Used to run business </li></ul></ul><ul><ul><li>Detailed data </li></ul></ul><ul><ul><li>Current up to date </li></ul></ul><ul><ul><li>Isolated Data </li></ul></ul><ul><ul><li>Repetitive access </li></ul></ul><ul><ul><li>Clerical User </li></ul></ul><ul><li>Warehouse (DSS) </li></ul><ul><ul><li>Subject Oriented </li></ul></ul><ul><ul><li>Used to analyze business </li></ul></ul><ul><ul><li>Summarized and refined </li></ul></ul><ul><ul><li>Snapshot data </li></ul></ul><ul><ul><li>Integrated Data </li></ul></ul><ul><ul><li>Ad-hoc access </li></ul></ul><ul><ul><li>Knowledge User (Manager) </li></ul></ul>
  44. 44. OLTP vs. Data Warehouse <ul><li>OLTP </li></ul><ul><ul><li>Performance Sensitive </li></ul></ul><ul><ul><li>Few Records accessed at a time (tens) </li></ul></ul><ul><ul><li>Read/Update Access </li></ul></ul><ul><ul><li>No data redundancy </li></ul></ul><ul><ul><li>Database Size 100MB -100 GB </li></ul></ul><ul><li>Data Warehouse </li></ul><ul><ul><li>Performance relaxed </li></ul></ul><ul><ul><li>Large volumes accessed at a time(millions) </li></ul></ul><ul><ul><li>Mostly Read (Batch Update) </li></ul></ul><ul><ul><li>Redundancy present </li></ul></ul><ul><ul><li>Database Size 100 GB - few terabytes </li></ul></ul>
  45. 45. OLTP vs. Data Warehouse <ul><li>OLTP </li></ul><ul><ul><li>Transaction throughput is the performance metric </li></ul></ul><ul><ul><li>Thousands of users </li></ul></ul><ul><ul><li>Managed in entirety </li></ul></ul><ul><li>Data Warehouse </li></ul><ul><ul><li>Query throughput is the performance metric </li></ul></ul><ul><ul><li>Hundreds of users </li></ul></ul><ul><ul><li>Managed by subsets </li></ul></ul>
  46. 46. To summarize ... OLTP Systems are used to “ run ” a business The Data Warehouse helps to “ optimize ” the business
  47. 47. Why Now? <ul><li>Data is being produced </li></ul><ul><li>ERP provides clean data </li></ul><ul><li>The computing power is available </li></ul><ul><li>The computing power is affordable </li></ul><ul><li>The competitive pressures are strong </li></ul><ul><li>Commercial products are available </li></ul>
  48. 48. Myths surrounding OLAP Servers and Data Marts <ul><li>Data marts and OLAP servers are departmental solutions supporting a handful of users </li></ul><ul><li>Million dollar massively parallel hardware is needed to deliver fast time for complex queries </li></ul><ul><li>OLAP servers require massive and unwieldy indices </li></ul><ul><li>Complex OLAP queries clog the network with data </li></ul><ul><li>Data warehouses must be at least 100 GB to be effective </li></ul><ul><ul><ul><ul><ul><li>Source -- Arbor Software Home Page </li></ul></ul></ul></ul></ul>
  49. 49. Advantages Of Data Warehouse <ul><li>High query performance </li></ul><ul><ul><li>But not necessarily most current information </li></ul></ul><ul><li>Doesn’t interfere with local processing at sources </li></ul><ul><ul><li>Complex queries at warehouse </li></ul></ul><ul><ul><li>OLTP at information sources </li></ul></ul><ul><li>Information copied at warehouse </li></ul><ul><ul><li>Can modify, annotate, summarize,  restructure, etc . </li></ul></ul><ul><ul><li>Can store historical information </li></ul></ul><ul><ul><li>Security </li></ul></ul>
  50. 50. Data Warehouse: Pain Beware <ul><li>Data Integration </li></ul><ul><li>Data Quality </li></ul><ul><li>Data Availability </li></ul><ul><li>End user education </li></ul><ul><li>Proper sizing – HW and Database environment </li></ul><ul><li>Lack of off - the - shelf product ( mature Packaged Analytics ) </li></ul><ul><li>Post implementation challenges </li></ul><ul><ul><li>Ensure usage </li></ul></ul><ul><ul><li>Identify new areas of Intelligence </li></ul></ul><ul><ul><li>Measure business benefits – productivity enhancements, savings, increase in revenue etc . </li></ul></ul>
  51. 51. Review RDBMS & SQL statement
  52. 52. Relational DBMS: Properties <ul><li>Each relation (or table) in a database has a unique name </li></ul><ul><li>An entry at the intersection of each row and column is atomic (or single-valued); </li></ul><ul><ul><li>there can be no multi-valued attributes in a relation </li></ul></ul><ul><li>Each row is unique; </li></ul><ul><ul><li>no two rows in a relation are identical </li></ul></ul><ul><li>Each attribute (or column) within a table has a unique name </li></ul><ul><li>The sequence of columns (left to right) is insignificant; </li></ul><ul><ul><li>the columns of a relation can be interchanged without changing the meaning or use of the relation </li></ul></ul><ul><li>The sequence of rows (top to bottom) is insignificant; </li></ul><ul><ul><li>rows of a relation may be interchanged or stored in any sequence </li></ul></ul>
  53. 53. The Relational Model... <ul><li>The relational model of data has three major components: </li></ul><ul><ul><li>Relational database objects </li></ul></ul><ul><ul><ul><li>allows to define data structures </li></ul></ul></ul><ul><ul><li>Relational operators </li></ul></ul><ul><ul><ul><li>allows manipulation of stored data </li></ul></ul></ul><ul><ul><li>Relational integrity constraints </li></ul></ul><ul><ul><ul><li>allows to defines business rules and ensure data integrity </li></ul></ul></ul>
  54. 54. The Relational Objects <ul><li>Location </li></ul><ul><ul><li>Most RDBMS can have multiple locations, all managed by the same database engine </li></ul></ul>Accounting Accounts Receivable Accounts Payable Marketing Sales Advertising Corporate Database Accounting Purchasing Marketing
  55. 55. The Relational Objects <ul><li>Location </li></ul>Database Server Multi-users DB Client Application Client Application Client Application
  56. 56. The Relational Objects... <ul><li>Database </li></ul><ul><ul><li>A set of SQL objects </li></ul></ul>UPDATE T SET INSERT INTO T DELETE FROM T CALL STPROG Client Application Database Server Stored Procedure BEGIN ... Table A Table B Table T Update Trigger Insert Trigger Delete Trigger BEGIN ... BEGIN ... BEGIN ...
  57. 57. The Relational Objects... <ul><li>Database </li></ul><ul><ul><li>A collection of tables and associated indexes </li></ul></ul>Table Department Table Product Table Customer Table Employee Index Files
  58. 58. The Relational Objects... <ul><li>Relation </li></ul><ul><ul><li>A named, two dimensional table of data </li></ul></ul><ul><li>Database </li></ul><ul><ul><li>A collection of databases, tables and related objects organised in a structured fashion. </li></ul></ul><ul><ul><li>Several database vendors use schema interchangeably with database </li></ul></ul>
  59. 59. Relational Objects... <ul><li>Tables are comprised of rows and a fixed number of named columns . </li></ul>Data is presented to the user as tables: Column 1 Column 2 Column 3 Column 4 Row Row Row Table
  60. 60. Relational Objects... <ul><li>Columns are attributes describing an entity. Each column must have an unique name and a data type. </li></ul>Data is presented to the user as tables: Name Designation Department Row Row Row Employee Structure of a relation (e.g. Employee) Employee(Name, Designation, Department)
  61. 61. Relational Objects... <ul><li>Rows are records that present information about a particular entity occurrence </li></ul>Data is presented to the user as tables: Name Designation Department Row Row Row Employee De Silva Manager Personnel Perera Secretary Personnel Dias Manager Sales
  62. 62. Relational Objects: Keys <ul><li>Key constraints </li></ul><ul><ul><li>If a relation has more than one key, they are called candidate keys </li></ul></ul><ul><ul><li>One of them is chosen as the primary key </li></ul></ul><ul><li>Relational Objects: Keys </li></ul><ul><ul><li>Primary Key: An attribute (or combination of attributes) that uniquely identifies each row in a relation. </li></ul></ul><ul><ul><ul><li>Employee( Emp_No , Emp_Name, Department) </li></ul></ul></ul><ul><ul><li>Composite Key: A primary key that consists of more than one attribute </li></ul></ul><ul><ul><ul><li>Salary( Emp_No , Eff_Date , Amount) </li></ul></ul></ul>
  63. 63. Relational Objects: Keys <ul><li>Each table has a primary key . The primary key is a column or combination of columns that uniquely identify each row of the table. </li></ul>Data is presented to the user as tables: Primary Key Employee E-No E-Name D-No 179 Silva 7 857 Perera 4 342 Dias 7 Primary Key Salary E-No Eff-Date Amt 179 1/1/98 8000 857 3/7/94 9000 179 1/6/97 7000 342 28/1/97 7500
  64. 64. Relational Objects: Relationship <ul><li>Foreign Key: An attribute in a relation of a database that serves as the primary key of another relation in the same database </li></ul><ul><ul><li>Employee( Emp_No , Emp_Name, Department) </li></ul></ul><ul><ul><li>Department( Dept_No , Dept_Name, M_No) </li></ul></ul>=== works for ==>
  65. 65. Relational Objects : Foreign Key <ul><li>A foreign key is a set of columns in one table that serve as the primary key in another table </li></ul>Data is presented to the user as tables: Foreign Key Primary Key Primary Key D-No D-Name M-No 4 Finance 857 7 Sales 179 Primary Key Department Rows in one or more tables are associated with each other solely through data values in columns (no pointers). Employee E-No E-Name D-No 179 Silva 7 857 Perera 4 342 Dias 7
  66. 66. SQL <ul><li>A relational database language </li></ul><ul><ul><li>It is not a programming language but a comprehensive database sub-language language for controlling and interacting with a database management system. </li></ul></ul><ul><li>NOT a DBMS </li></ul><ul><li>A powerful data manipulation language </li></ul><ul><ul><li>It has capabilities for: </li></ul></ul><ul><ul><ul><li>insertion </li></ul></ul></ul><ul><ul><ul><li>update </li></ul></ul></ul><ul><ul><ul><li>deletion </li></ul></ul></ul><ul><ul><ul><li>query </li></ul></ul></ul><ul><ul><ul><li>Protection </li></ul></ul></ul>
  67. 67. SQL (Cont’) <ul><li>Also designed for end users </li></ul><ul><li>Non-procedural </li></ul><ul><ul><li>We have to show ‘what is needed’ and not ‘how’, like in ‘relational algebra’ </li></ul></ul><ul><ul><li>Is similar more to ‘relational calculus’ </li></ul></ul><ul><li>Used in two ways: </li></ul><ul><ul><li>Interactive </li></ul></ul><ul><ul><li>Programmatic: Dynamic / Embedded </li></ul></ul>
  68. 68. Role of SQL <ul><li>A database programming language </li></ul><ul><li>A database administration language </li></ul><ul><li>A client/server language </li></ul><ul><li>A distributed database language </li></ul>
  69. 69. Role of SQL <ul><li>It is vendor independent. </li></ul><ul><ul><li>If a user was dissatisfied with a particular DBMS he could switch products easily without much overhead, as both would follow the same language standard. </li></ul></ul><ul><li>Client applications relatively portable. </li></ul><ul><li>Programmer skills are portable. </li></ul><ul><li>Supports many different client processes -- end-users, applications, developers, etc. </li></ul><ul><li>Database servers use SQL to request services from each other. </li></ul>
  70. 70. SQL Basics: DDL <ul><li>CREATE TABLE Adds new table </li></ul><ul><li>DROP TABLE Removes existing tables </li></ul><ul><li>ALTER TABLE Modifies structure of tables </li></ul><ul><li>CREATE VIEW Adds a new view </li></ul><ul><li>DROP VIEW Removes a view </li></ul><ul><li>CREATE INDEX Build an index for a column </li></ul><ul><li>DROP INDEX Removes an index </li></ul><ul><li>CREATE SYNONYM Defines an alias for a database object </li></ul><ul><li>DROP SYNONYM Remove an alias </li></ul><ul><li>COMMENTS Describes a table or column </li></ul><ul><li>LABEL Defines a title for a table or column </li></ul>Data Definition Language (DDL) DDL defines the database: Physical Design
  71. 71. SQL Basics: DML <ul><li>SELECT Retrieves data </li></ul><ul><li>INSERT Adds new rows of data </li></ul><ul><li>DELETE Removes row of data </li></ul><ul><li>UPDATE Modifies existing data </li></ul>Data Manipulation Language (DML)
  72. 72. SQL Basics: DCL <ul><li>GRANT Gives user access privileges </li></ul><ul><li>REVOKE Removes privileges </li></ul><ul><li>COMMIT Ends current transaction </li></ul><ul><li>ROLLBACK Aborts current transaction </li></ul>Data Control Language (DCL)
  73. 73. SQL Basics: Data Integrity <ul><li>Value of stored data can be lost in many ways: </li></ul><ul><ul><li>Invalid data added to data base </li></ul></ul><ul><ul><li>Existing data modified to a incorrect value </li></ul></ul><ul><ul><li>Changes made lost due to system error or power failure </li></ul></ul><ul><ul><li>Changes partially applied </li></ul></ul><ul><li>Types of integrity constraints: </li></ul><ul><ul><li>Required Data (NOT NULL) </li></ul></ul><ul><ul><li>Validity Checking (CHECK) </li></ul></ul><ul><ul><li>Entity Integrity (PRIMARY KEY & NOT NULL) </li></ul></ul><ul><ul><li>Referential Integrity (FOREIGN KEY) </li></ul></ul><ul><ul><li>Business Rules (ASSERTION, TRIGGER) </li></ul></ul><ul><ul><li>Consistency (CASCADE, RESTRICT, SET NULL) </li></ul></ul>
  74. 74. SQL Basics: NULL values <ul><li>Null values provides a systematic way of handling missing or inapplicable data in SQL. </li></ul><ul><li>It is inevitable that in real-world, some data are missing, not yet known or do not apply. </li></ul><ul><li>Null value is not a real data value. </li></ul><ul><li>Special Handling </li></ul><ul><ul><li>Null values require special handling by SQL and the DBMS. Null values can be handled inconsistently by various SQL products </li></ul></ul><ul><ul><li>Example: How do we handle null values in summaries like SUM, AVERAGE, etc.? </li></ul></ul>
  75. 75. Referential Integrity Referential integrity constraints define the rules for associating rows with each other, i.e. columns which reference columns in other tables: Every non-null value in a foreign key must have a corresponding value in the primary key which it references. A row can be inserted or a column updated in the dependent table only if (1) there is a corresponding primary key value in the parent table, or (2) the foreign key value is set null. Department ( Parent Table ) Dept-No D1 D3 D2 D7 Employee( Dependent Table ) Dept-No D7 ? D1 D3 ? D7 Emp-No D2 INSERT ROW UPDATE COLUMN
  76. 76. Referential Integrity Deleting parent rows <ul><li>Database designers must explicitly declare the effect if a delete from the parent table on the dependent table: </li></ul><ul><ul><ul><li>CASCADE deletes associated dependent rows </li></ul></ul></ul><ul><ul><ul><li>RESTRICT will not allow delete from the parent table if there are associated dependent rows. </li></ul></ul></ul><ul><ul><ul><li>SET NULL sets the value of associated dependent columns to null values. </li></ul></ul></ul>Department ( Parent Table ) Dept-No D1 D3 D2 D7 Dept-No D7 ? D1 D3 ? D7 Emp-No D2 DELETE ROW CASCADE RESTRICT SET NULL
  77. 77. SQL for Data Manipulation <ul><li>Manipulation </li></ul><ul><ul><li>SQL allows a user or an application program to update the database by adding new data, removing old data, and modifying previously stored data. </li></ul></ul><ul><li>Retrieval </li></ul><ul><ul><li>SQL allows a user or an application program to retrieve stored data from the database and use it. </li></ul></ul><ul><li>Most Commonly Used Commands </li></ul><ul><ul><li>SELECT INSERT </li></ul></ul><ul><ul><li>UPDATE DELETE </li></ul></ul>
  78. 78. SQL for Data Manipulation <ul><li>High-level Language for data manipulation </li></ul><ul><li>It does not require predefined navigation path </li></ul><ul><li>It does not require knowledge of any key items </li></ul><ul><li>It is uniform language for end-users and programmers </li></ul><ul><li>It operates on one or more tables based on set theory, not on a record at a time. </li></ul>
  79. 79. Command: SELECT <ul><li>Function: </li></ul><ul><ul><li>Retrieves data from one or more rows. Every SELECT statement produces a table of query results containing one or more columns and zero or more rows. </li></ul></ul><ul><li>SELECT {[ ALL , DISTINCT]} select-item, ) ,  ] </li></ul><ul><ul><ul><li>FROM ( table specification, ) </li></ul></ul></ul><ul><ul><ul><li>{ WHERE ( search condition )} </li></ul></ul></ul><ul><ul><ul><li>{ GROUP BY ( group-column, )} </li></ul></ul></ul><ul><ul><ul><li>{ HAVING ( search condition )} </li></ul></ul></ul><ul><ul><ul><li>{ ORDER BY ( sort specification, )} </li></ul></ul></ul>
  80. 80. Project Selected Columns The &quot; Persons &quot; table : SELECT LastName,FirstName FROM Persons SELECT P_id, Last Name, First Name FROM Persons ORDER BY LastName Stavanger Storgt 20 Kari Pettersen 3 Sandnes Borgvn 23 Tove Svendson 2 Sandnes Timoteivn 10 Ola Hansen 1 City Address FirstName LastName P_Id Kari Pettersen Tove Svendson Ola Hansen FirstName LastName Tove Svendson 2 Kari Pettersen 3 Tom Nilsen 4 Ola Hansen 1 FirstName LastName P_Id
  81. 81. Restrict Rows The &quot; Persons &quot; table : SELECT * FROM Persons WHERE City='Sandnes' Sandnes Borgvn 23 Tove Svendson 2 Sandnes Timoteivn 10 Ola Hansen 1 City Address FirstName LastName P_Id Stavanger Storgt 20 Kari Pettersen 3 Sandnes Borgvn 23 Tove Svendson 2 Sandnes Timoteivn 10 Ola Hansen 1 City Address FirstName LastName P_Id
  82. 82. Equal Join The &quot; Persons &quot; table : The &quot;Orders&quot; table: SELECT Persons.LastName, Persons.FirstName, Orders.OrderNo FROM Persons, Orders WHERE Persons.P_Id = Orders.P_Id ORDER BY Persons.LastName Stavanger Storgt 20 Kari Pettersen 3 Sandnes Borgvn 23 Tove Svendson 2 Sandnes Timoteivn 10 Ola Hansen 1 City Address FirstName LastName P_Id 15 34764 5 1 24562 4 1 22456 3 3 44678 2 3 77895 1 P_Id OrderNo O_Id 44678 Kari Pettersen 77895 Kari Pettersen 24562 Ola Hansen 22456 Ola Hansen OrderNo FirstName LastName
  83. 83. SQL Data Retrieval <ul><li>Comparison </li></ul><ul><ul><li>Equal to = </li></ul></ul><ul><ul><li>Not equal to != or <> or ^= </li></ul></ul><ul><ul><li>Less than to < </li></ul></ul><ul><ul><li>Less than or equal to <= </li></ul></ul><ul><ul><li>Greater than to > </li></ul></ul><ul><ul><li>Greater than or equal to >= </li></ul></ul>Basic Search Conditions:
  84. 84. SQL Data Retrieval <ul><li>Range ( [NOT] BETWEEN) </li></ul><ul><ul><li>expres-1 [NOT] BETWEEN expres-2 AND expres- 3 </li></ul></ul><ul><ul><li>Example: WEIGHT BETWEEN 50 AND 60 </li></ul></ul><ul><li>Set Membership ( [NOT] IN) </li></ul><ul><ul><li>Example 1: WHERE Emp_No IN (‘E1’, ‘E2’, ‘E3’) </li></ul></ul><ul><ul><li>Example 2: WHERE Emp_No IN (Select Emp_No FROM Employee WHERE Dept_No=‘7’) </li></ul></ul>Basic Search Conditions:
  85. 85. SQL Data Retrieval <ul><li>Pattern Matching ([NOT] LIKE) </li></ul><ul><ul><li>expres-1 [NOT] LIKE { special-register | host-variable | string-constant } </li></ul></ul><ul><ul><li>Example: WHERE Proj_Name LIKE “INFORM%” </li></ul></ul><ul><li>Null Value (IS [NOT] NULL) </li></ul><ul><ul><li>Example: WHERE Proj_Name IS NOT NULL </li></ul></ul>Basic Search Conditions:
  86. 86. SQL Data Retrieval <ul><li>AND, OR and NOT </li></ul><ul><ul><li>Example: </li></ul></ul><ul><ul><li>WHERE Proj_Name LIKE ‘INFORM%’ AND Emp_Name = ‘DIAS’ </li></ul></ul>Compound Search Conditions:
  87. 87. SQL Query Features <ul><li>Summary Queries </li></ul><ul><ul><li>Summarize data from the database. In general, summary queries use SQL functions to collapse a column of data values into a single value that summarizes the column. (AVG, MIN, MAX, SUM, COUNT..) </li></ul></ul><ul><li>Sub-Queries </li></ul><ul><ul><li>Use the results of one query to help define another query </li></ul></ul>
  88. 88. Summarising Data The &quot; Orders &quot; table : SELECT COUNT(Customer) AS CustomerNilsen FROM Orders WHERE Customer='Nilsen' SELECT AVG(OrderPrice) AS OrderAverage FROM Orders Nilsen 100 2008/10/04 6 Jensen 2000 2008/08/30 5 Hansen 300 2008/09/03 4 Hansen 700 2008/09/02 3 Nilsen 1600 2008/10/23 2 Hansen 1000 2008/11/12 1 Customer OrderPrice OrderDate O_Id 2 CustomerNilsen 950 OrderAverage
  89. 89. GROUP BY SELECT Customer,SUM(OrderPrice) FROM Orders GROUP BY Customer The &quot; Orders &quot; table : A result of a previous specified clause is grouped using the group by clause. Nilsen 100 2008/10/04 6 Jensen 2000 2008/08/30 5 Hansen 300 2008/09/03 4 Hansen 700 2008/09/02 3 Nilsen 1600 2008/10/23 2 Hansen 1000 2008/11/12 1 Customer OrderPrice OrderDate O_Id 2000 Jensen 1700 Nilsen 2000 Hansen SUM(OrderPrice) Customer
  90. 90. HAVING Used for select groups that meet specified conditions. Always used with GROUP BY clause. The &quot; Orders &quot; table : SELECT Customer,SUM ( OrderPrice ) FROM Orders GROUP BY Customer HAVING SUM ( OrderPrice ) <2000 Nilsen 100 2008/10/04 6 Jensen 2000 2008/08/30 5 Hansen 300 2008/09/03 4 Hansen 700 2008/09/02 3 Nilsen 1600 2008/10/23 2 Hansen 1000 2008/11/12 1 Customer OrderPrice OrderDate O_Id 1700 Nilsen SUM(OrderPrice) Customer
  91. 91. Nested Queries A sub query is SELECT statement that nest inside the WHERE clause of another SELECT statement. The results are need in solving the main query. The &quot; Orders &quot; table : SELECT Customer FROM Orders WHERE OrderPrice> ( SELECT AVG ( OrderPrice ) FROM Orders ) Nilsen 100 2008/10/04 6 Jensen 2000 2008/08/30 5 Hansen 300 2008/09/03 4 Hansen 700 2008/09/02 3 Nilsen 1600 2008/10/23 2 Hansen 1000 2008/11/12 1 Customer OrderPrice OrderDate O_Id Jensen Nilsen Hansen Customer
  92. 92. Case Study
  93. 93. Case Study: Wal*Mart <ul><li>Founded by Sam Walton </li></ul><ul><li>One the largest Super Market Chains in the US </li></ul><ul><li>Wal*Mart: 2000+ Retail Stores </li></ul><ul><li>SAM's Clubs 100+Wholesalers Stores </li></ul><ul><ul><ul><li>This case study is from Felipe Carino’s (NCR Teradata) presentation made at Stanford Database Seminar </li></ul></ul></ul>
  94. 94. Old Retail Paradigm <ul><li>Wal*Mart </li></ul><ul><ul><li>Inventory Management </li></ul></ul><ul><ul><li>Merchandise Accounts Payable </li></ul></ul><ul><ul><li>Purchasing </li></ul></ul><ul><ul><li>Supplier Promotions: National, Region, Store Level </li></ul></ul><ul><li>Suppliers </li></ul><ul><ul><li>Accept Orders </li></ul></ul><ul><ul><li>Promote Products </li></ul></ul><ul><ul><li>Provide special Incentives </li></ul></ul><ul><ul><li>Monitor and Track The Incentives </li></ul></ul><ul><ul><li>Bill and Collect Receivables </li></ul></ul><ul><ul><li>Estimate Retailer Demands </li></ul></ul>
  95. 95. New (Just-In-Time) Retail Paradigm <ul><li>No more deals </li></ul><ul><li>Shelf-Pass Through (POS Application) </li></ul><ul><ul><li>One Unit Price </li></ul></ul><ul><ul><ul><li>Suppliers paid once a week on ACTUAL items sold </li></ul></ul></ul><ul><ul><li>Wal*Mart Manager </li></ul></ul><ul><ul><ul><li>Daily Inventory Restock </li></ul></ul></ul><ul><ul><ul><li>Suppliers (sometimes SameDay) ship to Wal*Mart </li></ul></ul></ul><ul><li>Warehouse-Pass Through </li></ul><ul><ul><li>Stock some Large Items </li></ul></ul><ul><ul><ul><li>Delivery may come from supplier </li></ul></ul></ul><ul><ul><li>Distribution Center </li></ul></ul><ul><ul><ul><li>Supplier’s merchandise unloaded directly onto Wal*Mart Trucks </li></ul></ul></ul>
  96. 96. Wal*Mart System <ul><li>NCR 5100M 96 Nodes: 24 TB Raw Disk; 700 - 1000 Pentium CPUs </li></ul><ul><li>Number of Rows: > 5 Billions </li></ul><ul><li>Historical Data: 65 weeks (5 Quarters) </li></ul><ul><li>New Daily Volume: Current Apps: 75 Million </li></ul><ul><li> New Apps: 100 Million + </li></ul><ul><li>Number of Users: Thousands </li></ul><ul><li>Number of Queries: 60,000 per week </li></ul>
  97. 97. References/External Links (1) Data Warehousing & Data Mining S. Sudarshan Krithi Ramamritham IIT Bombay (2) Data Warehousing Hu Yan e-mail: [email_address] (3) What is a Data Warehouse? http://blog.maia-intelligence.com/2008/04/29/what-is-a-data-warehouse/ (4) Database Management Systems (DBMS) http://www.bit.lk/teachingmaterial/IT2302/index.htm (5) SQL Tutorial http :// www . w3schools . com / sql / default . asp
  98. 98. Thank you for your attention! [email_address] www.blueballgroup.com