2. Agenda
• Top Business Imperatives and Data
Requirements to Succeed <Insert Picture Here>
• Data Warehouse Basics and Challenges
• The Purpose of Data Warehouse
• Why Real-Time Data Warehouse for BI?
• Real-Time Data Integration Considerations
• Traditional Vs Real-time Data Warehouse
• Data Quality & Profiling
• Oracle Data Integration Solution
• Customer Case Studies
2
3. Top Business Imperatives and Data
Requirements to Succeed
Access to Timely, Trusted, and Consistent data
Operational & Analytical
Business Applications
Risk CRM / Direct
Improve Management Marketing
Mergers &
Decisions & Acquisitions
Regulatory
Compliance
Real-Time Data Quality
Data
Integration Data
DataMart / MDM /
Migration &
BI DWH / BI Consolidation
BI
IT Projects
3
4. Why Real-Time DataWarehouse for BI?
1. Business Driver: Real-Time= Relevant
and actionable
To compete more effectively information
by using better insights in
day to day operations and
Improved insights for
strategic decisions operational decision
making
2. Technical Driver
Better customer service
To move away from batch and cost savings via
ETL jobs and eliminate operational efficiencies
the impact it has on the IT
infrastructure Increased profitability,
customer retention, and
competitive advantage
4
5. Integration Challenges
Fragmented Approach
Analytics Business
Packaged Intelligence
Custom
Reporting Applications
Accessibility Enterprise
Performance
Data
Replication
Data
Migration Data
Warehousing
Up-To-Date
Data Silos Data Marts Data Hubs
Trusted
Data
Federation
Data Access
information Batch Scripts
Custom
SQL
InformationJava
OLTP & ODS Oracle, PeopleSoft, Files, Excel OLAP
Systems Data Warehouse,
Data Warehouse,
Data Warehouse, Siebel, SAP, XML
Data Mart
Data Mart
Data Mart Custom Apps
High Cost of Lack of clean, Multiple standards,
Custom Coding consistent data disciplines
5
6. Data Warehouse and It’s Process
A centralized repository containing comprehensive detailed and
summary data that provides a complete view of customers,
suppliers, business processes, and transactions, from a
historical perspective with little volatility.
Typically loaded on a nightly basis with batch extracts from source
transaction processing systems such as CRM, ERP, etc to support reporting
and analysis.
Sources Target
DATA WAREHOUSE
ERP/
CRM
Integrate DM1 DM2
Data Cleansing Load
DM3 DM4
Extract Data
RDBMS
Enrichment
Transform
Analytical Reporting
Ad-hoc Reporting
Dashboards
Scorecards
6
7. Business Drivers For a Modern Data Integration
Platform & Real-Time DWH
Demand for Continuous, Real-Time,
Trusted Information
Real-Time
Enterprise
Continuous Availability Real-Time Data
for 24/7 Global Operations for Intelligence & Operations
• Continuous uptime in event of disaster • Up-to-the second data for operations
• No downtime during planned outage • Access to timely information for analysis
• Low-impact data capture for integration • Data distribution across regions
Trusted Information
• Consistent with other systems
•High data integrity
7
8. Traditional Vs Real-Time Data Warehouse
Traditional Data Warehouse Real-Time Data Warehouse
Transformation & Quality
(ETL / ELT &
Data Cleansing)
ETL + CDC
Real-Time
Legacy, Continuous Feeds
Packaged Enterprise
Apps, Data Legacy,
OLTP Warehouse Packaged Enterprise Data
Databases Apps, OLTP Warehouse/MDM
• Day+ old data • Timely, relevant data are continuous
feed from operational systems
• Batch data extracts within specified
• No batch windows on OLTP
“off business hours” • Sub-second latency
• A middle-tier server for • No impact on source systems
transformations • Read-consistent changed data with
referential integrity
• Process interruptions impact data
• Transformations at capture, delivery or
recoverability within the database
8
10. Oracle Data Integration: 3 Key Products
Addressing operational and strategic analysis
Oracle Data Quality
Oracle Data Integrator Oracle GoldenGate & Profiling
• E-LT • Real-Time data • Discover data
• Bulk data movement capture and delivery problems
• Complex • Low impact and non- • Global data cleansing
transformation invasive • Data standardization
• Easy to use GUI • Transactional integrity • Fuzzy matching
design • Guaranteed delivery • Heterogeneous
• Data lineage & impact • Delivers continuous
analysis availability
• Heterogeneous • Heterogeneous
Accessibility Up-to-Date Information Trusted Information
10
11. Oracle Data Integrator Enterprise Edition
Optimized E-LT for Improved Performance
Legacy
Sources
E-LT Transformation Any Data
vs. E-T-L Warehouse
Application Declarative Set-based
Sources
design
Pluggable Knowledge Any
Modules Planning
System
OLTP DB Hot-pluggable Architecture
Sources
Change Data Capture for
Dynamic Updates
11
12. Data Warehouse Bulk Loading w/E-LT
Fastest ELT Solution for your Data Warehouse
Solution
• ODI for bulk loading Data Warehouse
Extract
• Run ODI Agent within Data
Warehouse JVM
• Fastest possible data
Load transformations
• Heterogeneous and loads any 3rd
party data warehouse.
Benefits
Transform
• No extra ETL servers
• RDBMS Specific Knowledge
Modules
• Exploit DW RDBMS Optimizer
• Easier to deploy than conventional
ETL tools
• Faster time to market
• Enforce DW Best Practices
12
13. Optimized for Exadata
Fastest E-L-T Processing
• Massively parallel high volume hardware to
quickly process vast amounts of data
OLAP • Exadata runs data intensive processing
directly in storage
• Most complete analytic capabilities
• OLAP, Statistics, Spatial, Data Mining, Real-time
transactional ETL, Efficient point queries
ELT
• Powerful warehouse specific optimizations
• Flexible Partitioning, Bitmap Indexing, Join indexing,
Materialized Views, Result Cache
Data Mining • E-LT runs 20X faster only with Oracle
New
13
14. ODI is Faster
Up to 7TB per hour of real world data loading and complex transformations
ODI ELT (on Exadata)
ODI scales with Exadata
ODI runs on Exadata – no ETL hardware required
Common administration, monitoring and
management
All the benefits of rapid tools-based ETL
development
Conventional ETL
As data sets grow additional hardware ($$) needed
ETL parallel optimization and design ($$$) is heavily
dependent on resources available to the ETL system
Poor performance – transformations take place
outside of database, require staging tables
Lack of light-weight architecture for rapid data
loading
ETL engine hardware resources only used for ETL
Hardware not co located, multiple vendors
Different management, monitoring and administration
from database and BI infrastructure ($$)
14
15. Differentiator: E-LT Architecture
High Performance
Conventional: Separate ETL Server Conventional ETL Architecture
• Proprietary ETL Engine
• Poor Performance Extract Transform Load
• High Costs for Separate Standalone Server
Oracle: No New Servers
• Lower Cost: Leverage Compute Resources &
Partition Workload efficiently
• Efficient: Exploits Database Optimizer
• Fast: Exploits Native Bulk Load & Other
Database Interfaces Next Generation Architecture
• Scalable: Scales as you add Processors to
Source or Target
“E-LT”
Benefits Transform Transform
Optimal Performance & Scalability Extract Load
Better Hardware Leverage
Easier to Manage & Lower Cost
15
16. Oracle Data Profiling and Quality
Integrated Data Profiling and Quality for Customer Data
Oracle Data Profiling, Oracle Data Quality
Metadata Profiling of Source & Target
Any Source Duplicate Detection, Matching & Merging
System
Global Address Cleansing
Data Control and Visibility Any Data
Warehouse
Optimized for Customer/Party Data &
MDM
Oracle Data Integrator Enterprise Edition
Any
Planning
System
16
17. Oracle Data Quality
Ensure Data Quality as Part of the Integration Process
Best-in-class data quality and profiling for
integration processes
ODI EE
Better Data Visibility
Tighter Data Control
Greater Data Accuracy
Visual Data Quality Oracle Data Profiling Oracle Data Quality
Tools
Metadata Profiling Duplicate Detection,
of Source & Target Matching & Merging
17
18. Sample Data Quality Issues
COMPLETENESS Completeness
What data is
missing or
CONFORMITY unusable?
CONSISTENCY Consistency
What data values
give conflicting
DUPLICATION information?
Accuracy Conformity
ACCURACY What data is stored
What data is
incorrect or out of in a non-standard
date? format?
Duplication
What data records
or attributes are
repeated?
18
19. Oracle GoldenGate
Enterprise-wide Solution for Real Time Data Needs
Zero Downtime
New DB/
Migration and OS/HW/App
Upgrades
Active-Active High Fully Active • Reduce Costs
Availability Distributed Database
Log Based, Real- • Lower Risks
Time Change Data
Capture Query Offloading Reporting • Achieve
Oracle Database
GoldenGate
Operational
ETL Excellence
ODS EDW
ETL
Heterogeneous Real-time BI EDW
Source Systems
Data Distribution Global Data Centers
SOA/EDA
19
20. Data Movement plus Real-Time
Oracle GoldenGate provides low-impact capture, routing, transformation,
and delivery of transactional data across heterogeneous environments in
real time
Key Differentiators:
Performance Non-intrusive, low-impact, sub-second latency
Flexible and Extensible Open, modular architecture - Supports
heterogeneous sources and targets
Reliable Maintains transactional integrity - Resilient
against interruptions and failures
20
21. Oracle GoldenGate Architecture
Designed for Speed, Flexibility and Reliability
Trail Trail
Capture Delivery
LAN/WAN
Internet
TCP/IP
Source Target
Oracle & Non-Oracle Bi-directional Oracle & Non-Oracle
Database(s) Database(s)
• Real-time change data capture, routing and delivery across
heterogeneous systems
• Non-invasive , log-based change data capture for minimal impact on
infrastructure
• Transactional integrity and guaranteed data delivery across regions
• Bi-directional replication and support for different replication topologies
21
24. Customer Example
Leading online retailer offering a wide variety of high-quality, brand-
name merchandise at discount prices and manufacturers, distributors
and other retailers an alternative sales channel.
Challenges & Objectives Solution & Benefits
Need to enable sales, finance, marketing and Oracle GoldenGate captures real-time change data
merchandising teams with near real-time data from ecommerce and auction systems. Oracle Data
Integrator is used for highly complex transformations
and data loading to user tables
Required to have business insight on company Resulted in updated, current Teradata data warehouse
performance meeting target metrics providing critical business intelligence for decision
making
Need to be able to handle high-volume data Complete, accurate data to give LOBs a trusted view of
loading and transformation requirements like business progress, etc.
1.2M+ SKUs, 5M+ daily transactions, 300+
users
24
25. Overstock.com
Supply Demand
Manufacturers Consumers
350,000 sq. ft. User
Fulfillment Friendly
Warehouse Web Front End
Distributors Small
• Information-sensitive business Businesses
• Demanding Business Community
• Pricing, Forecasting
• Rapid Growth
25
26. Overstock.com
Innovations in Data Warehousing
Traditional Framework Emerging Framework
• Batch extracts/feeds from • Near real-time feeds from
operational systems operational systems
• Transformations in ETL engine on • Thin middle tier with E-LT
the middle tier • Transformations on the database
• Bulk load to the data warehouse platform
• Large nightly batch, user online day • Small mini-batches throughout the
day
26
28. Overstock.com: Innovations in Data Integration
Speed translates to Improved Opportunities
• Batch windows nearly eliminated
• Low-latency data provides new opportunities
• Trigger/event campaigns
• Personalization, scoring, ranking
• Marketing and merchandising improvements
• Improved Customer Service
• Now ranked #2 in the nation
• Operational improvements in scoring and
logistics led to amazing results
• Upgraded Oracle 9i Database without business
interruption
28
29. Lifetouch Portrait Studios Inc
• Serve the portraiture needs of 2M Guests annually
• 730 Photographic studios across the country
We preserve memories
and help our guest
tell their unique story…
FLASH! Digital Portraits
The Studio at Target
JCPenney Portraits
29
30. Better Customer Insight with Oracle Data
Quality
Business Challenges
Implement a single, high-quality and consistent view of each
customer to be available throughout the organization
Streamline the sharing of customer data across all the
customer facing applications
Improve the accuracy of customer data to provide a better
visibility and to elevate the customer relationships.
Oracle Solution Return on Investment
Implemented Oracle Data Quality for Improved customer data quality by a
Data Integrator to cleanse customer minimum of 25%
data. Studio workflow performance
Combined Oracle solution with improved by 10% resulting in higher
Oracle Database, MySQL and Oracle customer satisfaction and retention
Data Integrator for every data Accurate and complete customer data
movement and the overall into data warehouse enabled better
orchestration of the process customer segmentation and targeted
marketing campaign
Reduced the time and budget required
to implement data quality processes.
30
31. Benefits
• Improved customer service with real time guest and appointment
data enabling improved Studio efficiencies
• Studio workflow performance improvement of 10%
• Higher guest satisfaction and retention
• Accurate and complete guest data in the Data Warehouse
enabling
• Improved guest segmentation
• Improved targeted marketing campaigns
• Cleansed and unique guest profiles shared across primary
applications resulting in a minimum of 25% improved data quality.
• Expected reduction of technical resource hours in implementing
custom code, and performing manual data quality implementation
and audits.
• Time and budget saved with implementing ODI/ODQ solution vs.
building functionality internally.
31
32. Benefits of Real-Time DW with Oracle
Cut Costs, Reduce Risk, and Revolutionize Business Insight
Cut Costs and Improve Efficiencies.
• Move only the changed data from redo logs and reduce source and
network overhead
• Shorten implementation times from months to weeks using pre-
packaged integrations to well-known applications, sources and targets.
Reduce Risk, Ensure Continuity
• Eliminate performance impact on source systems
• Reduce the risk of missed orders, poor customer interactions, missed
opportunities through improved recoverability, data quality
Improve Business Insight
• Enable near real-time decision making with real-time data flows
• Combine real-time data with historical context for better insights
32
