Real-Time Data Integration for Modern BI

Real-time Data Integration for Modern BI
Claudia Imhoff
Intelligent Solutions, Inc.
Boulder BI Brain Trust
Jake Freivald
Information Builders

Copyright © 2016, Intelligent Solutions, Inc. All Rights Reserved
Claudia Imhoff
2
President
Intelligent Solutions, Inc.
Founder
Boulder BI Brain Trust
A thought leader, visionary, and practitioner, Claudia
Imhoff, Ph.D., is an internationally recognized expert
on analytics, business intelligence, and the
architectures to support these initiatives. Dr. Imhoff
has co-authored five books on these subjects and
writes articles (totaling more than 150) for technical
and business magazines.
She is also the Founder of the Boulder BI Brain Trust,
a consortium of internationally recognized
independent analysts and experts. You can follow
them on Twitter at #BBBT or become a subscriber at
www.bbbt.us.
Email: claudia@bbbt.us
Phone: 303-444-6650
Twitter: Claudia_Imhoff

Agenda
 Introduction to Operational BI
 The Extended Data Warehouse Architecture
 Putting It All Together – Examples
 Getting Started with Operational BI
3

What is Operational BI?
A set of applications, services, and technologies
for monitoring, reporting on, analyzing, and
managing the business performance of an
organization’s daily business operations
4

The Business Case
 Enables more informed business decisions by directly
supporting specific business processes and activities
 Supports faster business decisions by seamlessly
integrating BI with business processes to create a
closed-loop decision-making environment
 Provides a more dynamic business environment where
the business can learn, adapt, and evolve based on
analysis of its operational business performance
5

Agenda
6

Next Generation – Extended Data
Warehouse Architecture (XDW)
Traditional EDW
environment
Investigative computing
platform
Data
refinery
Data integration
platform
Analytic tools & applications
Operational real-time environment
RT analysis
platform
Other internal & external
structured & multi-structured
data
Real-time streaming data
Operational systems
BI services
7

Data Provisioning Use Case:
Data Integration
8
 Heavy lifting process of extracting,
transforming to standard format
and loading structured data –
mostly batch
 Physically consolidates data into
“trusted” EDW sets for analysis
 Invokes data quality processing
where needed
 Employs low-cost hardware and
software to enable large data
volumes to be combined and stored
 Requires more formal governance
policies to manage data security,
privacy, quality, archiving and
destruction
Traditional EDW
environment
platform
Data
refinery
Data integration
platform

Data Provisioning Use Case:
Data Refinery
9
 Ingests raw detailed structured and
unstructured data in batch and/or
real-time into a managed data store
 Distills data into useful business
information and distributes the
results to downstream systems
 May also directly analyze certain
types of data
 Also employs low-cost hardware
and software to enable large
amounts of detailed data to be
managed cost effectively
 Requires (flexible) governance
policies to manage data security,
privacy, quality, archiving and
destruction
Traditional EDW
environment
platform
Data
refinery
Data integration
platform

Traditional EDW Use Cases
10
Most BI environments today
 New technologies can be
incorporated into EDW
environment to improve
performance, efficiency & reduce
costs
Use cases
 Production reporting
 Historical comparisons
 Customer analysis (next best
offer, segmentation,
life-time value scores,
churn analysis, etc.)
 KPI calculations
 Profitability analysis
 Forecasting
Traditional EDW
environment
Data
refinery
Data integration
platform
RT analysis
engine
Operational systems
BI services

Investigative Computing Use
Cases
New technologies used here
include:
 Hadoop, in-memory
computing, columnar storage,
data compression,
appliances, etc.
Use cases
 Data mining and predictive
modeling for EDW and real-
time environments
 Cause and effect analysis
 Data exploration (“Did this
ever happen?” “How often?”)
 Pattern analysis
 General, unplanned
investigations of data
11
Data
refinery
Data integration
platform
platform
RT analysis
engine
Operational systems
BI services

Operational BI Use Cases
Embedded or callable BI
services:
 Real-time fraud detection
 Real-time loan risk
assessment
 Optimizing online promotions
 Location-based offers
 Contact center optimization
 Supply chain optimization
Real-time analysis engine (Event analytics):
 Traffic flow optimization
 Web event analysis
 Natural resource exploration analysis
 Stock trading analysis
 Risk analysis
 Correlation of unrelated data streams
(e.g., weather effects on product sales)
12
RT analysis
platform
Other internal & external
structured & multi-structured
data
Real-time streaming data
Operational systems
BI services

Operational
BI
Operational BI Environment
Consists of:
RT Event
Analytics
Investigative
Analytics
13
Embedded BI
Services

Agenda
14

All Components Must Work
Together
15
analytic models
other analytics
New sources of data
Enterprise DW
Analytic tools
Investigative
computing
platform
Data refinery
Operational
systems
production
analyses
next best
customer offer
location-based offer
fraud detection
3rd party data
location data
social data
feedback
RT analysis engine
call center dashboard
or web event stream
NOTE: data virtualization has a big
role in the combining of analytics

Example – Online Promotions
 Real-time campaigns
 Adjust pricing and offers every
few minutes based on
inventory levels and customer
responses
 Faster reaction to supply-
demand imbalances
 Immediate customer metrics
 New customer, retention,
repeat buys
 Average revenue, profit, etc.
 Faster operational
(merchandizing) reports
16

Example – Location-based
Offers
 Empower local stores to do own special events and
promotions
 Non-technical, in-store personnel must be able to select
customer groups based on local criteria
 Appropriate reminder and incentive offers
 Exclusive invitations, coupons, etc., pushed to mobile
devices based on consumer history, proximity to store,
external events
17

Example – Algorithmic Trading
 Used in capital markets
to indicate when to buy or
sell stocks
 Applies event processing
by calculating complex
algorithms on the fly
 Example: When GM’s
price is .5% higher than
average price in last 30
seconds, buy 10,000 share
of Ford every 3 seconds
until average price drops
18

Example – Retail Supply Chain
Automation
 Use RFID to automate
supply chain
 Can track, trace items
wherever they are, at any
time
 RFID-sourced events must
be quickly collected,
organized, managed by
data management
infrastructure
 Event data volumes may
not be large but event
rates are fast
19

Example – Fraud Detection
 Idea is to detect fraudulent
transactions faster
 Credit card transactions
 Telecom transactions
 Claims
 Models are developed from
known fraudulent events in
data warehouse and fed to
streaming analytics engine
 Human can intercede,
eliminate proven fraudulent
transactions quickly
20

Other Operational BI Examples
 Web-site analytics
 Loan and credit card application processing
 Insurance claim processing
 Improving airline customer satisfaction (flight
delays, baggage handling, call-center handling)
 Monitoring and optimizing equipment servicing
and quality management (IoT)
 Package shipment, routing, and delivery
optimization
 Network outage tracking, prediction & correction
21

Agenda
22

Barriers and Challenges*
23
* From: tdwi.org/research/2009/09/ten-mistakes-to-avoid-when-designing-and-developing-operational-bi-applications.aspx?tc=page0

Does Your Business Need
Real-Time BI?
 Understand what the requirements for real time delivery of
data means to the business community
 What MUST be real time?
 What CAN be minutes to hours old?
 What CAN be days, weeks old?
 How important is real-time access and analytics to your
company – what is the ROI for real-time processing?
 Remember costs and complexity go up as need for real
time data and analysis increases
24

Cost Considerations: Operational
BI is Mission Critical
 Sound infrastructure
 A robust data integration
architecture (data analytics)
 An integrated BI and business
process environment
(streaming analytics)
 A services-based operational
BI infrastructure
 Support for mixed workloads
 Data analytics from DW and
experimental areas
 Streaming analytics from RT
analytics engine
 Reliable service levels – High
availability/fault tolerance
25

Assess Reality – 1
 ID time continuum for data in operational BI processes
 Assess existing data delivery capabilities
 Understand business operational BI requirements
 What is right time for accessing data?
 Use a time continuum (from real-time to high latency) to map
business needs to type and cost of data delivery approach used
26
Traditional query
Time
At-rest data
(past events)
Start
time End
time
processes existing data
processes real-time event streams
Stream query
In-motion data
(continuous events)

Assess Reality – 2
 Understand which weaknesses will be
exaggerated as you speed up analytical and
operational processes
 Weak data quality processes
 Weak ETL processes
 Weak workflow integration
 Weak adherence to procedures by employees
27

Pick a Project - 1
 Two options:
 Improve function and/or speed of existing BI solution
 Build a new operational BI solution that solves a new
or important business need
1. Improved BI solution
 More sophisticated analytics,
e.g., predictive analytics fed to operational personnel
 Reduce time – may be evolutionary process based
on experience, e.g., daily to intra-day to near real-
time data and analyses
28

Pick a Project - 2
2. New BI solution
 Recognized as having significant impact on
operations but achievable in reasonable time frame,
e.g., 3 to 6 months
 May fill a gap in current operations, e.g., fraud
detection, location-based offers
 May be made possible by advances in BI technology,
e.g., event analytics, extreme analytic DBMS
platform
29

Pick a Project - 3
 Look for operational workflow activities that
have a major impact on costs or revenues
 Bottlenecks or issues that can be made more
efficient through the use of operational BI
 Understanding business processes & costs of
activities within business processes is a critical
success factor
30

Pick a Project - 4
 Avoid making big changes
to operational processes
 Operational BI is an
evolutionary process
 Focus on improving the
speed and efficiency of
existing processes
 Realize you may have to
modify existing standard
operational policies and
procedures
 May need alerts if procedures
are not followed
31

Operational BI: Users are
Different
 Traditional BI supports a few
100 (or maybe a 1,000) users
 Opening BI up to operational
personnel means ramping up
to many more users
 May increase both software
and hardware costs
 Means tighter, faster
connectivity of enterprise
decision support environment
to rest of company
32

Operational BI: Dashboard
Example
 Mechanism to present analytics, alerts,
recommendations, decision workflows, operational KPIs
 Used to help make proactive decisions
 Send alerts and recommendations to other personnel
 Ability to drill down to further understand what’s happening
 Must be self-contained and easy to use – personalized
to each manager’s needs
 Self-service – Business person can change metrics, thresholds
being tracked; presentation and layout of results
 Easy to modify as business scenarios and needs evolve
 Level of changes allowed will depend on role of user
33

Operational Impact
 Workforce may need to be revamped – will
likely require retraining of operational personnel
 How they make decisions
 How they access and use BI information
 How they monitor impact of their decisions
 Training needs to be ongoing and flexible
 Some may not make the leap
34

Presentation Requirements – 1
 Continuous availability of
operational data and BI results
 Current information from
operational systems
 Integrated with BI analytics on
demand
 Minimal impact on operational
systems performance
 Presented in proactive manner
 May need dynamic modeling
 Business person may change
business rules on the fly
 Show different set of metrics
depending on situation
35

Presentation Requirements – 2
 Transparent underlying infrastructure
 Study their access methods and needs
 Develop appropriate dashboards, portals, or other interfaces
according to these needs
 Monitor community’s usage patterns and revamp, revise the
interface as needed
 Workbench based on workflow
 Bring together appropriate historical BI results, BI services,
streaming events, operational capabilities to support workflows
 Consider data virtualization to minimize data movement, simplify
changes
 Remain flexible – things will change!
36

Infrastructure Requirements - 1
 Operational BI is mission critical
 Both operational and BI
infrastructure may need
upgrading to meet service-level
agreements
 BI performance (# of users,
response times, volumes of data,
types of analytics performed)
 Availability, reliability, security
 Impact on operational processing
 Need accurate predictions of
data and user growth
37

Infrastructure Requirements - 2
 Solution should have flexible
(elastic) scalability, query support
and capacity
 Support for workload
management & mixed workflows
essential
 POC required to ensure all parts
work
 Data integration and data quality
will be different
 Sometimes “good enough” data
is good enough …
 Other times, data must be
perfect!
38

Data Integration Main
Considerations
 Consider:
 Source data type, organization, and scale
 Data quality requirements and techniques
 Source data transformation & validation
rules
 Target data type, organization, scale,
latency, granularity
 Data integration techniques & technologies
 Data integration approaches for data
analytics
 Data virtualization – connect BI insights
to operational data
 Change data capture (CDC)
 Data staging: ELT, streaming data &
Hadoop filtering
 Data integration infrastructure is THE
cornerstone of operational BI
39

Operational BI: 10 Mistakes to
Avoid
1. Assuming All Analytics Must Come from the Data Warehouse Alone
2. Failing to Match BI Agility to Business Needs
3. Failing to Determine if the Infrastructure Can Support Operational BI
4. Assuming that Operational BI is Just a Technology Solution
5. Assuming that Operational BI Simply Involves Capturing More Timely Data
6. Assuming Existing Data Quality Procedures Will Work for Operational BI
7. Failing to Realize Operational BI is Process-Centric Rather than Data-Centric
8. Assuming Operational BI IT Skills are the Same as Those for Other BI Types
9. Assuming Users of Operational BI are Same as Those for Other Types of BI
10.Failing to Monitor and Audit Automated Decision Making
From TDWI’s 10 Mistakes to Avoid When Designing and Developing Operational BI
Applications: tdwi.org/research/2009/09/ten-mistakes-to-avoid-when-designing-and-
developing-operational-bi-applications.aspx?tc=page0
40

Operational Analytics and BI
An Information Builders Perspective

The Information Builders 3i Capabilities
Portal Embedded InfoApps™
ApplicationsLegacy Systems Relational/Cubes Big Data Columnar/In Memory Unstructured Social Media Web Services Trading Partners
Integration
Mobile Write-Back
Data
Discovery
Reporting Dashboards
High-Performance
Data Store
Data
Quality
Data
Governanc
e
Master Data
Management
Batch
ETL
Real-Time
ESB
Integrity
Intelligence
Location
Analytics
Casting
and Archiving
In-Document
Analytics
SearchPredictive
Analytics
Sentiment and
Word Analytics
Performance
Management
Social
Hot
Bad
Feedback

Real-time / Operational as the New Normal

Different Sources, Different Latencies, Same Screen

The Power of Information Builders
 Create operational insights, analytical content
…and drive them immediately to 7x-20x more users
45
Interoperable integration, intelligence, and integrity

 Create predictive models
…and use them to score data in motion (e.g., into Hadoop)
46

 Leverage a Real-time Data Quality Firewall
47

 Virtualize data across operational data stores
48

 Virtualize data across operational data stores
 Drive information directly to users in real-time
Example…
49

Sample Architecture
50
iWay Service
Manager
WebFOCUS
Apache
ActiveMQ
Web Server
(WebFOCUS BIP)
Real-Time
Feed
Web Browser

Sample Architecture Benefits
 Event driven by input stream, real real-time
 Application content centrally managed by iSM
 Information is pushed to the consumer, not pulled
 Application logic not necessary in browser
 Scalable: a single object can be pushed to all users, no
handling of AJAX requests per browser session
 Reusability: output content stream available for any
application, not just browsers
 Based on accepted open standard protocols
51

Sample Architecture Benefits
 Event driven by input stream, real real-time
 Application content centrally managed by iSM
 Information is pushed to the consumer, not pulled
 Application logic not necessary in browser
 Scalable: a single object can be pushed to all users, no
handling of AJAX requests per browser session
 Reusability: output content stream available for any
application, not just browsers
 Based on accepted open standard protocols
52

Hadoop in Operational BI
 Generate data sources for business users / exploration
 Not everything needs to be modeled
 E.g., 1/2 to 1/3 of customer data lives in a data
warehouse
 How do you manage analytics that involves other data?
Traditional: DW
DWApp
New
Traditional: Op’l
DWApp
New
Modern
DWApp
New
Traditional: Rogue
DWApp
New

Integrating Big Data
54
Data at rest and data in motion
Sqoop,Flume…
Avro,JSON…
Traditional
applications
and data stores
iWay Hadoop Data Manager
Simplified, modern,
native Hadoop integration
Big Data Hadoop
Any distribution, Any data
BI &
Analytics WebFOCUS BI and analytics platform
Self-service for Everyone
WebFOCUS access,
ETL, metadata
WebFOCUS access,
ETL, metadata

Simplified
interface
Native Hadoop
script generation
Process mgmt
& governance
 Simplified, easy-to-use interface
to integrate in Hadoop
 Marshals Hadoop resources
and standards
 Takes advantage of performance
and resource negotiation
 Includes sophisticated process
management & governance
Sqoop,Flume…
Avro,JSON…
Data Sources
Big Data Native: Runs in Hadoop cluster
Purpose-built: Exploits all Hadoop services
Simple: Replaces coding with mapping
Integrating Big Data
55
Data at rest and data in motion

Real-Time Data Integration for Modern BI

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (19)

Ähnlich wie Real-Time Data Integration for Modern BI

Ähnlich wie Real-Time Data Integration for Modern BI (20)

Mehr von ibi

Mehr von ibi (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Real-Time Data Integration for Modern BI