Weitere ähnliche Inhalte Ähnlich wie AgileAnalytics: Agile Real-Time BI with Oracle Business Intelligence, Oracle Data Integrator and Oracle GoldenGate (20) Kürzlich hochgeladen (20) AgileAnalytics: Agile Real-Time BI with Oracle Business Intelligence, Oracle Data Integrator and Oracle GoldenGate3. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
About Red Pill Analytics
Capacity Analytics
Agile Cloud Development for Oracle BI & Analytics
!
Checkmate
DevOps for Oracle BI (Cloud-hosted or On-premises)
!
Single Doses
Training + Strategy + Solutioning + Infrastructure + Support
3
4. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Real Time BI with Kevin & Stewart
•19 episodes over the last 2
years
•Featured guests from the
Oracle BI & EPM industry
•Published on iTunes and
YouTube
•Join us at the Thursday
Deep-Dive session from
9-11am for a LIVE show that
will be simultaneously
broadcast online!
4
10. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Oracle Information Management Reference Architecture
1212
Virtualization&
QueryFederation
Enterprise
Performance
Management
Pre-built &
Ad-hoc
BI Assets
Information
Services
Data Ingestion
Information Interpretation
Access & Performance Layer
Foundation Data Layer
Raw Data Reservoir
Data
Science
Data Engines &
Poly-structured
sources
Content
Docs Web & Social Media
SMS
Structured
Data
Sources
•Operational Data
•COTS Data
•Master & Ref. Data
•Streaming & BAM
Immutable raw data reservoir
Raw data at rest is not interpreted
Immutable modeled data. Business
Process Neutral form. Abstracted from
business process changes
Past, current and future interpretation of
enterprise data. Structured to support agile
access & navigation
Discovery Lab Sandboxes Rapid Development Sandboxes
Project based data stores to
support specific discovery
objectives
Project based data stored to
facilitate rapid content /
presentation delivery
Data Sources
15. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Typical Waterfall Approach
•Interview users
•Construct requirements
•Create logical data model
•SQL prototyping of source
model
•Document source-to-target
mapping
•ETL development
•Front-end development
•Performance tuning
18
16. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Alternative: Iterative Waterfall?
•Design iterations around smaller chunks
• Iteration 1: Interviews and user requirements
• Iteration 2: Logical modeling
• Iteration 3: ETL Development
• Iteration 4: Front-end development
•Requires 4 iterations before we get any
usable content
19
18. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Manifesto for Agile Software Development
“We are uncovering
better ways of
developing software by
doing it and helping
others do it.
!
That is, while there is
value in the items on the
right, we value the items
on the left more.”
http://agilemanifesto.org/
22
19. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Manifesto Applied to AgileAnalytics
•User Stories instead of
requirements documents
• User asks for content or
functionality through a
narrative
• Typically includes current
version of the report
•Time-boxed iterations
• Iteration has a standard length
• Choose one or more user
stories to fit in that iteration
•Rework is part of the game
• There are no “missed
requirements”... only those that
haven’t been delivered yet.
23
26. RPD Development requires a BI
Server
!
RPD Development requires a query
tool
!
RPD Development requires data
visualization
30. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
GoldenGate for AgileAnalytics: Loading the Foundation Layer
35
map SUGARCRM.ACCOUNTS,
target SUGARFND.ACCOUNTS,
KEYCOLS(ID, FND_SCN),
INSERTALLRECORDS,
colmap(
USEDEFAULTS,
FND_SCN=@GETENV("TRANSACTION" , "CSN"),
FND_ROW_RANK=@TOKEN("TK_ROW_RANK"),
FND_COMMIT_DATE=@GETENV("GGHEADER" , "COMMITTIMESTAMP"),
FND_DML_TYPE=@GETENV("GGHEADER" , "OPTYPE")
);
Map SOURCE and
TARGET
KEYCOLS defines
matching criteria
USEDEFAULTS defines
column for column mapping Built-in attributes from the
“token” and the “header”
31. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
GoldenGate for AgileAnalytics: Loading the Foundation Layer
36
map SUGARCRM.ACCOUNTS,
target SUGARFND.ACCOUNTS,
KEYCOLS(ID, FND_SCN),
INSERTALLRECORDS,
colmap(
USEDEFAULTS,
FND_SCN=@GETENV("TRANSACTION" , "CSN"),
FND_ROW_RANK=@TOKEN("TK_ROW_RANK"),
FND_COMMIT_DATE=@GETENV("GGHEADER" , "COMMITTIMESTAMP"),
FND_DML_TYPE=@GETENV("GGHEADER" , "OPTYPE")
);
Map SOURCE and
TARGET
KEYCOLS defines
matching criteria
USEDEFAULTS defines
column for column mapping Built-in attributes from the
“token” and the “header”
INSERTALLRECORDS makes
this our Foundation Layer
33. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
GoldenGate INSERTALLRECORDS
38
NAME TICKER_SYMBO
L
OWNERSHIP FND_SCN FND_COMMIT_DATE FND_DML_TYP
ERed Pill Analytics 2992758 08/20/2012 12:00 AM INSERT
Red Pill Analytics, LLC Private 2992760 03/13/2013 12:00 AM UPDATE
NAME TICKER_SYMBOL OWNERSHIP
Red Pill Analytics, LLC Private
34. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
GoldenGate INSERTALLRECORDS
39
NAME TICKER_SYMBO
L
OWNERSHIP FND_SCN FND_COMMIT_DATE FND_DML_TYP
ERed Pill Analytics 2992758 08/20/2012 12:00 AM INSERT
Red Pill Analytics, LLC Private 2992760 03/13/2013 12:00 AM UPDATE
Red Pill Analytics, Inc. RPAI Public 2992762 05/02/2013 12:00 AM UPDATE
NAME TICKER_SYMBOL OWNERSHIP
Red Pill Analytics, Inc. RPAI Public
35. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
GoldenGate INSERTALLRECORDS
40
NVL (!
LEAD ( fnd_commit_date ) over !
!! ( PARTITION BY id !
!! ORDER BY fnd_commit_date ),!
to_date( '12/31/9999','mm/dd/yyyy' )!
)
NAME FND_SCN FND_COMMIT_DATE FND_EXPIRE_DATE FND_DML_TYPE FND_CURREN
TRed Pill Analytics 2992758 08/20/2012 12:00 AM 03/13/2013 12:00 AM INSERT N
Red Pill Analytics, LLC 2992760 03/13/2013 12:00 AM 05/02/2013 12:00 AM UPDATE N
Red Pill Analytics, Inc. 2992762 05/02/2013 12:00 AM 12/31/1999 12:00 AM UPDATE Y
36. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
GoldenGate INSERTALLRECORDS
41
CASE!
WHEN ( RANK() over (PARTITION BY id !
! ! ORDER BY fnd_commit_date desc )) = 1!
THEN 'Y'!
ELSE 'N'!
END
NAME FND_SCN FND_COMMIT_DATE FND_EXPIRE_DATE FND_DML_TYPE FND_CURREN
TRed Pill Analytics 2992758 08/20/2012 12:00 AM 03/13/2013 12:00 AM INSERT N
Red Pill Analytics, LLC 2992760 03/13/2013 12:00 AM 05/02/2013 12:00 AM UPDATE N
Red Pill Analytics, Inc. 2992762 05/02/2013 12:00 AM 12/31/1999 12:00 AM UPDATE Y
37. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Model-Driven Development
• Start with thin requirements
• Sample reports
• SQL whenever possible
• Annotated discrepancies
• Development against the
Foundation Layer
• Top-down approach
• Build empty BMM containing
logical “facts”, “dimensions”
and “hierarchies”
• Map physical sources to the
BMM
• Develop permanent, high-
impact dashboards
• No work in this iteration is
throw-away
43
38. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Model-Driven Development
44
• Start with thin requirements
• Sample reports
• SQL whenever possible
• Annotated discrepancies
• Development against the
Foundation Layer
• Top-down approach
• Build empty BMM containing
logical “facts”, “dimensions”
and “hierarchies”
• Map physical sources to the
BMM
• Develop permanent, high-
impact dashboards
• No work in this iteration is
throw-away
43. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics 49
Logical Table Sources
(LTS) provide the
functionality for mapping
transactional schemas as
dimensional models.
!
Siebel initially acquired
nQuire and the
technology that would
become OBIEE to report
against Siebel CRM.
46. Dragging and dropping tables from
the Physical Layer is the only way to
create logical tables in the Business
Model
65. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
ETL Development
•We us GoldenGate to
populate our Staging
layer
• Standard batch
• Micro batch
•Use Oracle Data
Integrator (ODI) to
transform our data to
meet business
requirements
•Need to decide what
flavor of GoldenGate to
use
• Extend our current
implementation
• Use ODI journalization
framework
72
69. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Loading Access and Performance: AgileAnalytics Approach
74
ODI from Staging for
transactional table loading
GoldenGate handles
Staging and Foundation
layers
Staging Layer
Foundation Layer
Source System Access & Performance
ODI from
Foundation
for complex
calculations
and
integration
70. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Oracle Data Integrator (ODI)
•Data Integration tool within Oracle
Fusion Middleware 12c
•Robust sources and targets
• Relational databases
• Essbase
• Flat file and XML
• Web services
• Big Data
•Declarative design
• Logical implementation
• Physical implementation
•ETL or E-LT
• Database as ETL engine
• Java Agents as ETL engine
75
82. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
ETL Development: Collateral from Model-Driven Development
84
1
2
3
WITH
SAWITH0 AS (select count(T2196.ID) as c1,
sum(T2196.AMOUNT) as c2,
concat(concat(T3853.FIRST_NAME, ' '), T3853.LAST_NAME) as c3,
T2196.LEAD_SOURCE as c4,
T2196.SALES_STAGE as c5
from
SUGARFND.ACCOUNTS T2041 /* Accounts (Standard) */ ,
SUGARFND.USERS T3853 /* Users (Account Owner Role) */ ,
SUGARFND.OPPORTUNITIES T2196 /* Opportunties (Standard) */ ,
SUGARFND.ACCOUNTS_OPPORTUNITIES T2221 /* Accounts Opportunities (Standard) */
where ( T2041.ASSIGNED_USER_ID = T3853.ID and T2041.ID = T2221.ACCOUNT_ID and T2196.ID = T2221.OPPORTUNITY_ID )
group by T2196.LEAD_SOURCE, T2196.SALES_STAGE, concat(concat(T3853.FIRST_NAME, ' '), T3853.LAST_NAME)),
SAWITH1 AS (select distinct 0 as c1,
D1.c3 as c2,
D1.c4 as c3,
D1.c5 as c4,
D1.c2 as c5,
D1.c1 as c6
from
SAWITH0 D1)
select D1.c1 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c5 as c5, D1.c6 as c6, D1.c7 as c7, D1.c8 as c8 from (
select D1.c1 as c1,
D1.c2 as c2,
D1.c3 as c3,
D1.c4 as c4,
D1.c5 as c5,
D1.c6 as c6,
sum(D1.c5) over (partition by D1.c3) as c7,
sum(D1.c6) over (partition by D1.c3) as c8
from
SAWITH1 D1
order by c1, c7 desc, c3 ) D1 where rownum <= 65001 4
91. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Build Now, Refactor Later
93
Front-end
developers
aggressively
close stories any
way they can
Complex
calculations,
direct SQL, BI
Publisher,
Custom
JavaScript:
Nothing is off
limits
A new “Refactor”
Story is opened
for the Metadata
Developer to
consider this
content for
inclusion in the
RPD
92. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Build Now, Refactor Later
94
Metadata developer
decides if the complex
front-end content is
viable
If viable, the
developer aggressively
solves the Refactor
Story using complex
calculations or opaque
views
The original Refactor
Story is reassigned to
the front-end
developer to have the
analysis point instead
to the new metadata
A new Refactor Story
is opened for the ETL
developer for any
complex
transformations that
may belong in the ETL
layer
93. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Build Now, Refactor Later
95
ETL Developer
decides if the
transformations
are viable, and
if so, adds new
transformations
The Refactor
Story is then
reassigned to the
Metadata
developer to
rework the RPD
using new
database
attributes
103. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Realtime: Foundational Federation
107
WITH
SAWITH0 AS (select T8974.OPPORTUNITY_COUNT as c1,
T8974.AMOUNT as c2,
T8939.INDUSTRY as c3,
T8981.LEAD_SOURCE as c4,
T8981.NAME as c5,
T8981.OPPORTUNITY_ID as c6
from
EDW.OPPORTUNITY_DIM T8981 /* Opportunity Dim (Standard) */ ,
EDW.ACCOUNT_DIM T8939 /* Account Dim (Standard) */ ,
EDW.OPPORTUNITIES_FACT T8974 /* Opportunities Fact (Standard) */
where ( T8939.ACCOUNT_KEY = T8974.ACCOUNT_KEY and T8939.INDUSTRY = 'Technology' and
T8974.OPPORTUNITY_KEY = T8981.OPPORTUNITY_KEY and T8981.LEAD_SOURCE = 'Conference' and 'EDW' =
'Foundation' ) ),
SAWITH1 AS (select cast(1 as INTEGER ) as c1,
T2196.AMOUNT as c2,
T2041.INDUSTRY as c3,
T2196.ID as c4,
T2196.LEAD_SOURCE as c5,
T2196.NAME as c6
from
SUGARFND.ACCOUNTS T2041 /* Accounts (Standard) */ ,
SUGARFND.OPPORTUNITIES T2196 /* Opportunties (Standard) */ ,
SUGARFND.ACCOUNTS_OPPORTUNITIES T2221 /* Accounts Opportunities (Standard) */
where ( T2041.ID = T2221.ACCOUNT_ID and T2196.ID = T2221.OPPORTUNITY_ID and T2196.LEAD_SOURCE =
'Conference' and case T2041.INDUSTRY when ' ' then 'Not Applicable' else T2041.INDUSTRY end =
'Technology' and TO_DATE('2013-08-15 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T2196.DATE_ENTERED ) ),
!
(continued...)
104. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Realtime: Foundational Federation
108
WITH
SAWITH0 AS (select T8974.OPPORTUNITY_COUNT as c1,
T8974.AMOUNT as c2,
T8939.INDUSTRY as c3,
T8981.LEAD_SOURCE as c4,
T8981.NAME as c5,
T8981.OPPORTUNITY_ID as c6
from
EDW.OPPORTUNITY_DIM T8981 /* Opportunity Dim (Standard) */ ,
EDW.ACCOUNT_DIM T8939 /* Account Dim (Standard) */ ,
EDW.OPPORTUNITIES_FACT T8974 /* Opportunities Fact (Standard) */
where ( T8939.ACCOUNT_KEY = T8974.ACCOUNT_KEY and T8939.INDUSTRY = 'Technology' and
T8974.OPPORTUNITY_KEY = T8981.OPPORTUNITY_KEY and T8981.LEAD_SOURCE = 'Conference' and 'EDW' =
'Foundation' ) ),
SAWITH1 AS (select cast(1 as INTEGER ) as c1,
T2196.AMOUNT as c2,
T2041.INDUSTRY as c3,
T2196.ID as c4,
T2196.LEAD_SOURCE as c5,
T2196.NAME as c6
from
SUGARFND.ACCOUNTS T2041 /* Accounts (Standard) */ ,
SUGARFND.OPPORTUNITIES T2196 /* Opportunties (Standard) */ ,
SUGARFND.ACCOUNTS_OPPORTUNITIES T2221 /* Accounts Opportunities (Standard) */
where ( T2041.ID = T2221.ACCOUNT_ID and T2196.ID = T2221.OPPORTUNITY_ID and T2196.LEAD_SOURCE =
'Conference' and case T2041.INDUSTRY when ' ' then 'Not Applicable' else T2041.INDUSTRY end =
'Technology' and TO_DATE('2013-08-15 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T2196.DATE_ENTERED ) ),
!
(continued...)
105. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Realtime: Foundational Federation
108
WITH
SAWITH0 AS (select T8974.OPPORTUNITY_COUNT as c1,
T8974.AMOUNT as c2,
T8939.INDUSTRY as c3,
T8981.LEAD_SOURCE as c4,
T8981.NAME as c5,
T8981.OPPORTUNITY_ID as c6
from
EDW.OPPORTUNITY_DIM T8981 /* Opportunity Dim (Standard) */ ,
EDW.ACCOUNT_DIM T8939 /* Account Dim (Standard) */ ,
EDW.OPPORTUNITIES_FACT T8974 /* Opportunities Fact (Standard) */
where ( T8939.ACCOUNT_KEY = T8974.ACCOUNT_KEY and T8939.INDUSTRY = 'Technology' and
T8974.OPPORTUNITY_KEY = T8981.OPPORTUNITY_KEY and T8981.LEAD_SOURCE = 'Conference' and 'EDW' =
'Foundation' ) ),
SAWITH1 AS (select cast(1 as INTEGER ) as c1,
T2196.AMOUNT as c2,
T2041.INDUSTRY as c3,
T2196.ID as c4,
T2196.LEAD_SOURCE as c5,
T2196.NAME as c6
from
SUGARFND.ACCOUNTS T2041 /* Accounts (Standard) */ ,
SUGARFND.OPPORTUNITIES T2196 /* Opportunties (Standard) */ ,
SUGARFND.ACCOUNTS_OPPORTUNITIES T2221 /* Accounts Opportunities (Standard) */
where ( T2041.ID = T2221.ACCOUNT_ID and T2196.ID = T2221.OPPORTUNITY_ID and T2196.LEAD_SOURCE =
'Conference' and case T2041.INDUSTRY when ' ' then 'Not Applicable' else T2041.INDUSTRY end =
'Technology' and TO_DATE('2013-08-15 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T2196.DATE_ENTERED ) ),
!
(continued...)
SAWITH2 AS ((select D0.c3 as c3,
D0.c4 as c4,
D0.c5 as c5,
'EDW' as c6,
D0.c6 as c7,
D0.c1 as c8,
D0.c2 as c9
from
SAWITH0 D0
union all
select case D0.c3 when ' ' then 'Not Applicable' else D0.c3 end as c3,
D0.c5 as c4,
D0.c6 as c5,
'Foundation' as c6,
D0.c4 as c7,
D0.c1 as c8,
D0.c2 as c9
from
SAWITH1 D0)),
SAWITH3 AS (select sum(D3.c8) as c1,
sum(D3.c9) as c2,
D3.c3 as c3,
D3.c4 as c4,
D3.c5 as c5,
D3.c6 as c6,
D3.c7 as c7
from
SAWITH2 D3
!
(continued...)
106. www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Realtime: Foundational Federation
109
WITH
SAWITH0 AS (select T8974.OPPORTUNITY_COUNT as c1,
T8974.AMOUNT as c2,
T8939.INDUSTRY as c3,
T8981.LEAD_SOURCE as c4,
T8981.NAME as c5,
T8981.OPPORTUNITY_ID as c6
from
EDW.OPPORTUNITY_DIM T8981 /* Opportunity Dim (Standard) */ ,
EDW.ACCOUNT_DIM T8939 /* Account Dim (Standard) */ ,
EDW.OPPORTUNITIES_FACT T8974 /* Opportunities Fact (Standard) */
where ( T8939.ACCOUNT_KEY = T8974.ACCOUNT_KEY and T8939.INDUSTRY = 'Technology' and
T8974.OPPORTUNITY_KEY = T8981.OPPORTUNITY_KEY and T8981.LEAD_SOURCE = 'Conference' and 'EDW' =
'Foundation' ) ),
SAWITH1 AS (select cast(1 as INTEGER ) as c1,
T2196.AMOUNT as c2,
T2041.INDUSTRY as c3,
T2196.ID as c4,
T2196.LEAD_SOURCE as c5,
T2196.NAME as c6
from
SUGARFND.ACCOUNTS T2041 /* Accounts (Standard) */ ,
SUGARFND.OPPORTUNITIES T2196 /* Opportunties (Standard) */ ,
SUGARFND.ACCOUNTS_OPPORTUNITIES T2221 /* Accounts Opportunities (Standard) */
where ( T2041.ID = T2221.ACCOUNT_ID and T2196.ID = T2221.OPPORTUNITY_ID and T2196.LEAD_SOURCE =
'Conference' and case T2041.INDUSTRY when ' ' then 'Not Applicable' else T2041.INDUSTRY end =
'Technology' and TO_DATE('2013-08-15 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T2196.DATE_ENTERED ) ),
!
(continued...)
SAWITH2 AS ((select D0.c3 as c3,
D0.c4 as c4,
D0.c5 as c5,
'EDW' as c6,
D0.c6 as c7,
D0.c1 as c8,
D0.c2 as c9
from
SAWITH0 D0
union all
select case D0.c3 when ' ' then 'Not Applicable' else D0.c3 end as c3,
D0.c5 as c4,
D0.c6 as c5,
'Foundation' as c6,
D0.c4 as c7,
D0.c1 as c8,
D0.c2 as c9
from
SAWITH1 D0)),
SAWITH3 AS (select sum(D3.c8) as c1,
sum(D3.c9) as c2,
D3.c3 as c3,
D3.c4 as c4,
D3.c5 as c5,
D3.c6 as c6,
D3.c7 as c7
from
SAWITH2 D3
!
(continued...)
BI Server does a union
between the
Foundation and
Performance layers
Union is eliminated
when the
“fragmentation
column” is used as a
filter