Replacing Oracle CDC with Oracle GoldenGate

www.RedPillAnalytics.com info@RedPillAnalytics.com @RedPillA © 2014 RED PILL Analytics
Is Oracle GoldenGate a
Replacement for Oracle Change
Data Capture?
1
Stewart Bryson
medium.com/@stewartbryson @stewartbryson linkedin.com/in/stewartbryson

© 2014 RED PILL Analytics
Who is Red Pill Analytics?

Red Pill Analytics: Leadership
3
Stewart Bryson
Oracle ACE Director
Previously From:
Rittman Mead
Informix Software
Data Warehousing &
Business Intelligence
since 1996
Kevin McGinley
Oracle ACE Director
Previously From:
Accenture
BI Consulting Group
Data Warehousing &
Business Intelligence
since 1997

About Red Pill Analytics
Other Services
• Expert Retainer
• Support
• Training
4
• DevOps for BI and
DI
• Hosted or On-
premise
• Used in Capacity
Analytics
Capacity Analytics
• Development-as-a-
service
• Agile methodology
• Faster/cheaper
than traditional
consulting
• Subscription billing

“Change” Has Different Meanings

ODI Journalization
Framework
A B
Oracle GoldenGate
(Traditionally)
A B
Oracle CDC
(Deprecated)
A B

A B
ODI Journalization
Framework
A B

Oracle CDC
(Deprecated)
A B
A

A
ODI Journalization
Framework
A B

Is the A-Team Right About This?
13

Online Retailing Example

Publishing

Oracle CDC: Publish and Subscribe
17
EDW
R Sandbox
Partner
Feed

Oracle CDC: Publishing
18
-- prepare source table
alter table sugarcrm.accounts ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
BEGIN
DBMS_CAPTURE_ADM.PREPARE_TABLE_INSTANTIATION(TABLE_NAME => 'sugarcrm.accounts');
END;
/
-- create change set
BEGIN
DBMS_CDC_PUBLISH.CREATE_CHANGE_SET
(
change_set_name => 'CRM_ACCOUNTS',
description => 'Change set for the CRM Accounts Data',
change_source_name => 'HOTLOG_SOURCE',
stop_on_ddl => 'y',
begin_date => sysdate
);
END;

19
BEGIN
END;
/
BEGIN
(
stop_on_ddl => 'y',
);
END;
-- create change table
BEGIN
DBMS_CDC_PUBLISH.CREATE_CHANGE_TABLE
( owner => 'cdcpub',
change_table_name => 'accounts_ct',
source_schema => 'SUGARCRM',
source_table => 'ACCOUNTS',
column_type_list => 'ID CHAR(36), NAME VARCHAR2(150), DATE_ENTERED DATE,DATE_MODIFIED DATE,
ACCOUNT_TYPE VARCHAR2(50), INDUSTRY VARCHAR2(50), PARENT_ID CHAR(36),
ANNUAL_REVENUE VARCHAR2(100), BILLING_ADDRESS_STREET VARCHAR2(150),
BILLING_ADDRESS_CITY VARCHAR2(100), BILLING_ADDRESS_STATE VARCHAR2(100),
BILLING_ADDRESS_POSTALCODE VARCHAR2(20), OWNERSHIP VARCHAR2(100),
BILLING_ADDRESS_COUNTRY VARCHAR2(255), RATING VARCHAR2(100),
WEBSITE VARCHAR2(255),TICKER_SYMBOL VARCHAR2(10), SIC_CODE VARCHAR2(10),
CAMPAIGN_ID CHAR(36)',
capture_values => 'new',
rs_id => 'y',
row_id => 'n',
user_id => 'n',
timestamp => 'n',
object_id => 'n',
source_colmap => 'n',
target_colmap => 'y',
options_string => 'TABLESPACE UTILS');
END;

-- create change table
BEGIN
DBMS_CDC_PUBLISH.CREATE_CHANGE_TABLE
( owner => 'cdcpub',
change_table_name => 'accounts_ct',
source_schema => 'SUGARCRM',
source_table => 'ACCOUNTS',
column_type_list => 'ID CHAR(36), NAME VARCHAR2(150), DATE_ENTERED DATE,DATE_MODIFIED DATE,
ACCOUNT_TYPE VARCHAR2(50), INDUSTRY VARCHAR2(50), PARENT_ID CHAR(36),
ANNUAL_REVENUE VARCHAR2(100), BILLING_ADDRESS_STREET VARCHAR2(150),
BILLING_ADDRESS_CITY VARCHAR2(100), BILLING_ADDRESS_STATE VARCHAR2(100),
BILLING_ADDRESS_POSTALCODE VARCHAR2(20), OWNERSHIP VARCHAR2(100),
BILLING_ADDRESS_COUNTRY VARCHAR2(255), RATING VARCHAR2(100),
WEBSITE VARCHAR2(255),TICKER_SYMBOL VARCHAR2(10), SIC_CODE VARCHAR2(10),
CAMPAIGN_ID CHAR(36)',
capture_values => 'new',
rs_id => 'y',
row_id => 'n',
user_id => 'n',
timestamp => 'n',
object_id => 'n',
source_colmap => 'n',
target_colmap => 'y',
options_string => 'TABLESPACE UTILS');
END;
20
BEGIN
END;
/
BEGIN
(
stop_on_ddl => 'y',
);
END;
-- enable change set
BEGIN
DBMS_CDC_PUBLISH.ALTER_CHANGE_SET(
enable_capture => 'y');
END;

www.redpillanalytics.com info@redpillanalytics.com @RedPillA © 2014 RED PILL Analytics
Oracle CDC Subscription View
22
NAME TICKER_SYMBOL OWNERSHIP CSCN$ COMMIT_TIMESTAMP
$
OPERATION$
Red Pill Analytics 2992758 06/01/2014 12:00 AM INSERT
NAME TICKER_SYMBOL OWNERSHIP
Red Pill Analytics, LLC Private

23
$
OPERATION$
Red Pill Analytics, LLC Private 2992760 07/14/2014 12:00 AM UPDATE
Red Pill Analytics, LLC Private

24
$
OPERATION$
Red Pill Analytics, LLC Private 2992760 07/14/2014 12:00 AM UPDATE
Red Pill Analytics, Inc. RPAI Public 2992762 01/20/2015 12:00 AM UPDATE
Red Pill Analytics, Inc. RPAI Public

25
NVL (
LEAD ( fnd_commit_date ) over
( PARTITION BY id
ORDER BY fnd_commit_date ),
to_date( '12/31/9999','mm/dd/yyyy' )
)
NAME CSCN$ COMMIT_TIMESTAMP$ EXPIRE_TIMESTAMP$ OPERATION$ CURRENT_IND
Red Pill Analytics 2992758 06/01/2014 12:00 AM 07/14/2014 12:00 AM INSERT N
Red Pill Analytics, LLC 2992760 07/14/2014 12:00 AM 01/20/2015 12:00 AM UPDATE N
Red Pill Analytics, Inc. 2992762 01/20/2015 12:00 AM 12/31/9999 12:00 AM UPDATE Y

26
NAME CSCN$ COMMIT_TIMESTAMP$ EXPIRE_TIMESTAMP$ OPERATION$ CURRENT_IND
Red Pill Analytics 2992758 06/01/2014 12:00 AM 07/14/2014 12:00 AM INSERT N
Red Pill Analytics, LLC 2992760 07/14/2014 12:00 AM 01/20/2015 12:00 AM UPDATE N
Red Pill Analytics, Inc. 2992762 01/20/2015 12:00 AM 12/31/9999 12:00 AM UPDATE Y
CASE
WHEN ( RANK() over (PARTITION BY id
ORDER BY fnd_commit_date desc )) = 1
THEN 'Y'
ELSE 'N'
END

DEMO:
Viewing Change Data with GoldenGate

Am I in Love With a Deprecated Product?

What’s the Problem with Oracle CDC?
29
11g

11g
30

31
11g
12c
11g

32
11g
12c
11g

33
11g
12c
11g

Pragmatism

Text Here

Werner Heisenberg

Don’t Use Particles to Measure Particles

Don’t Use Streams to Capture Change

CDC Uses Streams (Capture and Queue)
41
Oracle CDC
(Deprecated)
A B

CDC Uses Streams (Capture and Queue)
42
Oracle CDC
(Deprecated)
A B
Oracle Streams
(Deprecated)
A B

Oracle GoldenGate Architecture
43

Installing GoldenGate
44

45

46

Configuring GoldenGate EXTRACT
47
EXTRACT EXT
USERID oggcore_1, PASSWORD ogg
SETENV (ORACLE_HOME="/app/oracle/product/11.2.0/dbhome_1")
SETENV (ORACLE_SID="orcl")
WARNLONGTRANS 1h, CHECKINTERVAL 30m
EXTTRAIL ./dirdat/lt
WILDCARDRESOLVE IMMEDIATE
TABLE SUGARCRM.ACCOUNTS,
TOKENS
(
FND_SCN = @GETENV('TRANSACTION','CSN'),
FND_ROW_RANK = @GETENV('RECORD','RSN'),
FND_COMMIT_DATE=@GETENV('GGHEADER','COMMITTIMESTAMP'),
FND_DML_TYPE=@GETENV('GGHEADER','OPTYPE')
);
System change number
(SCN) for the transaction
USEDEFAULTS defines
column for column mapping
Transaction Commit Date
(with Time)
Operation Type: Delete, Update,
Insert, PK Update, and others

Configuring GoldenGate EXTRACT
48
EXTRACT EXT
WARNLONGTRANS 1h, CHECKINTERVAL 30m
EXTTRAIL ./dirdat/lt
WILDCARDRESOLVE IMMEDIATE
TABLE SUGARCRM.ACCOUNTS,
TOKENS
(
FND_SCN = @GETENV('TRANSACTION','CSN'),
FND_ROW_RANK = @GETENV('RECORD','RSN'),
FND_COMMIT_DATE=@GETENV('GGHEADER','COMMITTIMESTAMP'),
FND_DML_TYPE=@GETENV('GGHEADER','OPTYPE')
);
GGSCI (oracle.localdomain) 1> add extract ext, integrated tranlog, begin now
EXTRACT added.
GGSCI (oracle.localdomain) 2> add exttrail ./dirdat/lt, extract EXT, megabytes 200
EXTTRAIL added.
GGSCI (oracle.localdomain) 3> dblogin userid oggcore_1, password ogg
Successfully logged into database.
GGSCI (oracle.localdomain as oggcore_1@orcl) 4> register extract ext database
Extract EXT successfully registered with database at SCN 2239208.
GGSCI (oracle.localdomain as oggcore_1@orcl) 5> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
EXTRACT RUNNING EXT 00:00:10 00:00:04

Configuring GoldenGate DATA PUMP
49
EXTRACT PMP
PASSTHRU
RMTHOST oracle.localdomain MGRPORT 7810, COMPRESS
RMTTRAIL ./dirdat/rt
TABLE SUGARCRM.*;

Configuring GoldenGate DATA PUMP
50
EXTRACT PMP
PASSTHRU
RMTHOST linux-db MGRPORT 15000, COMPRESS
RMTTRAIL ./dirdat/rt
TABLE SUGARCRM.*;
GGSCI (oracle.localdomain) 1> add extract PMP, exttrailsource ./dirdat/lt
EXTRACT added.
GGSCI (oracle.localdomain) 2> add rmttrail ./dirdat/rt, extract PMP, megabytes 200
RMTTRAIL added.
GGSCI (oracle.localdomain) 3> start extract PMP
Sending START request to MANAGER ...
EXTRACT PMP starting
GGSCI (oracle.localdomain) 4> info all
MANAGER RUNNING
EXTRACT RUNNING EXT 00:00:02 00:00:07
EXTRACT RUNNING PMP 00:00:00 00:04:15

Configuring GoldenGate REPLICAT
51
REPLICAT REP
DISCARDFILE ./dirrpt/rep.dsc, append, megabytes 500
SOURCEDEFS ./dirdef/sugarcrm.def
MAP SUGARCRM.ACCOUNTS,
TARGET SUGARSTG.ACCOUNTS,
KEYCOLS(ID, FND_SCN, FND_ROW_RANK),
INSERTALLRECORDS,
COLMAP
(
USEDEFAULTS,
FND_SCN=@token('FND_SCN'),
FND_ROW_RANK=@token('FND_ROW_RANK'),
FND_COMMIT_DATE=@token('FND_COMMIT_DATE'),
FND_DML_TYPE=@token('FND_DML_TYPE')
);
Map SOURCE and
TARGET
KEYCOLS defines
matching criteria
USEDEFAULTS defines
Our “Tokens” mapped to
columns

52
REPLICAT REP
SOURCEDEFS ./dirdef/sugarcrm.def
TARGET SUGARSTG.ACCOUNTS,
KEYCOLS(ID, FND_SCN, FND_ROW_RANK),
INSERTALLRECORDS,
COLMAP
(
USEDEFAULTS,
FND_SCN=@token('FND_SCN'),
FND_ROW_RANK=@token('FND_ROW_RANK'),
FND_COMMIT_DATE=@token('FND_COMMIT_DATE'),
FND_DML_TYPE=@token('FND_DML_TYPE')
);
Map SOURCE and
TARGET
KEYCOLS defines
matching criteria
USEDEFAULTS defines
Our “Tokens” mapped to
columns
INSERTALLRECORDS is the
secret sauce for CDC

53
REPLICAT REP
TARGET SUGARFND.ACCOUNTS,
KEYCOLS(ID, FND_SCN),
INSERTALLRECORDS,
COLMAP(
USEDEFAULTS,
FND_SCN=@GETENV("TRANSACTION" , "CSN"),
FND_ROW_RANK=@TOKEN("TK_ROW_RANK"),
FND_COMMIT_DATE=@GETENV("GGHEADER" , "COMMITTIMESTAMP"),
FND_DML_TYPE=@GETENV("GGHEADER" , "OPTYPE")
);
GGSCI (oracle.localdomain) 1> dblogin userid oggcore_2, password ogg
Successfully logged into database.
GGSCI (oracle.localdomain as oggcore_2@orcl) 2> add checkpointtable oggcore_2.checkpoint
Successfully created checkpoint table oggcore_2.checkpoint.
GGSCI (oracle.localdomain as oggcore_2@orcl) 3> add replicat REP integrated, exttrail ./dirdat/rt
REPLICAT (Integrated) added.
GGSCI (oracle.localdomain as oggcore_2@orcl) 4> start replicat REP
Sending START request to MANAGER ...
REPLICAT REP starting
GGSCI (oracle.localdomain as oggcore_2@orcl) 5> info all
MANAGER RUNNING
REPLICAT RUNNING REP 00:00:00 00:20:54

DEMO:
Change Data in GoldenGate

What Are We Missing?
Subscriptions

Publish and Subscribe
57
EDW
R Sandbox
Partner
Feed

create table ogg_subscription
(
sub_name varchar2(30) not null enable,
sub_type varchar2(6),
server_name varchar2(30),
effective_scn number,
expiration_scn number
);
Create the
Subscription
Purge the
Window
Extend the
Window
SCN Window
Subscription Name

create table ogg_subscription
(
sub_name varchar2(30) not null enable,
sub_type varchar2(6),
server_name varchar2(30),
effective_scn number,
expiration_scn number
);
Create the
Subscription
create or replace view ogg_accounts_view
as select id,
name,
ticker_symbol,
ownership,
fnd_scn,
fnd_dml_type,
fnd_commit_date,
NVL ( LEAD ( fnd_commit_date ) over
( PARTITION BY id
ORDER BY fnd_commit_date ),
to_date( '12/31/9999','mm/dd/yyyy' )) fnd_expire_dt,
CASE
WHEN ( RANK() over (PARTITION BY id
ORDER BY fnd_commit_date desc )) = 1
THEN 'Y'
ELSE 'N'
END current_ind
from sugarstg.accounts
join ogg_subscription
on sub_name='FND_ACCOUNTS'
and fnd_scn >=effective_scn
and fnd_scn < expiration_scn
order by fnd_commit_date;
SCN Window
Subscription Name
Purge the
Window
Extend the
Window

Create the
Subscription
MERGE INTO ogg_subscription t
USING ( SELECT to_number( applied_low_position ) complete_scn,
server_name
FROM all_gg_inbound_progress
) s
ON ( s.server_name = t.server_name )
WHEN MATCHED
THEN UPDATE
SET t.expiration_scn = s.complete_scn,
t.effective_scn = t.expiration_scn
WHEN NOT MATCHED
THEN INSERT
( t.effective_scn,
t.expiration_scn,
t.sub_name,
t.server_name )
values
( '0',
s.complete_scn,
'FND_ACCOUNTS',
s.server_name );
Slide our window over if
the subscription exists
Last SCN applied by
GoldenGate
Purge the
Window
Extend the
Window
Create our subscription if it
doesn’t already exist

Create the
Subscription
delete sugarstg.accounts
where fnd_scn < ( select min(expiration_scn)
from ogg_subscription );
Delete all the records that have
been read by all subscriptions
Purge the
Window
Extend the
Window

Is Oracle GoldenGate a
Replacement for Oracle Change
Data Capture?
62
Stewart Bryson
medium.com/@stewartbryson @stewartbryson linkedin.com/in/stewartbryson

Replacing Oracle CDC with Oracle GoldenGate

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Replacing Oracle CDC with Oracle GoldenGate

Ähnlich wie Replacing Oracle CDC with Oracle GoldenGate (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Replacing Oracle CDC with Oracle GoldenGate