Enterprise Data warehouse using Oracle and Ellucian products. GoldenGate real-time replication from 20 sources to one DW target

1. Summit 2012: Supporting Student Success…
Data Replication: The power of
filtering using GoldenGate
Presented by: Greg Turmel
Senior Database Administrator
Tennessee Board of Regents
1

2. Simple Overview: Classic
 Classic Capture
 Route GUI: Director
 Transform
 Delivery
CLI: Manager CLI: Manager
Ellucian EDW 8.3 / 8.4
Ellucian Banner 9
Source Target
trails trails
capture pump replicat
Arc/Redo
2

3. Simple Overview: Integrated
 Integrated Capture
GUI: Director
 Route
 Transform
 Delivery
CLI: Manager CLI: Manager
Ellucian EDW 8.3 / 8.4
Ellucian Banner 9
Source Extract 1: Direct Target
trails replicat
Log mining
or
Serverlet
Legend:
2: Pump GUI: Graphical User Interface
Arc/Redo CLI: Command Line Interface
trails TCP/IP: Network Protocols
Arc/Redo: Transaction logging
Extract pump Trails: Change Data Extraction files
Source: Banner
Target: Common Repository
3

4. Oracle Replication Whitepapers
Unidirectional Bidirectional
Peer – to – Peer
4
Oracle © 11g Fusion Middleware: Using Oracle © GoldenGate for Oracle © Database Whitepaper 1539665 03/2012

5. Oracle Replication Whitepapers
“Due to the nature of Oracle GoldenGate functionality
and operations, as an enterprise software (solution), it
involves all layers of the technology stack from source
to target database to networking”.
What does that mean?
GoldenGate is highly integrated with database software, OS
processes, and (even Applications like Ellucian Banner 8 – 9
Change Management.) [e.g.] Different Versions of Operating
Systems, Storage Systems, and the applications used
throughout the enterprise.
Prusinski, B., Steve Phillips, and Richard Chung, (2011), Expert Oracle GoldenGate, Apress: New York, N.Y.
5

6. Oracle Replication Webcast
Webcast: Oracle GoldenGate 11g Release 2 Launch Webcast (Sept 12 10AM PT)
6

7. Oracle Replication Webcast
Webcast: Oracle GoldenGate 11g Release 2 Launch Webcast (Sept 12 10AM PT)
7

8. Oracle Replication Webcast
Webcast: Oracle GoldenGate 11g Release 2 Launch Webcast (Sept 12 10AM PT)
8

9. Replication: Campus Requirements
 Create a service account: ogg (GoldenGate service account – UNIX dba group user)
 Create a database account: ggs (GoldenGate service account – DBA user)
 Create a firewall port (access point) for ssh (22), ogg (78**), and database (1521)
 Space: Separate tablespace for process synchronization (less than 100 meg)
 Space: Separate multi‐purpose file system space for the UNIX user
 /home/GoldenGate
 /home/GoldenGate/Installation_files
 /home/GoldenGate/Campus_extract_trails
 /home/GoldenGate/Campus_data_pump_exports (20+ gb)
 3000+ student system every 12 hours
 12000+ student system (expect 400 meg) every 12 hours (Committed)
 Note: purge policy allows fine tune planning including purge once shipped
9

10. Replication Filtering
 Defined list of Banner© ERP tables or even selected columns from those tables:
table ALUMNI.AABDUES; table ALUMNI.AABDUES;
table ALUMNI.AABMINT; table ALUMNI.AABMINT;
table ALUMNI.AABMSHP; table ALUMNI.AABMSHP;
List of tables captured: Defined / limited by the Ellucian ODS/EDW product
 Mapping column values across the two systems using key words and variables:
MAP ALUMNI.AABDUES, TARGET ALUMNI.AABDUES, INSERTALLRECORDS,
colmap (usedefaults,
AABDUES_SURROGATE_ID = @GETENV ("GGHEADER", "BEFOREAFTEARINDICATOR"),
AABDUES_VERSION = @GETENV ("GGHEADER", "OPTYPE"),
AABDUES_USER_ID = @GETENV ("GGHEADER", "OPTYPE"),
AABDUES_VPDI_CODE = @STRCAT (@GETENV ("RECORD", "FILESEQNO"), @GETENV
("RECORD", "FILERBA")));
10

11. Replication Filtering
 Mapping column values across the two systems:
MAP SATURN.TWGRWMRL, TARGET SATURN.TWGRWMRL,
INSERTALLRECORDS,
colmap (usedefaults,
TWGRWMRL_SURROGATE_ID = "",
TWGRWMRL_VERSION = "",
TWGRWMRL_USER_ID = "",
TWGRWMRL_VPDI_CODE = "008863"); (FICE)
11

12. Replication Filtering
 Add Oracle Database logging on the tables sending changes to redo/archive logs:
add trandata SATURN.STVVETC
add trandata SATURN.STVVOED
add trandata SATURN.STVVTAB
 Structural changes to the tables (DDL or Data Definition Language) can also be added
but requires source database modification to capture and store in redo.
 Extraction process: Encrypted
 Pump process: decrypt / encrypt / compress / transports
Table ALUMNI.*;
 Replicat: Retrieve / decrypt / load / archive
Table ALUMNI.*;
12

13. Replication Filtering: Instantiation
 Replicat table: data direct on SATURN into ODS 8.3 staging using eload
SATURN.SGBSTDN – success
 Replicat table: data direct on SATURN into ODS 8.3 staging using SCN and
datapump
SATURN.SGBSTDN – success
 Replicat (Processing over on the TARGET: destination database server)
HANDLECOLLISIONS
sourcedefs ./dirdef/*_tables.defs
DISCARDFILE /u04/*_trails/rws.dsc, purge
DBOPTIONS DEFERREFCONST
13

14. Replication Filtering
Priority
1.
General
Fimsmgr
Taismgr
Payroll
Posnctl
Saturn
Faismgr
Alumni
14

15. Replication Filtering
 Validating a replicat run using Oracle GoldenGate CLI:
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
REPLICAT RUNNING RWS 00:00:00 00:00:09
REPLICAT RWS Last Started 2012‐05‐02 15:37 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:04 ago)
Log Read Checkpoint File /u04/ws_trails/ws000000
2012‐05‐02 15:48:15.995504 RBA 20707111
23> !
REPLICAT RWS Last Started 2012‐05‐02 15:37 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:06 ago)
Log Read Checkpoint File /u04/ws_trails/ws000000
2012‐05‐02 15:49:32.993561 RBA 20712107
15

16. Replication Filtering
 Results of 200 hour network outage
 Scenario: Hosting site network outage – Juniper router issues
 School “A” extract continues collecting (Banner source)
 School “B” extract continues collecting (Banner source)
 Target “Z” replicat “A” abend after 60 minutes of auto‐reconnection
 Target “Z” replicat “B” abend after 60 minutes…
 Network comes back online
 Target “Z” replicat “A” restarted – load 200 hours of commit in 1.5 minutes
 Target “Z” replicat “B” restarted – ditto
 Total time to re‐sync both ODS pre‐stage tables after network outage was restored:
 In 2 minutes (resync of 4400 tables established)
 Down line application tables loading in real‐time.
16

17. Replication Security Design
1. Extract (Source)
– uses protected UNIX service account
– select table data using sql account with encrypted password
– write to file encrypted and SAN/NAS/ASM can be a protected source file system
– file compressed in flight (writing file out to disk)
2. Pump (Source)
– decrypts extract / encrypts extract file
– uses secure file transfer protocol (SFTP / SSH)
3. Replicat (Target)
– uses protected UNIX service account
– uses secure file transfer protocol (SFTP / SSH)
– decompress / decrypts extract file and writes to protected target file system
– uploads to database using encrypted password
– uses SCN (change number for instantiation and synchronization between databases)
4. ARGOS (Reporting)
– Campus report server integrated with MS Active Directory
– Campus report server ADO string secured on MAPS server with password
– Access accounts to CR Project tables will be built by module/schema/table
– Oracle Virtual Private Database Identifiers for data retrieval/reporting
17

18. Replication Security Questions
Answering Right
Questions Alignment
Global Impact
(Tennessee)
Finding Solutions
Building a structure
Data Elements Defined
Help putting the
pieces together
Addressed all concerns
and Security Issues
Educating customers,
Management, stakeholders
Gold Star Effort
18

19. Replication Design Considerations
 Data acquisition
 Data separation – keeping the loads separated, processes separated
 Isolated reporting – Quality assurance processes and campus oversight
 Business Process Management – Defines the rules
19

20. Replication Project Planning
20

21. Data Replication Destination
21

22. Replication Benefits and Impact
 Automation ( with tunable parameters)
 Delivers low‐impact across heterogeneous systems
 Moves committed transactions with minimal overhead
 Reduce impact on OLTP system for reporting
 Disaster recovery – data replication offers unique opportunities to refine plans
 Data distribution, data synchronization, and high availability
 Manage dissimilar Oracle or other database versions
 Replicate and filter Oracle DDL operations between heterogeneous databases
22

23. Replication Benefits and Impact
 Common reporting system is achievable very quickly
 Campus #1 informed us that 400 Banner and 80 ODS reports are used
 Campus #2 said they use 100 Banner and 1 ODS report
 CR Project sub‐system has Banner ODS stage tables used for kpi reports
 Reduces reliance on separate data extracts taking months to near real time
 Dedicate a disk to this directory: dirtmp
 Extract, Pump, Replicat, and Manager processes must operate as an operating
system user that has privileges to read, write, and delete files and
subdirectories in the Oracle GoldenGate directory
23

24. Replication Benefits and Impact
 5,000 concurrent Extract and Replicat processes per instance of Oracle GoldenGate.
 Each Extract and Replicat process needs approximately 25‐55 MB of memory.
 Physical memory used by any Oracle GoldenGate process is controlled by the operating
system, not the Oracle GoldenGate program
 Classic capture mode, the Extract process reads the redo logs directly
 50‐150 MB for installation and 40‐100 MB for the working directories and binaries
 Binaries on a shared file system available to all cluster nodes
http://docs.oracle.com/cd/E28323_01/doc.1121/e27278.pdf
24

25. Replication Benefits and Impact
 Extract should not be stopped during a failure
 Transaction data might be missed if the transaction logs recycle
 Or if removed from the system before the data is completely captured
 There must be enough disk space to hold the data or Extract will abend
 Trail file clean up is configured and set according to the purge rules
 Set with the PURGEOLDEXTRACTS parameter (Source or Target)
 You will need to resynchronize target data if the outage outlasts disk
capacity. (e.g. – instantiation using datapump or ogg eload)
25

26. Replication Benefits and Impact
CR Project ‐ Data Transfer Feeds:
FICE Begin End Begin End
Order Source VPDI Extract Extract Replicat Replicat
1 APSU 003478 1 8 1 8 8
2 ChSCC 003998 11 18 11 18 8
3 ClSCC 003999 21 28 21 28 8
4 CoSCC 003483 31 38 31 38 8
5 DSCC 006835 41 48 41 48 8
6 ETSU 003487 51 58 51 58 8
7 JSCC 004937 61 68 61 68 8
8 MSCC 006836 71 78 71 78 8
9 MTSU 003510 81 88 81 88 8
10 NeSCC 005378 91 98 91 98 8
11 NSCC 008145 101 108 101 108 8
12 PSCC 012693 111 118 111 118 8
13 RSCC 009914 121 128 121 128 8
14 STCC 010439 131 138 131 138 8
15 TBR 000001 141 148 141 148 8
16 TSU 003522 151 158 151 158 8
17 TTU 003523 161 168 161 168 8
18 UoM 003509 171 178 171 178 8
19 VSCC 009912 181 188 181 188 8
20 WSCC 008863 191 198 191 198 8
160
26

27. Replication Benefits and Impact
 Support multi‐byte character data
 The source and target database schema definition must be logically identical
 The character sets between the two databases must be one of the following:
o Identical / Equivalent
o Target is superset of the source: [e.g.] UNICODE is a superset
o Multi‐byte data is supported when length semantics are in bytes or characters.
 Does not support negative dates: Supports the capture and replication of TIMESTAMP
with TIME ZONE as a UTC offset (TIMESTAMP '2011‐01‐01 8:00:00 ‐8:00')
 Binary or unprintable characters are not supported
 Ignores any virtual column that is part of a unique key or index
27

28. Replication Benefits and Impact
Identifying Customers and their Requirements
 Chancellor Morgan for THEC / Regents / Decision Support
 Campus Presidents for System measurements
 Vice Chancellor Wendy Thompson
for Access and Diversity Research
for Completion Delivery Unit (CDU)
 Vice Chancellor Nichols
for Complete College Tennessee Act (CCTA)
 Interim Vice Chancellor Clark
for Institutional Research
for Tennessee Higher Education requests (THEC)
for Legislative requests (Ad‐Hoc information)
 Campus Operations as necessary (gap analysis)
28

29. Replication Limitations
 Supports the capture of direct‐load INSERT(s)
 Supplemental logging must be enabled
 Database must be in archive log mode
 Does not capture from a view
 Supports capture from the underlying tables of a view
 Can replicate to a view as long as the view is inherently updatable
 Materialized views created WITH ROWID are not supported
 Truncates on materialized views are not supported.
29

30. Replication Challenges
 Encrypted tables are not supported in classic capture
 Supports the replication of sequence values in a uni‐directional.
 Merging / Converging data: from sub‐system to Common Repository
 Data Integrity: One way data migration (No Campus edits at target system)
 Conflict management: Managing logging and error queue: Resolution
 Avoiding a single point of failure – breaks in data stream/time stamps/outages
 Change at the Banner Source impacts feeds when columns / structures change
 Change includes Banner source DDL and ODS target baseline patching
 Campus “A” patching different than Campus “B” – BPM priorities – impact/gap
30

31. GoldenGate Replication Challenges
 Campus versioning impacts coding requirements on the CR Project server
 Physical requirements: Allocated space at campus (direct cost)
 [e.g.] /oback runs out of shared space and stopping replication
 OGG Processes abend when they can’t write out the data captured
 Indexing used for loading integrity vs. indexing used for reporting integrity
31

32. GoldenGate Replication Strategies
Strategies:
 Secure access to Banner ERP data adding immediate value to Board of Regents,
Presidents, Chancellor and Campus reporting
 product issues with OGG©, Oracle STREAMS©, and Ellucian© ODS/EDW
 Create and coordinate Banner© patch / apply (change management) processes
 Evaluate break/fix for minimizing manual intervention (Automation)
 Rolling out stable, integrated, mature products
 Coordinating Source / Target Hardware Shutdown / Database Quiesced
32

33. Summit 2012: Supporting Student Success…
Data Replication: Objective Statement
The Common Data Repository Project: Supporting the…
 Access and Diversity (Investigative Reporting)
 Completion Delivery Unit (CDU Project)
 Complete College of Tennessee Act (2010) for a common student experience
 Executive Reporting needs for system wide decision support (Governance)
 Legislative Reporting – Requests for Information (Ad‐Hoc queries)
 Race to the top – Collaboration events in measuring student success
 Tennessee Higher Education Commission reporting
 Workforce development efforts at the Board of Regents
(Referred to as CDR or CR Project) It is scoped to provide a real time reporting system for
the Regents, Campus Presidents, Chancellor Morgan and his Executive Staff.
TBR supports over 200,000 students. The goal is to have a common system aimed at
providing the best possible service using modern technologies in a more efficient approach
to information system resources and reporting.
33

34. GoldenGate Replication Questions
Questions?
34

35. Contact Information
Greg Turmel
Sr. Database Administrator
Office of Information Technology
Tennessee Board of Regents
1415 Murfreesboro Rd. #358
Nashville, TN. 37217
615.366.4467 (Office)
http://itinfo.tbr.edu (IT website)
http://twitter.com/datahaulr
http://www.slideshare.net/gturmel
http://www.linkedin.com/in/gturmel
35