Retail Analytics, with Oracle Data Integrator 11G. Points about ODI Objects, Interfaces, Variables, Packages, Scenarios, Load Plans, Scheduling. Batch Scheduling with RA 14.2, UAF in 14.2, Error Managment in RA 14.2

1. RI –ARCHITECTURE
ORACLE INDIA PVT.
LTD.
Puneet Kala

3. Key Concepts of ODI
 Execution Context
 Knowledge Modules
 Models
 Interfaces
 Packages
 Scenarios
 Procedures
 Variables

4. Variables
 Defined in Designer Tab of ODI
 Scope: Global, Project
 Operations
1. Refresh
2. Declare
3. Set
4. Evaluate

5. Interfaces
 What
1. The destination data store for the data (the target).
This will be chosen from an ODI model.
2. The data stores that supply the input data (the
sources). These too will be chosen from ODI
models.
 How
1. The transformations that are applied to the data
during the transition from sources to target (the
mappings). These are expressed in SQL.
2. The physical transfer mechanisms that are used
between sources and target (the flow). This role is
performed by the Knowledge Modules

6. Interfaces

7. Packages
 Packages are the basic element of
orchestration in ODI.
 Sequence your interfaces, and define what
operation to perform when a step (or interface)
fails
 Sequential Execution

8. Packages

9. Models
 Models in ODI are used to store the metadata
imported from the databases.
 Created using Designer Tab in ODI

10. Procedures
 A Procedure is a set of commands that can
be executed by an agent.
 Procedures should be considered only when
what you need to do can't be achieved in an
interface
 Procedures require you to develop all your
code manually, as opposed to interfaces

11. Scenario
 Scenario is the "compiled" version of an ODI
object.
 It is possible to generate scenarios for
packages, procedures, interfaces or variables

12. Load Plans
 Execution of scenarios in a hierarchy of
sequential and parallel steps
 Scenarios, Packages, interfaces, variables
and Procedures can be added to load plans.
 It is an executable object that contains steps
that needs to be executed.

13. Load Plans

14. Execution Context

15. Knowledge Modules

16. Knowledge Modules
LKM: MySQL To Oracle and File to Oracle
IKM: Oracle Incremental Update

17. Knowledge Modules
 Issue the join-and-filter query to extract the
MySQL data into the staging area in Oracle
 Extract the data from the flat file and upload
that into the staging area
 Combine the data in the staging area and
upload it from there into the target data
schema

18. Final Structure

19. Confused?

20. ODI general Flow

21. SDE SIL PLP

22. SDE SIL PLP
 SDE
Retail Analytics SDE programs are source dependent extraction programs that
extract data from source system, transform data, and load data to Retail
Analytics staging tables. SDE programs name have "sde" as the suffix.
 SIL
Retail Analytics SIL programs are source independent loading programs that
load data from Retail Analytics staging tables to Retail Analytics base level data
mart tables. SIL programs name have "sil" as the suffix.
 PLP
Retail Analytics PLP programs are post loading programs that load data from
Retail Analytics base level tables or Retail Analytics temporary tables, created
and populated during batch cycle, to Retail Analytics data mart tables. The PLP
programs include ETL maintenance and fact aggregations. PLP programs
name have "plp" as the suffix.

23. SDE SIL PLP

24. Batch Process and Running Jobs
 SDE, SIL & PLP each have a job associated with
them which are shell process
 User has two options, either execute from ODI or
execute the batch.
 When executing the Batch, please follow:
1. Verify in C_LOAD_DATES table for any existing
Entry.
2. Verify Parameter values on C_ODI_PARAMS
3. Execute the Batch.
 In case of Errors look into : $MMHOME/Errors/Log

25. Example SDE, SIL
 SDE: SDE_RetailColorDimension
1. Package: SDE_RetailColorDimension
2. Source Table: DIFF_IDS
3. Target Table: W_RTL_PRODUCT_COLOR_DS
4. Batch Name: prdclrsde.ksh
 SIL: SIL_RetailProductColorDimension
1. Package: SIL_RetailProductColorDimension
2. Source Table: W_RTL_PRODUCT_COLOR_DS
3. Target Table: W_RTL_PRODUCT_D
4. Batch Name: rtlprodcolorsil.ksh

26. Universal Adaptor Framework

27. Universal Adaptor Framework

28. UAF

29. UAF
 The goal of the Universal Adapter Framework (UAF) is
to simplify the process of moving source dependent
extracts into Retail Insights staging tables for
customers
 In terms of implementation, the UAF first requires pipe
(‘|’) separated value (DAT file) text file extracts to be
provided as inputs
 Once the DAT files are in place, the UAF can be used
to move that data into Retail Insights staging tables
through the use of sqlldr
 The control files required for sqlldr will be created
automatically during the process that is written in ODI.

30. Error Management

31. Error Management
 Three Main Category :-
1. Data Errors
2. Execution Errors
3. Operational Error

32. Data Error
 These occur when we encounter "bad" data
during data integration tasks
 Data type issues, ODI Constraints
 Keys, References, Conditions
 Detection of Data errors is done by setting
Flow_Control option in the Flow tab of an IKM
 Errors detected move to ‘Error Hospital’
 Only data check in RA! ‘Dimension Identifier
Missing’

33. Data Error

34. Error Table
 All columns, definitions of the corresponding table. For
example: Error occurred while Loading Customer data
in Customers Table, hence E$_CUSTOMER Table.
 Additional column about control, “S” Static or “F” Flow
 Error Message
 Check Date
 ODI Object Info
 Constraint Name
 Constraint Type ex. FK(Reference Failure)
 ODI Session

35. Correcting Error
 RECYCLE_ERROR Option in IKM to True
 Correcting Data in Error Tables and Executing
the Packages.

36. Execution Errors

37. Example of Execution Errors
 These occur when one or more steps in an
interface, procedure, package, or scenario fail
to complete successfully
 Database is down, Which can be handled and
anticipated error, hence managed through flow
control using KO and OK steps in a package
 Unexpected Error during the Execution of a
package, Log could be found in Operator
Navigator in ODI Studio example: TO_DATA
function on an XML source Data Server

38. Operational Errors
 These arise if one or more of the underlying
platform-level or infrastructure components
encounter some kind of exception.
 This area forms part of management and
monitoring, especially locating and viewing
system component (for example, ODI Agent)

39. Restart / Recovery
 What happens when a Job fails?
1. All successful interfaces in the package commit data
to the database, we can restart the package from
operational navigator tab. Default behavior to restart
from point of failure
2. When we restart a package again, KM takes care of
changes.
1. Trunk and Load
2. Incremental Update
3. KM written to create a File when successful
execution and verify the existence of that file before
starting again.

40. Load Plan Restart Options
 If Load Plan is created we have three options
for each step.
1. Restart from new session
2. Restart from failed step
3. Restart from Failed Task

42. THANKS