1. Inside Oracle Data Integrator 11g
Mark Rittman, Technical Director, Rittman Mead
OUG Ireland Conference 2012
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
2. Mark Rittman
• Mark Rittman, Co-Founder of Rittman Mead
• Oracle ACE Director, specialising in Oracle BI&DW
• 14 Years Experience with Oracle Technology
• Regular columnist for Oracle Magazine
• Author of forthcoming Oracle Press book on OBIEE 11g
• Writer for Rittman Mead Blog :
http://www.rittmanmead.com/blog
• Email : mark.rittman@rittmanmead.com
• Twitter : @markrittman
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
3. About Rittman Mead
• Oracle BI and DW platinum partner
• World leading specialist partner for technical excellence, solutions delivery and innovation in
Oracle BI
• Approximately 50 consultants worldwide
• All expert in Oracle BI and DW
• Offices in US (Atlanta), Europe, Australia and India
• Skills in broad range of supporting Oracle tools:
‣ OBIEE
‣ OBIA
‣ ODIEE
‣ Essbase, Oracle OLAP
‣ GoldenGate
‣ Exadata
‣ Endeca
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
4. Oracle Data Integrator 11g
• Data Integration tool within Oracle Fusion Middleware 11g
• Used for database, web service, file, XML etc integration
• Declarative design, separation of
business and technical integration
• Uses ELT approach to leverage
target platform capabilities
• Licensed together with OWB as
Oracle Data Integrator EE
• Strategic tool within Oracle for
data integration
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
5. Oracle Data Integrator Key Features
• Same philosophy as OWB and Oracle RDBMS – Use the Database as the ETL engine
‣ ODI supports heterogeneous databases, not just Oracle
• Built for SOA environments
‣ Support for Web Services, EII etc
• Supports batch, event-based and real-time integration
• Data Integrity Controls create a data “firewall”
‣ Reduces data prep time by not
processing erroneous data
• Extensible through “Knowledge Modules”
‣ Change Data Capture
‣ Slowly Changing Dimensions
‣ Bulk load
• Java client application
with server elements
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
6. Oracle Data Integrator 11g Key Components
• ODI Studio (the main development IDE)
‣ Designer Navigator
‣ Operator Navigator
‣ Topology Navigator
‣ Security Navigator
• ODI Agents (used for executing mappings and packages)
• ODI Console (web-based console for running published packages)
• ODI Repositories (contain definitions of models, interfaces and other objects)
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
7. ODI Studio Navigators
• Modules within a single IDE used for performing certain tasks
• Each module may be used by different category of ODI user (usage can be scoped)
Designer Operator Topology Security
Model data sources View running
Create data sources
Define projects interfaces and Manage users and
Define agents
Create interfaces packages priviledges
Manage infrastructure
Release scenarios Operate production
Repository
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
8. Designer Navigator
• Used for designing ODI data, transformation and mapping objects
• Import (reverse) metadata on data objects from source systems
• Define variables, sequences, functions
• Interfaces and packages
• Use knowledge modules to
move data from source to target
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
9. Topology Navigator
• Defines the physical and logical architecture for the ODI system
• Connections through to databases, OLAP servers, applications, files etc
• Logical aliases for these connections
• Contexts (environments)
• Standalone and JEE agents
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
10. Operator Navigator
• Monitor the execution of interfaces, static controls and packages
• Import and run compiled packages
• Drill-into the actual code executed at each KM step
• View errors, warnings and
execution metrics
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
11. Agents
• Java processes (applications) that are used for executing interfaces and packages
• ODI Studio comes with an embedded agent,
or you can install them standalone on Designer Operator
one or more servers Model data sources View running
• ODI 11g introduces option to install into Define projects
Create interfaces
interfaces and
packages
WebLogic server (more resilent, Release scenarios Operate production
clustering etc)
Repository
Standalone
Agent
Orchestrate
interfaces
and scenarios
Information Systems
(Databases, files, applications)
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
12. Master and Work Repositories
• ODI systems connects to a single Master Repository, and one or more Work Repositories
‣ Master Repository contains topology, security and version management details for all of an
organization’s resources (generally only one per organization)
‣ Work Repository contains data models, projects, interfaces etc
- Can have more than one work repository;
subdivide by project, department etc Security
‣ Work Repositories can either be Topology
Development or Execution work repositories Versioning
- Development work repositories are full-function,
and can be developed in Master Repository
- Execution work repositories can only import
compiled ETL routines (scenarios) and Models
execute them Projects Execution
Execution
Work Repository Execution Repository
(Development) (Production)
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
13. ODI 11g Overall Logical Architecture, incl. Java EE Components
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
14. ODI Heterogenous Connectivity
• Extract and load to many data sources
• Structured Data
‣ Oracle, SQL Server, IBM DB/2 etc
• Multi-dimensional Data
‣ Essbase, Oracle OLAP, MS AS
• Packaged Applications
• Legacy
‣ CICS, ISM, VSAM etc
• B2B
‣ EDIFACT, X12, RosettaNet
• Financial Management
‣ Hyperion, XBRL
• Broad range of protocols
‣ JCA, JDBC, ODBC, FTP, HTTP etc
T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
15. Business Rules and Technical Details
• ODI splits data mappings in to business rules, and
technical implementation
• Business rules define what goes where,
and using which transformation rules
• Technical implementation defines
how data is moved
‣ Changed Data Capture
‣ SQL to SQL
‣ File to SQL
‣ Slowly Changing Dimension
• Allows you to split mapping role into
business analyst and technical specialist
T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
16. Extensible Knowledge Modules
• Provided “out of the box”, and are user extensible
• Written for specific source and target platforms
• User created example : Oracle 10g Data Pump
T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
17. ODI Within Oracle Fusion Middleware 11g
• ODI is the data integration technology within Oracle Fusion Middleware 11g
• Leverages key FMW technologies such as
‣ WebLogic Server
‣ Oracle Platform Security Services
(optional)
‣ Web Services and SOA
‣ Clustering and HA
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
18. Oracle Data Integrator 11g New Features
• New Fusion IDE (Same as JDeveloper, SQL Developer)
• New Mapping Elements -
• J2EE Deployment Option (using WebLogic Server)
• Enterprise Scheduler (ESS) Integration
• Enterprise Manager Integration
• New ODI Console thin-client (replaces Metadata Navigator)
• Java SDK APIs for embedding in applications
• New SAP knowledge modules
• New Golden Gate knowledge modules
• New OBIEE knowledge module
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
19. New Fusion User Interface - ODI Studio
• New JDeveloper-based UI, same as SQL Developer, OWB 11gR2
• Now called ODI Studio, replaces
separate Designer, Topology,
Security etc tools
• Benefits from Fusion IDE
Framework features -
multi-tabs, log viewer panel etc
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
20. New Interface Elements
• Set operators for combining datasets (MINUS, INTERSECT etc)
• Temporary interface as derived table
• Complex file handling (multiple record files)
• Technology updates and editions
• Many more
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
21. Load Plans
• Alternative to packages for sequencing and orchestrating interfaces and procedures (scenarios)
• Allows conditional logic for mapping exection
• Define resume / restart settings for steps or groups of steps
• Explicitely define parallel / serial execution
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
22. Use of Repository Creation Utility
• ODI Repository can now be created using RCU, as well as direct through ODI
• No additional functionality, however aligns repository creation with rest of FMW11g
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
23. Use of Oracle WebLogic Server
• Agent can now be deployed within WebLogic Server domain
• Allows agent to take advantage of connection pooling, HA etc
• Deploy using a WebLogic Domain template
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
24. Oracle Data Integrator Console
• Evolution of ODI Metadata Navigator
• View metadata information, run compiled packages
• JEE thin-client application, runs in WebLogic Server
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
25. Oracle Fusion Middleware Control Integration
• Leverages Oracle Enterprise Manager Fusion Middleware Control
• View status of agenda
• View status of packages, interfaces
• Manage clustered agents
• View ODI metrics
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
26. ETL Integration with OBIEE 11g
• Connect the metadata in ODI and Oracle Business Intelligence
• View metadata from OBIEE catalog object through to ODI source
• OBIEE metadata extracted and loaded into ODI work repository, then reported on using
OBIEE dashboards
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
27. Demonstration
Oracle Data Integrator 11g
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
28. Some Typical ODI Development Tasks
• Creating an interface (mapping)
• Using some of the new 11g interface features - set operators, temporary table as sub-select
• Creating a package and use of scenarios
‣ And where load plans come in
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
29. Interfaces within ODI
• Interfaces define the mappings between sources and targets
• Uses an E-L-T approach
‣ Extract from sources; Load onto target; and Transform
• Defined using the Designer navigator
• Sequenced using packages
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
30. Business Rules and Technical Details
• ODI splits data interfaces in to business rules (mappings), and
technical implementation (flows)
• Business rules define what goes where,
and using which transformation rules
• Technical implementation defines
how data is moved
‣ Changed Data Capture
‣ SQL to SQL
‣ File to SQL
‣ Slowly Changing Dimension
• Allows you to split mapping role into
business analyst and technical specialist
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
31. Interface Designer within ODI Studio
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
32. Interface Designer within ODI Studio
Available Sources & Targets
• Drag and drop objects onto
mapping canvas
• Access files, database tables, XML
files & other datastores
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
33. Interface Designer within ODI Studio
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
34. Interface Designer within ODI Studio
Sources Canvas
• Data sources for the interface
• Add joins, filters
• Additional data sets through
set operations (union, intersect..)
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
35. Interface Designer within ODI Studio
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
36. Interface Designer within ODI Studio
Target Datastore
• Datastore that will be loaded
by the interface
• Columns can be transformed
• Can be defined as temporary
interface (input to other interface)
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
37. Interface Designer within ODI Studio
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
38. Interface Designer within ODI Studio
Property Inspector
• Define keys to be used
• Expressions for columns
• Other properties of the datastore
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
39. Interface Designer within ODI Studio
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
40. Components of an Interface
• An interface is made up of three elements
‣ The Mapping : the set of logical expressions mapping sources to target
‣ The Flow : the choice of KMs plus their run-time settings
‣ Controls (optional) : data-checking KMs that enforce data rules
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
41. Components of an Interface
• An interface is made up of three elements
‣ The Mapping : the set of logical expressions mapping sources to target
‣ The Flow : the choice of KMs plus their run-time settings
‣ Controls (optional) : data-checking KMs that enforce data rules
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
42. Components of an Interface
• An interface is made up of three elements
‣ The Mapping : the set of logical expressions mapping sources to target
‣ The Flow : the choice of KMs plus their run-time settings
‣ Controls (optional) : data-checking KMs that enforce data rules
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
43. Creating an Interface Step 1 : Create & Name Interface
• Right-click on Interfaces section in Project panel, select New Interface
• Name the interface, provide description
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
44. Creating an Interface Step 2 : Select Sources
• Drag and drop source datastores from Models panel onto sources area
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
45. Creating Joins Between Multiple Source Datastores
• More than one datastore can be dropped onto the sources canvas
• Join datastores by dragging and dropping key columns between datastores
• Set join condition in the properties panel if join is non-equijoin
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
46. Creating Filters
• Individual datastores can be filtered
• Drag and drop the column from the datastore onto the canvas
• Set filter condition in the properties inspector
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
47. Combining DataSets using Set Operations
• DataSets with same column definition can be combined using set operations
‣ Union, Union All, Intersect, Minus etc
• Press Add/Remove Dataset button, choose set operation
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
48. Combining DataSets using Set Operations
• DataSets with same column definition can be combined using set operations
‣ Union, Union All, Intersect, Minus etc
• Press Add/Remove Dataset button, choose set operation
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
49. Combining DataSets using Set Operations
• DataSets with same column definition can be combined using set operations
‣ Union, Union All, Intersect, Minus etc
• Press Add/Remove Dataset button, choose set operation
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
50. Creating an Interface Step 3 : Select Target Datastore
• Interfaces have one target datastore
• Drag and drop from the Models panel into the Target Datastore area
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
51. Creating an Interface Step 4 : Define Mappings - Part I
• Columns from the source area can either be dragged and dropped directly onto the
target datastore
• Or they can be created or transformed using SQL expressions
‣ SQL syntax is based on where the transformation is executed
‣ Source, staging or target
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
52. Creating an Interface Step 4 : Define Mappings - Part II
• Execute on property specifies where the mapping expression will get substituted in the
query generated by KM task
‣ Source - substituted in query to load data from source datastores into C$ table for
LKM and I$ table for IKM
‣ Staging - substituted in query to load data from C$ table into I$ table for LKM or
source datastores into I$ table for IKM
‣ Target - substituted in query to load data into target datastore either from I$ table or
source datastores
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
53. Temporary Datastores (“Yellow Interfaces”)
• Interfaces can become inputs into other interfaces
‣ Designated by yellow interface icon in Project panel
• Enter details in Temporary Target Properties panel to designate in this way
• Interface icon becomes yellow, can be dropped onto other interfaces as source
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
54. Creating an Interface Step 5 : Define Flows (KMs)
• Second part of the interface, defines the technologies used for the mapping
• Mapping is broken down into execution units
• Select the appropriate KM for each unit
• Select options presented by the KM (truncate table, check controls etc)
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
55. Creating an Interface Step 6 : Optionally, Use Check KM
• When enabled, runs all the checks defined by the constraints on the tables
• Moves error rows to E$ error tables
‣ Can be recycled later on if errors fixed
• Generally good practice to include CKM in interface if constraints defined
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
56. Creating an Interface Step 7 : Execute and Test Interface
• Press Execute to run interface
• View progress and outcome using the Operator navigator
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
57. Demonstration
Defining an Interface
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
58. What are Modular Interfaces?
• Complex Integration Task - Requirement is broken down into set of regular and
temporary (yellow) interfaces
• Pipeline - Interfaces are linked together to generate complex code or execute in an
orderly fashion
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
59. What are Temporary Interfaces?
• Modular - Interfaces designed to generate intermediate SQL, part of a complex query
or populate a temporary datastore
• Re-usable - can be used as a source in any regular or temporary interface
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
60. Create Modular Interface Step 1 - Create a Temporary Interface
• Provide a Name and Description
• On the Overview tab, select the right logical schema from the drop down menu
• Logical schema forces ODI to generate code specific to target technology
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
61. Create Modular Interface Step 2 - Set the Temporary Datastore
• Provide a temporary datastore name under Temporary Target Properties
• Under Target Datastore, create target columns and specify mappings
• Interface icon becomes yellow on saving
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
62. Create Modular Interface Step 3 - Set as a Source
• Drag and Drop temporary interface as a source
• Used like any other source to carry transformations building integration pipeline
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
63. Create Modular Interface Step 4 - Sub-Select Property
• Enable Sub-Select Property
‣ Code generated from temporary interface acts as a derived table
‣ Temporary interface execution not required
• Disable Sub-Select Property
‣ Temporary interface execution required
‣ Temporary datastore used as a source
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
64. Code Generation - Sub-Select Enabled
• Use simulation to generate
complex SQL
• Simulation can be performed at
any linked interface
• Verify the SQL
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
65. Demonstration
Temporary Interfaces, Sub-Select Tables and Interface Pipelines
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
66. Sequencing Interfaces using Packages and Scenarios
• ODI package is an organized sequence of steps
• Each step performs a small integration task
• They make up a workflow for data integration / data warehousing loads
• Provides execution control like looping, branching , serial/parallel execution of steps, etc
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
67. ODI Package Editor within ODI Studio
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
68. ODI Package Editor within ODI Studio
Diagram Palette
• Organizes sequence of steps to be
executed
• Provides visual look to the workflow
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
69. ODI Package Editor within ODI Studio
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
70. ODI Package Editor within ODI Studio
Properties Panel
• Each Step has different properties
• Sets value for properties of a step
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
71. ODI Package Editor within ODI Studio
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
72. ODI Package Editor within ODI Studio
Toolbar
• Icons to reorganize, delete, duplicate steps
• Icons to draw transitions between steps to
control executions
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
73. ODI Package Editor within ODI Studio
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
74. ODI Package Editor within ODI Studio
Toolbox
• Provides list of all the available ODI
tools
• ODI tools can be added as step within
the package
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
75. ODI Package Editor within ODI Studio
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
76. Scenarios (Compiled Code) In ODI
• Interfaces and other project objects can be added to packages in their regular,
uncompiled state
‣ Useful during development
• Or they can be compiled into Scenarios
‣ Fixes the code at a certain state
‣ Allows interfaces to be run in parallel
‣ Required when promoting into production
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
77. ODI Package - Controlling Execution
• Each step can have two possible next steps
‣ Next step upon success, transitioned using green “ok” line
‣ Next step upon failure, transitioned using red “ko” line
• Package stops If no next step is specified
• Every package should have a first step, indicated by green arrow icon
‣ First step is where the execution begins
‣ No first step, package does not compile
• Execution can branch
‣ On success or failure of a step
‣ Using evaluate variable as a step
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
78. Creating a Package Step 1 : Create and Name Package
• Right-click on the Packages entry under the required project, select New Package
• Name the package and fill the Description
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
79. Creating a Package Step 2 : Add Steps
• Switch to the Diagram tab, drag and drop the required ODI objects (scenarios,
variables, etc) as steps from the project
• ODI tool as step
‣ To add click the ODI tool under Toolbox and then click under diagram palette
‣ To stop adding ODI tool, click the arrow icon on the toolbar
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
80. Creating a Package Step 3 : Set Properties for Steps
• Select the step, under Properties panel set values for required properties
‣ Runtime substitution - ODI variablessubstitution api’s
‣ Static substitution - Hardcode values
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
81. Creating a Package Step 4 : Add Transitions
• Click the ok or ko transition icon on the toolbar and draw lines between steps
‣ ok (green line) to next step on success
‣ ko (red line) to next step on failure
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
82. Creating a Package Step 5 : Save and Execute
• Save ODI Package once all steps are added
• Test execution to confirm functionality
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
83. Demonstration
Compiling Interfaces into Scenarios, and Sequencing using a Package
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
84. New in ODI 11.1.1.5+ : Load Plans
• Alternative to packages for sequencing interfaces and other steps
• Helps organize an optimal execution schedule for a batch
• Advanced sequencing capabilities
‣ Parallel or Serial
‣ Conditional branching
‣ Exception handling
• Complements Scenarios and Packages, does not replace them
• Can be created and edited in both Development and Execution work repositories
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
85. Load Plans
• Built up of a hierarchy of steps
‣ Serial and Parallel run the steps within them
either sequentially, or in parallel
- Parallelize work to reduce net runtime
- for example, dimensions can often be
loaded independently, and thus
parallelized.
- Serialise work where dependencies exist
- eg. all dimensions must be loaded before
the facts can be
- eg. load base fact tables before
aggregates
‣ Run Scenario executes a named Scenario
- Doesn’t have to exist yet, can just be a
placeholder
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
86. Conditional Branching
• Variables can be included in a Load Plan, and used for conditional
execution of packages or steps
‣ Also where required by
• Case steps define a conditional branch in the Load Plan, based on
the evaluation of a variable
‣ When and Else steps define the condition on the variable
‣ Flexible list of operators
‣ Only the first matching condition is executed
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
87. Exception Handling
• If a load plan step fails, it can call an Execution Step
• It can either continue running, or raise the exception
• If the exception is raised, then the exception handler for the parent step is invoked
‣ Granular control of error handling, allowing groups of steps to fail whilst protecting the
overall batch
• Serial/Parallel tasks can be configured to raise an exception after x failures of child
steps, or a fixed timeout
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
88. Exception Steps
• Exception steps can be defined and attached to one or many Load Plan Steps
• Exception steps can invoke serial/parallel steps, scenarios
• Conditional branching can be used within exception steps
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
89. Restarting Failed Load Plans
• The behavior of each step and scenario in a load plan on restart can be configured
‣ Serial steps:
- Restart all children steps (failed or not)
- Restart from the failing child step
‣ Parallel steps:
- Restart all children steps (failed or not)
- Restart just the children step(s) that failed
• For Run Scenario steps:
‣ Restart session, session step, or session task
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
90. Running Load Plans
• Load Plans can be executed in several ways:
‣ On schedule, via the ODI scheduler
‣ Command Line
‣ Web Service
‣ ODI Studio (i.e. manually)
• A Load Plan must run on a JEE agent or standalone agent
‣ it cannot run on a Local Agent
• When a Load Plan runs, a Load Plan Instance is created
‣ This Load Plan Instance will have one or many Load Plan Runs associated with it
‣ Unlike restarting a session, which overwrites the previous one, a restarted Load Plan
will create a new Load Plan Run within the Load Plan Instance
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
91. Running Load Plans
• When a Load Plan is running, it can be monitored through Operator
• The view is the same as when editing, i.e. the steps and flow are shown in context
• Session IDs are hyperlinked to the session detail
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
92. Demonstration
Load Plans
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
93. Summary, and More Information
• ODI is a fully-featured ETL and data integration tool, and available now on OTN
• New features in ODI 11g to support complex mappings, enterprise deployments
• More information can be found at http://www.rittmanmead.com
• Contact us at info@rittmanmead.com or mark.rittman@rittmanmead.com
• Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/rittmanmead)
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
94. Inside Oracle Data Integrator 11g
Mark Rittman, Technical Director, Rittman Mead
OUG Ireland Conference 2012
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12