1.
No more unknown members!
Smart data load validation for Hyperion Planning
using ODI
Ricardo Giampaoli
Rodrigo Radtke

2.
About the Speakers
Giampaoli, Ricardo
● Oracle Ace Associate
● Master in Business Administration
and IT management
● Founder of TeraCorp Consulting
● EPM training instructor
● Essbase/Planning/OBIEE/ODI
Certified Specialist
● Blogger @ devepm.com
Radtke, Rodrigo
● Oracle Ace Associate
● Graduated in Computer
Engineering
● Software Developer Advisor at Dell
● ODI, Oracle and Java Certified
● Blogger @ devepm.com

3.
 TeraCorp is a company specialized in products and services
focused on EPM
 TeraCorp mission is to create innovate solutions that helps
people, businesses and partners to exceed their goals reaching
their full potential.
 Learn more @ www.teracorp.com.br/en
About TeraCorp

4.
 Planning uses RDBMS to store its metadata
information
● One main repository is used for Planning system
● One additional repository for each Planning App
 Planning uses Essbase as Database
● The Planning App repositories has the same
Metadata information than Essbase Outline
 The Metadata sync is done by “Refreshing” the
metadata from Planning to Essbase
Hyperion Planning Architecture

5.
 Store all Data used in Hyperion
Planning
 One Essbase App for each
Planning App
 One Essbase cube per each
Planning Plan Type
● Each cube has its own outline that
matches the Planning metadata
● Each cube has its own Data Files
that stores the Planning data
Hyperion Essbase Architecture

6.
 In order to load data in Essbase, its mandatory to inform a
combination of members from all dimension and a data value
● Essbase reads data sources starting at the top and proceeding from left
to right
● Essbase gets the sparse member’s combination informed and
brings/creates the “block” in memory
● Essbase tries to load this member combination data into the block
● If all members from that combination are valid, the data is successfully
loaded
Essbase Data Load

7.
 Three ways to load data:
●Using Essbase Load Data
● If a member is not valid, the process is aborted and no data is
loaded
●Using Essbase Load Rules
● If a member is not valid, the process continues (depending of
the configuration) and all data but the erroneous ones is loaded
●Using ODI
● ODI is a powerful ETL tool used to integrate all different data
sources with Essbase using Java Essbase API to load data
Essbase Data Load

8.
 ODI sends chunks of data to Essbase based on a “Commit Interval”
defined in the Knowledge Module
 If a member is not valid, Essbase cancel the entire Chunk of data and
ODI API switches to “Cell mode”, which sends one individual row to
Essbase per time, until the commit interval ends, making the process
extremely slow
 Invalid member combination causes an error in the data load
● “2015-03-30 18:00:52,234 DEBUG [DwgCmdExecutionThread]: Error occurred in sending
record chunk…Cannot end data load. Analytic Server Error(1003014): Unknown
Member [ACT001] in Data Load, [0] Records Completed 2015-03-30 18:00:52,234
DEBUG [DwgCmdExecutionThread]: Sending data record by record to Essbase”
Essbase Data Load using ODI

9.
 In order to prevent “Cell mode” to happen, we can:
● Check the metadata before the load process to guarantee that all
members are valid
 We can do this dynamically and easily using ODI Constraints
(CKM) and Planning Metadata Repository
Essbase Data Load using ODI

10.
 ODI uses Knowledge Modules (KM) to interact with different
technologies
ODI Architecture
KM Type Description
LKM Loading Loads heterogeneous data to a staging area
IKM Integration Uses a given strategy to populate the target datastore from the Staging Area
CKM Check Checks data in a datastore for errors statically or during an integration process
RKM Reverse-
engineering
Retrieves the structure of a data model from a database. It is needed only for
customized reverse-engineering.
JKM Journalizing Creates the Change Data Capture framework objects in the source staging area
SKM Data Services Deploys data services that provide access to data in datastores

11.
 ODI Constraints checks automatically if the data inside a column
is valid according to a specific condition
 Constraints are used by the CKMs to log all invalid data to a E$
table
 Need to validate all metadata columns before load to Essbase to
guarantee that no invalid member will be sent (preventing “Cell
mode” to trigger)
ODI Constraints (CKM)

12.
 ODI interacts with Planning/Essbase using diverse KMs
● For Planning ODI has only a Load Metadata KM
● For Essbase ODI has a load and extract KMs for both Metadata and
Data
● ODI Knowledge Modules manipulates data to Planning/Essbase using
Java API
 ODI does not have a CKM for Essbase
● Data validation needs to happen in the inbound tables, before the data is
sent to Essbase
ODI Constraints (CKM) and Essbase

13.
 CKM validation is tied to the DataStore (Inbound tables)
● For each inbound Table we need to create a set of constraints to
validate the data
● This makes difficult when we have more planning applications to load
data since we will have multiple inbounds jobs
 If we have only one generic inbound table that contains one
column for each planning dimension (Distinct of all dimensions
from all Applications) we can have dynamic ODI Constraints to
validate any number of Hyperion Planning applications
ODI CKM and Inbound Tables

14.
 For each dimension column in this table, an
ODI constraint will be created to guarantee that
the data is valid for that dimension
● Create the table respecting the Sparse dimensions first and
dense dimension later, this way we can load a entire block
each time
● The dense members are set before the data columns to
improve performance
● If the job loads an entire year the Period members could go
into columns (Loads an entire block per time)
● APP_NAME, CUBE and INTERFACE_NAME is used for
multiple application purposes
Inbound Generic table design

15.
 In order to validate against Essbase outline, we would need to
extract all outline members first
● One ODI interface per dimension
● Can be very slow in some large dimensions
 Easier if we validate against Planning Repository (RDBMS)
● All metadata information from all dimensions is inside the Planning App
repository
● A lot faster to extract the metadata since it uses SQL
● Can be used even without the need of extracting the metadata
ODI Constraints (CKM) Estrategy

16.
 Option 1:
● Inbound table is in the same Database as Planning App Repository or
using DB Link
● Uses the Planning Repository SQL inside the ODI Constraints querying the
Planning App repository on the fly
 Option 2:
● Inbound table in a different Database as Planning App Repository
● Needs to export all existing metadata from Planning App Repository to a temporary
table first and then create the ODI Constraints to query this temp table
Data Validation Architecture

17.
 All Planning App metadata is stored in only one table
Planning Repository (HSP_OBJECT)
Column Name Description
OBJECT_ID Object ID for all objects in planning.
OBJECT_NAME Stores all metadata description in Planning (e.g. Alias, Members)
OBJECT_TYPE Type of the Object (e.g. Entity, Account, Attribute…)
PARENT_ID Parent ID of the object. Used for build the parent/child relationship with OBJECT_ID
GENERATION Inform which generation that object belongs.
HAS_CHILDREN Inform if the object has or not a child

18.
 All existing metadata can be retrieved by a single query on
HSP_OBJECT
 If we change the query a little we can bring all members at once
Repository Query
This query will return
all Accounts Hierarchy
This query will return
all Hierarchies

19.
 Not all members in HSP_OBJECT will be loaded to all Essbase
Cubes
● The member’s Plan Type defines in which Cube it will exist
 Each Essbase Cube is represented by a Plan Type in Planning
 Each Planning member may belong to multiple Plan Types
 Data load occurs in one cube at a time
● That’s why we must find out which Plan Types a member
belongs in order to validate the data for the correct cube
Planning Repository

20.
 The properties of the members are stored in this table
Planning Repository (HSP_MEMBER)
Column Name Description
MEMBER_ID Same as Object ID.
USED_IN
Defines in what Plan Types (Cubes) that member will belong.
In case of a member exists in more than one Plan Type, Planning Sums the value
of all plan types its belongs
Plan Types: 1 2 3 4 5
Used In: 1 2 4 8 16

21.
 This table stores the Plan Types created in the Planning App
Planning Repository (HSP_PLAN_TYPE)
Column Name Description
PLAN_TYPE ID of the Plan Type
TYPE_NAME Name of the Plan Type

22.
 For each plan type that the member belongs
Planning SUMs the Used In
● If a Member exists in Plan Type 1 and 2 Used In 1 + 2 = 3
● All Plan Types = 1 + 2 + 4 + 8 + 16 = 31
 To figure out if a member belongs to a plan type we
need to verify if its “Used In” is included in the SUM
● Plan Type 5 (Used in 16) exists in 31?
● Plan Type 1 (Used in 1) exists in 30?
● Which Plan Type exists in 20?
Planning Repository
Plan Type Used In
Plan Type 1 1
Plan Type 2 2
Plan Type 3 4
Plan Type 4 8
Plan Type 5 16

23.
 MOD returns the remainder
(modulus) of argument 1 divided
by argument 2
● If MOD of Used In per actual Plan
Type * 2 is >= actual Plan Type then
it exists in the SUM of Plan Types.
● Plan Type 5 exists in 31
● Plan Type 1 not exists in 30
● Plan Type 3 and 5 exists in 20
Planning Repository

24.
 All necessary validation data will come from these three tables
● A member that belongs to two Plan Types will appear two times
● MOD technique is used to discover which Plan Type that member
belongs to and “OR” is used to combine more than one Plan Type
Repository Query
This query multiplies all
members depending of
how many Plan type it
belongs

25.
 Another way to select a
Plan Type is using
BITAND
● BITAND computes an
AND operation of the bits
Planning Repository
Plan Type Used In Binary
Plan Type 1 1 00001
Plan Type 2 2 00010
Plan Type 3 4 00100
Plan Type 4 8 01000
Plan Type 5 16 10000

26.
 Three types of ODI constraints:
● Key: Checks if there are duplicated records or NULL
values in the data
● Reference: Checks the relationship (Foreign key) between two tables.
Both tables needs to be reverse-engineered inside ODI models
● Condition: Checks for any other specific rule using custom queries
 Better to use Condition:
● No additional ODI models required (just the inbound table)
● More SQL flexibility
● Allow complex logic checks
ODI Constraints (CKM) Implementation

27.
 Validates directly against Hyperion Planning repository
● One constraint per dimension
● No need of any “extra” metadata extract
 Inbound table and Planning Repository schemas needs to be in
the same database or using DB Link
● ODI needs to execute a SQL that access both tables (Inbound table and
Planning repository) in the same connection
Option 1: Constraints Against Planning App

28.
 2 changes in the
query for all
constraints
 2 ODI variables
that dynamically
switches between
Planning
Applications and
Plan Types
Option 1: Constraints Against Planning App

29.
 Need to extract the metadata to a temporary table every time
 One constraint per dimension
 Inbound table and Planning Repository schemas may exist in
different Databases
 Temp table will be created on the same Inbound table schema
Option 2: Constraints Against a Temporary Table

30.
Option 2: Constraints against a temporary table
 Gets all Dimensions at
same time
 1 ODI variables that
dynamically switches
between Planning
Applications

31.
Option 2: Constraints Creation
 2 changes in the
query for all
constraints
 2 ODI variables
that dynamically
switches between
Planning
Applications and
Plan Types

32.
ODI Constraints Execution
 When we open ODI
Operator, each constraint
validation will be shown
in the “Control” steps
 Each step will show how
may invalid members
was removed from the
inbound data
 Invalid data is sent to the
E$ error table

33.
 With only one inbound generic table, we will have only one
generic E$ table
● Stores all the POV and the data that fails the validation
● ODI_Cons_name, Interface_Name, App_Name, Cube and
ODI_Sess_NO identifies what was the error, from which package that
error came from and to which Planning app it should have loaded
Generic E$

34.
 Easy to add new Planning Applications
 Generic Inbound tables makes easier to create Generic
components
● Send Email
● Text Logs (export from E$ table)
● Error Handling
● Fewer ODI components to Maintain
Generic Structure Benefits

35.
Overall Architecture
Excel
Oracle
Sql
Server
XML
Teradata
Sources Stage Area
Inbound Table
Table 1
Table 2
Table 3
Table 4
Table N
Inbound
Generic
E$ Table
E$ Inbound
Generic
Generic Components
Error
Handling
Send
Email
Planning
Cube
1
Cube
2
Cube
3
Cube
N

36.
QUESTIONS?

37.
Ricardo Giampaoli – TeraCorp
Rodrigo Radtke de Souza - Dell
Thank you!