Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

No more unknown members! Smart data load validation for Hyperion Planning using ODI

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Wird geladen in …3
×

Hier ansehen

1 von 37 Anzeige

No more unknown members! Smart data load validation for Hyperion Planning using ODI

Herunterladen, um offline zu lesen

Usually, ODI data load interfaces for Essbase are simple and fast to be developed. But, depending on the data source quality, those interfaces may become a performance challenge. Essbase demands that all POV members in which we are trying to insert data to exist in the Essbase outline and when this is not true, Essbase switches its load method from Block Mode to Cell Mode. When this situation happens, one data load that would take only five minutes to complete may take several hours, degrading the Hyperion environment performance. Join us in this session to discover how we solved this problem in Dell's Computers in a dynamic way for any number of Hyperion Planning applications only using ODI data constraints and Hyperion Planning metadata repository to validate all POV members that will be used in the data load, guaranteeing the best performance and data quality on the Hyperion Planning environment.

Usually, ODI data load interfaces for Essbase are simple and fast to be developed. But, depending on the data source quality, those interfaces may become a performance challenge. Essbase demands that all POV members in which we are trying to insert data to exist in the Essbase outline and when this is not true, Essbase switches its load method from Block Mode to Cell Mode. When this situation happens, one data load that would take only five minutes to complete may take several hours, degrading the Hyperion environment performance. Join us in this session to discover how we solved this problem in Dell's Computers in a dynamic way for any number of Hyperion Planning applications only using ODI data constraints and Hyperion Planning metadata repository to validate all POV members that will be used in the data load, guaranteeing the best performance and data quality on the Hyperion Planning environment.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Anzeige

Ähnlich wie No more unknown members! Smart data load validation for Hyperion Planning using ODI (20)

Aktuellste (20)

Anzeige

No more unknown members! Smart data load validation for Hyperion Planning using ODI

  1. 1. No more unknown members! Smart data load validation for Hyperion Planning using ODI Ricardo Giampaoli Rodrigo Radtke
  2. 2. About the Speakers Giampaoli, Ricardo ● Oracle Ace Associate ● Master in Business Administration and IT management ● Founder of TeraCorp Consulting ● EPM training instructor ● Essbase/Planning/OBIEE/ODI Certified Specialist ● Blogger @ devepm.com Radtke, Rodrigo ● Oracle Ace Associate ● Graduated in Computer Engineering ● Software Developer Advisor at Dell ● ODI, Oracle and Java Certified ● Blogger @ devepm.com
  3. 3.  TeraCorp is a company specialized in products and services focused on EPM  TeraCorp mission is to create innovate solutions that helps people, businesses and partners to exceed their goals reaching their full potential.  Learn more @ www.teracorp.com.br/en About TeraCorp
  4. 4.  Planning uses RDBMS to store its metadata information ● One main repository is used for Planning system ● One additional repository for each Planning App  Planning uses Essbase as Database ● The Planning App repositories has the same Metadata information than Essbase Outline  The Metadata sync is done by “Refreshing” the metadata from Planning to Essbase Hyperion Planning Architecture
  5. 5.  Store all Data used in Hyperion Planning  One Essbase App for each Planning App  One Essbase cube per each Planning Plan Type ● Each cube has its own outline that matches the Planning metadata ● Each cube has its own Data Files that stores the Planning data Hyperion Essbase Architecture
  6. 6.  In order to load data in Essbase, its mandatory to inform a combination of members from all dimension and a data value ● Essbase reads data sources starting at the top and proceeding from left to right ● Essbase gets the sparse member’s combination informed and brings/creates the “block” in memory ● Essbase tries to load this member combination data into the block ● If all members from that combination are valid, the data is successfully loaded Essbase Data Load
  7. 7.  Three ways to load data: ●Using Essbase Load Data ● If a member is not valid, the process is aborted and no data is loaded ●Using Essbase Load Rules ● If a member is not valid, the process continues (depending of the configuration) and all data but the erroneous ones is loaded ●Using ODI ● ODI is a powerful ETL tool used to integrate all different data sources with Essbase using Java Essbase API to load data Essbase Data Load
  8. 8.  ODI sends chunks of data to Essbase based on a “Commit Interval” defined in the Knowledge Module  If a member is not valid, Essbase cancel the entire Chunk of data and ODI API switches to “Cell mode”, which sends one individual row to Essbase per time, until the commit interval ends, making the process extremely slow  Invalid member combination causes an error in the data load ● “2015-03-30 18:00:52,234 DEBUG [DwgCmdExecutionThread]: Error occurred in sending record chunk…Cannot end data load. Analytic Server Error(1003014): Unknown Member [ACT001] in Data Load, [0] Records Completed 2015-03-30 18:00:52,234 DEBUG [DwgCmdExecutionThread]: Sending data record by record to Essbase” Essbase Data Load using ODI
  9. 9.  In order to prevent “Cell mode” to happen, we can: ● Check the metadata before the load process to guarantee that all members are valid  We can do this dynamically and easily using ODI Constraints (CKM) and Planning Metadata Repository Essbase Data Load using ODI
  10. 10.  ODI uses Knowledge Modules (KM) to interact with different technologies ODI Architecture KM Type Description LKM Loading Loads heterogeneous data to a staging area IKM Integration Uses a given strategy to populate the target datastore from the Staging Area CKM Check Checks data in a datastore for errors statically or during an integration process RKM Reverse- engineering Retrieves the structure of a data model from a database. It is needed only for customized reverse-engineering. JKM Journalizing Creates the Change Data Capture framework objects in the source staging area SKM Data Services Deploys data services that provide access to data in datastores
  11. 11.  ODI Constraints checks automatically if the data inside a column is valid according to a specific condition  Constraints are used by the CKMs to log all invalid data to a E$ table  Need to validate all metadata columns before load to Essbase to guarantee that no invalid member will be sent (preventing “Cell mode” to trigger) ODI Constraints (CKM)
  12. 12.  ODI interacts with Planning/Essbase using diverse KMs ● For Planning ODI has only a Load Metadata KM ● For Essbase ODI has a load and extract KMs for both Metadata and Data ● ODI Knowledge Modules manipulates data to Planning/Essbase using Java API  ODI does not have a CKM for Essbase ● Data validation needs to happen in the inbound tables, before the data is sent to Essbase ODI Constraints (CKM) and Essbase
  13. 13.  CKM validation is tied to the DataStore (Inbound tables) ● For each inbound Table we need to create a set of constraints to validate the data ● This makes difficult when we have more planning applications to load data since we will have multiple inbounds jobs  If we have only one generic inbound table that contains one column for each planning dimension (Distinct of all dimensions from all Applications) we can have dynamic ODI Constraints to validate any number of Hyperion Planning applications ODI CKM and Inbound Tables
  14. 14.  For each dimension column in this table, an ODI constraint will be created to guarantee that the data is valid for that dimension ● Create the table respecting the Sparse dimensions first and dense dimension later, this way we can load a entire block each time ● The dense members are set before the data columns to improve performance ● If the job loads an entire year the Period members could go into columns (Loads an entire block per time) ● APP_NAME, CUBE and INTERFACE_NAME is used for multiple application purposes Inbound Generic table design
  15. 15.  In order to validate against Essbase outline, we would need to extract all outline members first ● One ODI interface per dimension ● Can be very slow in some large dimensions  Easier if we validate against Planning Repository (RDBMS) ● All metadata information from all dimensions is inside the Planning App repository ● A lot faster to extract the metadata since it uses SQL ● Can be used even without the need of extracting the metadata ODI Constraints (CKM) Estrategy
  16. 16.  Option 1: ● Inbound table is in the same Database as Planning App Repository or using DB Link ● Uses the Planning Repository SQL inside the ODI Constraints querying the Planning App repository on the fly  Option 2: ● Inbound table in a different Database as Planning App Repository ● Needs to export all existing metadata from Planning App Repository to a temporary table first and then create the ODI Constraints to query this temp table Data Validation Architecture
  17. 17.  All Planning App metadata is stored in only one table Planning Repository (HSP_OBJECT) Column Name Description OBJECT_ID Object ID for all objects in planning. OBJECT_NAME Stores all metadata description in Planning (e.g. Alias, Members) OBJECT_TYPE Type of the Object (e.g. Entity, Account, Attribute…) PARENT_ID Parent ID of the object. Used for build the parent/child relationship with OBJECT_ID GENERATION Inform which generation that object belongs. HAS_CHILDREN Inform if the object has or not a child
  18. 18.  All existing metadata can be retrieved by a single query on HSP_OBJECT  If we change the query a little we can bring all members at once Repository Query This query will return all Accounts Hierarchy This query will return all Hierarchies
  19. 19.  Not all members in HSP_OBJECT will be loaded to all Essbase Cubes ● The member’s Plan Type defines in which Cube it will exist  Each Essbase Cube is represented by a Plan Type in Planning  Each Planning member may belong to multiple Plan Types  Data load occurs in one cube at a time ● That’s why we must find out which Plan Types a member belongs in order to validate the data for the correct cube Planning Repository
  20. 20.  The properties of the members are stored in this table Planning Repository (HSP_MEMBER) Column Name Description MEMBER_ID Same as Object ID. USED_IN Defines in what Plan Types (Cubes) that member will belong. In case of a member exists in more than one Plan Type, Planning Sums the value of all plan types its belongs Plan Types: 1 2 3 4 5 Used In: 1 2 4 8 16
  21. 21.  This table stores the Plan Types created in the Planning App Planning Repository (HSP_PLAN_TYPE) Column Name Description PLAN_TYPE ID of the Plan Type TYPE_NAME Name of the Plan Type
  22. 22.  For each plan type that the member belongs Planning SUMs the Used In ● If a Member exists in Plan Type 1 and 2 Used In 1 + 2 = 3 ● All Plan Types = 1 + 2 + 4 + 8 + 16 = 31  To figure out if a member belongs to a plan type we need to verify if its “Used In” is included in the SUM ● Plan Type 5 (Used in 16) exists in 31? ● Plan Type 1 (Used in 1) exists in 30? ● Which Plan Type exists in 20? Planning Repository Plan Type Used In Plan Type 1 1 Plan Type 2 2 Plan Type 3 4 Plan Type 4 8 Plan Type 5 16
  23. 23.  MOD returns the remainder (modulus) of argument 1 divided by argument 2 ● If MOD of Used In per actual Plan Type * 2 is >= actual Plan Type then it exists in the SUM of Plan Types. ● Plan Type 5 exists in 31 ● Plan Type 1 not exists in 30 ● Plan Type 3 and 5 exists in 20 Planning Repository
  24. 24.  All necessary validation data will come from these three tables ● A member that belongs to two Plan Types will appear two times ● MOD technique is used to discover which Plan Type that member belongs to and “OR” is used to combine more than one Plan Type Repository Query This query multiplies all members depending of how many Plan type it belongs
  25. 25.  Another way to select a Plan Type is using BITAND ● BITAND computes an AND operation of the bits Planning Repository Plan Type Used In Binary Plan Type 1 1 00001 Plan Type 2 2 00010 Plan Type 3 4 00100 Plan Type 4 8 01000 Plan Type 5 16 10000
  26. 26.  Three types of ODI constraints: ● Key: Checks if there are duplicated records or NULL values in the data ● Reference: Checks the relationship (Foreign key) between two tables. Both tables needs to be reverse-engineered inside ODI models ● Condition: Checks for any other specific rule using custom queries  Better to use Condition: ● No additional ODI models required (just the inbound table) ● More SQL flexibility ● Allow complex logic checks ODI Constraints (CKM) Implementation
  27. 27.  Validates directly against Hyperion Planning repository ● One constraint per dimension ● No need of any “extra” metadata extract  Inbound table and Planning Repository schemas needs to be in the same database or using DB Link ● ODI needs to execute a SQL that access both tables (Inbound table and Planning repository) in the same connection Option 1: Constraints Against Planning App
  28. 28.  2 changes in the query for all constraints  2 ODI variables that dynamically switches between Planning Applications and Plan Types Option 1: Constraints Against Planning App
  29. 29.  Need to extract the metadata to a temporary table every time  One constraint per dimension  Inbound table and Planning Repository schemas may exist in different Databases  Temp table will be created on the same Inbound table schema Option 2: Constraints Against a Temporary Table
  30. 30. Option 2: Constraints against a temporary table  Gets all Dimensions at same time  1 ODI variables that dynamically switches between Planning Applications
  31. 31. Option 2: Constraints Creation  2 changes in the query for all constraints  2 ODI variables that dynamically switches between Planning Applications and Plan Types
  32. 32. ODI Constraints Execution  When we open ODI Operator, each constraint validation will be shown in the “Control” steps  Each step will show how may invalid members was removed from the inbound data  Invalid data is sent to the E$ error table
  33. 33.  With only one inbound generic table, we will have only one generic E$ table ● Stores all the POV and the data that fails the validation ● ODI_Cons_name, Interface_Name, App_Name, Cube and ODI_Sess_NO identifies what was the error, from which package that error came from and to which Planning app it should have loaded Generic E$
  34. 34.  Easy to add new Planning Applications  Generic Inbound tables makes easier to create Generic components ● Send Email ● Text Logs (export from E$ table) ● Error Handling ● Fewer ODI components to Maintain Generic Structure Benefits
  35. 35. Overall Architecture Excel Oracle Sql Server XML Teradata Sources Stage Area Inbound Table Table 1 Table 2 Table 3 Table 4 Table N Inbound Generic E$ Table E$ Inbound Generic Generic Components Error Handling Send Email Planning Cube 1 Cube 2 Cube 3 Cube N
  36. 36. QUESTIONS?
  37. 37. Ricardo Giampaoli – TeraCorp Rodrigo Radtke de Souza - Dell Thank you!

×