WSO2's API Vision: Unifying Control, Empowering Developers
Effective capture of metadata using ca e rwin data modeler 09232010
1. Effective Capture of Metadata Using
CA ERwin Data Modeler
Metadata –Data gets meaning.
2. Abstract
• In any data centric environment either Data warehouse or an OLTP data
environment ,vital information anybody looks for is the purpose and content of the
tables and columns. Its metadata of the data, provides more insight about the
structure. In many organization, the lack of metadata has lead to redundant
definition of table and columns , ignorance of real capability of your data centric
system, inability to define standards and build the knowledge layer for the business.
In the case of data warehouse its vital to capture the source and transformation
rules along with dimensional model as it will help fixing incorrect mappings early in
the life cycle, effective communication to ETL team and store the ETL rules close to
the data model. Without metadata ,it will lead to individual's interpretation of data
just like blind folded touching the elephant. In this webinar we will discuss about the
various flexible features provided by CA ERwin Data Modeler for data warehouse
and relational model.
PAGE 2
3. Speaker Bio
• Sampath Kumar brings 11 years of experience in implementing
small, medium and large scale data centric environments (both
relational and data warehouse ) using CA ERwin Modeling suite of
products. He is currently working for Infosys Technologies Limited
as Technology Architect in their DW/BI practice group. Prior to that
he was working with American Express Credit Cards as Sr System
Analyst for their Worldwide Risk Information Management group.
In all his experience he has worked extensively in various database
products ,BI tools ,data modeling and related products offered by
CA such as CA ERwin Data Modeler, CA ERwin Model Validator,CA
ERwin Model Manager and CA ERwin Data Profiler.
PAGE 3
4. Agenda
• Not going to focus on the known fundamentals and data jargons
• Effective capture of metadata in Data warehouse environment
– Case study using Customer_Dim
• Other Flexible Options to capture metadata into data model
– Using an example.
PAGE 4
5. Effective capture of metadata in Data
warehouse environment
Case study using Customer_Dim
6. Problem Statement
In any data warehouse development project, some of the major
challenges include
• Effective capture of metadata information in data model such as
data source ,transformation, enrichment and data synchronization
rules etc.
• Keeping data model in synch with changing ETL rules and vice versa
i.e. keeping ETL rules close to DW Data model (blueprint of your
DW data)
• Early identification of incorrect ETL mappings in the complete
lifecycle.
PAGE 6
7. Problem Statement contd…
• Effective communication of captured metadata information by data
modeler to other teams such as ETL
PAGE 7
8. Background
3 Important pieces of information:
• Source of data
• Transformation rules-The method in which the data is getting
extracted, transformed and loaded
• Frequency: The frequency and timing of data warehouse updates.
PAGE 8
9. CA ERwin Features
CA ERwin Data Modeler supports the following salient features to
capture the metadata information effectively.
• Data warehouse Sources Dialog
• Columns Editor
• Data Movement Rules Editor
PAGE 9
10. Customer_Dim
Snapshot Communication
– customer_SKID Email Address
– snapshot_Begin_Date Phone
– snapshot_End_Date Fax
– current_ind Segmentation
• Basic Information Shopping
– Customer name Behavior
– Customer Date of Birth
– Driving License
• Address
– Mailing Address
– Physical Address
PAGE 10
18. Data warehouse Sources
The “Import other” provides three options to import the table structure
• Flat File
• Database/Script
• Model Manager
PAGE 18
19. Importing table from CA ERwin® Model Manager
Customer_Address
customer_id (FK)
mailing_address_line1
mailing_address_line2
mailing_city
mailing_state
mailing_county
mailing_country
physical_address_line1
Customer physical_address_line2
physical_county
customer_id
physical_city
customer_first_name physical_state
Opens Model Mart customer_last_name physical_country
dob
Library driving_license_nbr
driving_license_state
Behavioral_Segment
behavioral_segment_nbr
behavioral_segment_name
Customer_Segmentation
customer_id (FK)
behavioral_segment_nbr (FK)
Shopping_Segment
shopping_segment_nbr (FK)
shopping_segment_nbr
shopping_segment_name
PAGE 19
35. Metadata Capture from MS Excel
• When it would be useful
– Import the definitions available already into data model
– Import the definitions from business stakeholders for key columns to avoid wrong
interpretation.
• Step 1: Store the model locally in the hard disk
• Step 2: Use the excel sheet “Import Definitions” or VBA macro provided by CA .
• Step 3 :Import the metadata into the model by running VBA code.
PAGE 35
39. Final Step
Open the first sheet and click on “Update Entity Defns” which will
update the definitions written for that particular table into the data
model. Similarly click on the “Update Attribute Defns” which will
update the attribute definitions.
Note:
• Keep the data model closed otherwise you will get error that it’s open.
• Make sure table and column names are exactly same as in the data model.
• It’s not only for business people but also for the data modelers who can enter
the definitions in MS Excel and get the approval from the business or data
management team, then it can uploaded separately using this utility.
PAGE 39
42. Conclusion
• The metadata information such as “Data Source”, “Transformations
rules” and “Data Movement rules” are very important for any Data
warehousing efforts and it’s very critical to capture the correct
information.
• Metadata from data management standpoint , reduces
considerable amount of time while consolidating the attributes or
entities or databases during acquisition or merger.
• Knowing the importance of metadata for the data model ,CA ERwin
has provided these flexible options which can be leveraged to
make the data model & data more meaningful.
PAGE 42
43. Questions?
In case of any additional questions you can reach me in
Sampath_Kumar01@infosys.com
Sampath.k.kumar@gmail.com
PAGE 43
