SlideShare a Scribd company logo
Effective Capture of Metadata Using
CA ERwin Data Modeler
Metadata –Data gets meaning.
Abstract

• In any data centric environment either Data warehouse or an OLTP data
  environment ,vital information anybody looks for is the purpose and content of the
  tables and columns. Its metadata of the data, provides more insight about the
  structure. In many organization, the lack of metadata has lead to redundant
  definition of table and columns , ignorance of real capability of your data centric
  system, inability to define standards and build the knowledge layer for the business.
  In the case of data warehouse its vital to capture the source and transformation
  rules along with dimensional model as it will help fixing incorrect mappings early in
  the life cycle, effective communication to ETL team and store the ETL rules close to
  the data model. Without metadata ,it will lead to individual's interpretation of data
  just like blind folded touching the elephant. In this webinar we will discuss about the
  various flexible features provided by CA ERwin Data Modeler for data warehouse
  and relational model.



    PAGE 2
Speaker Bio

• Sampath Kumar brings 11 years of experience in implementing
  small, medium and large scale data centric environments (both
  relational and data warehouse ) using CA ERwin Modeling suite of
  products. He is currently working for Infosys Technologies Limited
  as Technology Architect in their DW/BI practice group. Prior to that
  he was working with American Express Credit Cards as Sr System
  Analyst for their Worldwide Risk Information Management group.
  In all his experience he has worked extensively in various database
  products ,BI tools ,data modeling and related products offered by
  CA such as CA ERwin Data Modeler, CA ERwin Model Validator,CA
  ERwin Model Manager and CA ERwin Data Profiler.


  PAGE 3
Agenda

• Not going to focus on the known fundamentals and data jargons
• Effective capture of metadata in Data warehouse environment
  – Case study using Customer_Dim
• Other Flexible Options to capture metadata into data model
  – Using an example.




  PAGE 4
Effective capture of metadata in Data
warehouse environment
Case study using Customer_Dim
Problem Statement
 In any data warehouse development project, some of the major
  challenges include
• Effective capture of metadata information in data model such as
  data source ,transformation, enrichment and data synchronization
  rules etc.
• Keeping data model in synch with changing ETL rules and vice versa
  i.e. keeping ETL rules close to DW Data model (blueprint of your
  DW data)
• Early identification of incorrect ETL mappings in the complete
  lifecycle.



  PAGE 6
Problem Statement contd…

• Effective communication of captured metadata information by data
  modeler to other teams such as ETL




  PAGE 7
Background

3 Important pieces of information:
• Source of data
• Transformation rules-The method in which the data is getting
  extracted, transformed and loaded
• Frequency: The frequency and timing of data warehouse updates.




  PAGE 8
CA ERwin Features

CA ERwin Data Modeler supports the following salient features to
 capture the metadata information effectively.


• Data warehouse Sources Dialog


• Columns Editor


• Data Movement Rules Editor



  PAGE 9
Customer_Dim

  Snapshot                          Communication

       –   customer_SKID                 Email Address

       –   snapshot_Begin_Date           Phone

       –   snapshot_End_Date             Fax

       –   current_ind              Segmentation

 • Basic Information                     Shopping

       –   Customer name                 Behavior

       –   Customer Date of Birth
       –   Driving License
 • Address
       –   Mailing Address
       –   Physical Address




 PAGE 10
Capturing Data Source




 PAGE 11
Creating Customer_Dim




 PAGE 12
Make it Dimensional Model




 PAGE 13
Make it Dimensional Model…




 PAGE 14
Data Source Enabled…




 PAGE 15
Data warehouse Source Selector




 PAGE 16
Data warehouse Source




 PAGE 17
Data warehouse Sources

The “Import other” provides three options to import the table structure
• Flat File
• Database/Script
• Model Manager




   PAGE 18
Importing table from CA ERwin® Model Manager
                                                           Customer_Address
                                                            customer_id (FK)

                                                            mailing_address_line1
                                                            mailing_address_line2
                                                            mailing_city
                                                            mailing_state
                                                            mailing_county
                                                            mailing_country
                                                            physical_address_line1
                             Customer                       physical_address_line2
                                                            physical_county
                              customer_id
                                                            physical_city
                              customer_first_name           physical_state
      Opens Model Mart        customer_last_name            physical_country
                              dob
           Library            driving_license_nbr
                              driving_license_state


                                                          Behavioral_Segment
                                                           behavioral_segment_nbr

                                                           behavioral_segment_name
                           Customer_Segmentation
                            customer_id (FK)

                            behavioral_segment_nbr (FK)
                                                          Shopping_Segment
                            shopping_segment_nbr (FK)
                                                           shopping_segment_nbr

                                                           shopping_segment_name


 PAGE 19
Import source tables




 PAGE 20
Source Tables Populated




 PAGE 21
Data warehouse Source Selector.      Multiple
                                  sources can be
                                      added




                                      Transformation
                                       and business
                                        rules can be
 PAGE 22
                                        added here.
Source as Flat File




  PAGE 23
ETL Mapping Template-using Data Browser




 PAGE 24
Report Template Builder




 PAGE 25
Customize the Report




 PAGE 26
Generate the Metadata Report




 PAGE 27
ETL Mapping Sheet




 PAGE 28
Data Movement Rules




 PAGE 29
Documenting the rule




 PAGE 30
Attaching table to rule.




  PAGE 31
Export as Report




 PAGE 32
Other Flexible Options to capture
metadata into data model
Using simple example
Import from MS Excel
Using simple example
Metadata Capture from MS Excel

• When it would be useful
   – Import the definitions available already into data model
   – Import the definitions from business stakeholders for key columns to avoid wrong
     interpretation.


• Step 1: Store the model locally in the hard disk


• Step 2: Use the excel sheet “Import Definitions” or VBA macro provided by CA .


• Step 3 :Import the metadata into the model by running VBA code.




   PAGE 35
Person Table




 PAGE 36
Import Definitions




 PAGE 37
In this format




  PAGE 38
Final Step

Open the first sheet and click on “Update Entity Defns” which will
 update the definitions written for that particular table into the data
 model. Similarly click on the “Update Attribute Defns” which will
 update the attribute definitions.
Note:
• Keep the data model closed otherwise you will get error that it’s open.
• Make sure table and column names are exactly same as in the data model.
• It’s not only for business people but also for the data modelers who can enter
  the definitions in MS Excel and get the approval from the business or data
  management team, then it can uploaded separately using this utility.




   PAGE 39
Capture metadata in Data Browser.
Metadata Capture using Data Browser




 PAGE 41
Conclusion

• The metadata information such as “Data Source”, “Transformations
  rules” and “Data Movement rules” are very important for any Data
  warehousing efforts and it’s very critical to capture the correct
  information.
• Metadata from data management standpoint , reduces
  considerable amount of time while consolidating the attributes or
  entities or databases during acquisition or merger.
• Knowing the importance of metadata for the data model ,CA ERwin
  has provided these flexible options which can be leveraged to
  make the data model & data more meaningful.


  PAGE 42
Questions?




             In case of any additional questions you can reach me in
                        Sampath_Kumar01@infosys.com
                          Sampath.k.kumar@gmail.com
 PAGE 43

More Related Content

What's hot

Generating XML schemas from a Logical Data Model (EDW 2011)
Generating XML schemas from a Logical Data Model (EDW 2011)Generating XML schemas from a Logical Data Model (EDW 2011)
Generating XML schemas from a Logical Data Model (EDW 2011)George McGeachie
 
ActiveWarehouse/ETL - BI & DW for Ruby/Rails
ActiveWarehouse/ETL - BI & DW for Ruby/RailsActiveWarehouse/ETL - BI & DW for Ruby/Rails
ActiveWarehouse/ETL - BI & DW for Ruby/RailsPaul Gallagher
 
Peoplesoft Basic App designer
Peoplesoft Basic App designerPeoplesoft Basic App designer
Peoplesoft Basic App designermbtechnosolutions
 
George McGeachie's Favourite PowerDesigner features
George McGeachie's Favourite PowerDesigner featuresGeorge McGeachie's Favourite PowerDesigner features
George McGeachie's Favourite PowerDesigner featuresGeorge McGeachie
 
Introduction to database
Introduction to databaseIntroduction to database
Introduction to databaseshukriyah
 
Excellence In Excel Presentation
Excellence In Excel PresentationExcellence In Excel Presentation
Excellence In Excel Presentationcynosure76
 
People soft application-designer-practice-8.43
People soft application-designer-practice-8.43People soft application-designer-practice-8.43
People soft application-designer-practice-8.43cesarvii
 
Sharing data in a multitenant architecture
Sharing data in a multitenant architectureSharing data in a multitenant architecture
Sharing data in a multitenant architectureDan Fey
 
Internet Environment
Internet  EnvironmentInternet  Environment
Internet Environmentguest8fdbdd
 
Merchant Product Datafeeds for Affiliates 101
Merchant Product Datafeeds for Affiliates 101Merchant Product Datafeeds for Affiliates 101
Merchant Product Datafeeds for Affiliates 101Carsten Cumbrowski
 
Lightning talk at PG Conf UK 2018
Lightning talk at PG Conf UK 2018Lightning talk at PG Conf UK 2018
Lightning talk at PG Conf UK 2018George McGeachie
 
Analysis for office training
Analysis for office   trainingAnalysis for office   training
Analysis for office trainingKibrom Gebrehiwot
 
Migrating from CA AllFusionTM ERwin® Data Modeler to Embarcadero ER/Studio
Migrating from CA AllFusionTM ERwin® Data Modeler to Embarcadero ER/StudioMigrating from CA AllFusionTM ERwin® Data Modeler to Embarcadero ER/Studio
Migrating from CA AllFusionTM ERwin® Data Modeler to Embarcadero ER/StudioMichael Findling
 
Learning Open Source Business Intelligence
Learning Open Source Business IntelligenceLearning Open Source Business Intelligence
Learning Open Source Business IntelligenceSaltmarch Media
 
Guidelines data cite_denmark_ver3
Guidelines data cite_denmark_ver3Guidelines data cite_denmark_ver3
Guidelines data cite_denmark_ver3DTU Library
 
Advanced Excel 2013 2016 Tips and Tricks by Spark Training
Advanced Excel 2013 2016 Tips and Tricks by Spark TrainingAdvanced Excel 2013 2016 Tips and Tricks by Spark Training
Advanced Excel 2013 2016 Tips and Tricks by Spark TrainingAhmed Yasir Khan
 

What's hot (18)

Generating XML schemas from a Logical Data Model (EDW 2011)
Generating XML schemas from a Logical Data Model (EDW 2011)Generating XML schemas from a Logical Data Model (EDW 2011)
Generating XML schemas from a Logical Data Model (EDW 2011)
 
ActiveWarehouse/ETL - BI & DW for Ruby/Rails
ActiveWarehouse/ETL - BI & DW for Ruby/RailsActiveWarehouse/ETL - BI & DW for Ruby/Rails
ActiveWarehouse/ETL - BI & DW for Ruby/Rails
 
Peoplesoft Basic App designer
Peoplesoft Basic App designerPeoplesoft Basic App designer
Peoplesoft Basic App designer
 
George McGeachie's Favourite PowerDesigner features
George McGeachie's Favourite PowerDesigner featuresGeorge McGeachie's Favourite PowerDesigner features
George McGeachie's Favourite PowerDesigner features
 
Tableau Developer
Tableau DeveloperTableau Developer
Tableau Developer
 
Introduction to database
Introduction to databaseIntroduction to database
Introduction to database
 
Excellence In Excel Presentation
Excellence In Excel PresentationExcellence In Excel Presentation
Excellence In Excel Presentation
 
People soft application-designer-practice-8.43
People soft application-designer-practice-8.43People soft application-designer-practice-8.43
People soft application-designer-practice-8.43
 
Sharing data in a multitenant architecture
Sharing data in a multitenant architectureSharing data in a multitenant architecture
Sharing data in a multitenant architecture
 
Internet Environment
Internet  EnvironmentInternet  Environment
Internet Environment
 
Merchant Product Datafeeds for Affiliates 101
Merchant Product Datafeeds for Affiliates 101Merchant Product Datafeeds for Affiliates 101
Merchant Product Datafeeds for Affiliates 101
 
Lightning talk at PG Conf UK 2018
Lightning talk at PG Conf UK 2018Lightning talk at PG Conf UK 2018
Lightning talk at PG Conf UK 2018
 
Analysis for office training
Analysis for office   trainingAnalysis for office   training
Analysis for office training
 
Migrating from CA AllFusionTM ERwin® Data Modeler to Embarcadero ER/Studio
Migrating from CA AllFusionTM ERwin® Data Modeler to Embarcadero ER/StudioMigrating from CA AllFusionTM ERwin® Data Modeler to Embarcadero ER/Studio
Migrating from CA AllFusionTM ERwin® Data Modeler to Embarcadero ER/Studio
 
I18n
I18nI18n
I18n
 
Learning Open Source Business Intelligence
Learning Open Source Business IntelligenceLearning Open Source Business Intelligence
Learning Open Source Business Intelligence
 
Guidelines data cite_denmark_ver3
Guidelines data cite_denmark_ver3Guidelines data cite_denmark_ver3
Guidelines data cite_denmark_ver3
 
Advanced Excel 2013 2016 Tips and Tricks by Spark Training
Advanced Excel 2013 2016 Tips and Tricks by Spark TrainingAdvanced Excel 2013 2016 Tips and Tricks by Spark Training
Advanced Excel 2013 2016 Tips and Tricks by Spark Training
 

Similar to Effective capture of metadata using ca e rwin data modeler 09232010

VSSML18. Data Transformations
VSSML18. Data TransformationsVSSML18. Data Transformations
VSSML18. Data TransformationsBigML, Inc
 
Overview of business intelligence
Overview of business intelligenceOverview of business intelligence
Overview of business intelligenceAhsan Kabir
 
OLAP Cubes in Datawarehousing
OLAP Cubes in DatawarehousingOLAP Cubes in Datawarehousing
OLAP Cubes in DatawarehousingPrithwis Mukerjee
 
Data+Modelling.pptx
Data+Modelling.pptxData+Modelling.pptx
Data+Modelling.pptxssuser58c23b
 
Data+Modelling.pptx
Data+Modelling.pptxData+Modelling.pptx
Data+Modelling.pptxssuser58c23b
 
VSSML17 L5. Basic Data Transformations and Feature Engineering
VSSML17 L5. Basic Data Transformations and Feature EngineeringVSSML17 L5. Basic Data Transformations and Feature Engineering
VSSML17 L5. Basic Data Transformations and Feature EngineeringBigML, Inc
 
Single View of the Customer
Single View of the Customer Single View of the Customer
Single View of the Customer MongoDB
 
1585625790_SQL-SESSION1.pptx
1585625790_SQL-SESSION1.pptx1585625790_SQL-SESSION1.pptx
1585625790_SQL-SESSION1.pptxMullaMainuddin
 
CDI-MDMSummit.290213824
CDI-MDMSummit.290213824CDI-MDMSummit.290213824
CDI-MDMSummit.290213824ypai
 
Data Standardisation in the Public Sector
Data Standardisation in the Public  SectorData Standardisation in the Public  Sector
Data Standardisation in the Public SectorDatabase Answers Ltd.
 
RahulSoni_ETL_resume
RahulSoni_ETL_resumeRahulSoni_ETL_resume
RahulSoni_ETL_resumeRahul Soni
 
Database Management Systems and SQL SERVER.pptx
Database Management Systems and SQL SERVER.pptxDatabase Management Systems and SQL SERVER.pptx
Database Management Systems and SQL SERVER.pptxsmg1723
 
BSSML17 - Basic Data Transformations
BSSML17 - Basic Data TransformationsBSSML17 - Basic Data Transformations
BSSML17 - Basic Data TransformationsBigML, Inc
 
UNIT - 1 Part 2: Data Warehousing and Data Mining
UNIT - 1 Part 2: Data Warehousing and Data MiningUNIT - 1 Part 2: Data Warehousing and Data Mining
UNIT - 1 Part 2: Data Warehousing and Data MiningNandakumar P
 

Similar to Effective capture of metadata using ca e rwin data modeler 09232010 (20)

VSSML18. Data Transformations
VSSML18. Data TransformationsVSSML18. Data Transformations
VSSML18. Data Transformations
 
Overview of business intelligence
Overview of business intelligenceOverview of business intelligence
Overview of business intelligence
 
MDM and Reference Data
MDM and Reference DataMDM and Reference Data
MDM and Reference Data
 
OLAP Cubes in Datawarehousing
OLAP Cubes in DatawarehousingOLAP Cubes in Datawarehousing
OLAP Cubes in Datawarehousing
 
Data+Modelling.pptx
Data+Modelling.pptxData+Modelling.pptx
Data+Modelling.pptx
 
Data+Modelling.pptx
Data+Modelling.pptxData+Modelling.pptx
Data+Modelling.pptx
 
VSSML17 L5. Basic Data Transformations and Feature Engineering
VSSML17 L5. Basic Data Transformations and Feature EngineeringVSSML17 L5. Basic Data Transformations and Feature Engineering
VSSML17 L5. Basic Data Transformations and Feature Engineering
 
Single View of the Customer
Single View of the Customer Single View of the Customer
Single View of the Customer
 
1585625790_SQL-SESSION1.pptx
1585625790_SQL-SESSION1.pptx1585625790_SQL-SESSION1.pptx
1585625790_SQL-SESSION1.pptx
 
CDI-MDMSummit.290213824
CDI-MDMSummit.290213824CDI-MDMSummit.290213824
CDI-MDMSummit.290213824
 
Data Standardisation in the Public Sector
Data Standardisation in the Public  SectorData Standardisation in the Public  Sector
Data Standardisation in the Public Sector
 
CRM Roadmap - Sample
CRM Roadmap - SampleCRM Roadmap - Sample
CRM Roadmap - Sample
 
PANKAJ SINGH-061.pptx
PANKAJ SINGH-061.pptxPANKAJ SINGH-061.pptx
PANKAJ SINGH-061.pptx
 
RahulSoni_ETL_resume
RahulSoni_ETL_resumeRahulSoni_ETL_resume
RahulSoni_ETL_resume
 
DBMS.pptx
DBMS.pptxDBMS.pptx
DBMS.pptx
 
Database Management Systems and SQL SERVER.pptx
Database Management Systems and SQL SERVER.pptxDatabase Management Systems and SQL SERVER.pptx
Database Management Systems and SQL SERVER.pptx
 
Kaizentric Presentation
Kaizentric PresentationKaizentric Presentation
Kaizentric Presentation
 
BSSML17 - Basic Data Transformations
BSSML17 - Basic Data TransformationsBSSML17 - Basic Data Transformations
BSSML17 - Basic Data Transformations
 
Mdm And Ref Data
Mdm And Ref DataMdm And Ref Data
Mdm And Ref Data
 
UNIT - 1 Part 2: Data Warehousing and Data Mining
UNIT - 1 Part 2: Data Warehousing and Data MiningUNIT - 1 Part 2: Data Warehousing and Data Mining
UNIT - 1 Part 2: Data Warehousing and Data Mining
 

More from ERwin Modeling

Zen of metadata 09212010
Zen of metadata 09212010Zen of metadata 09212010
Zen of metadata 09212010ERwin Modeling
 
Using ca e rwin modeling to asure data 09162010
Using ca e rwin modeling to asure data 09162010Using ca e rwin modeling to asure data 09162010
Using ca e rwin modeling to asure data 09162010ERwin Modeling
 
Staying relevant in todays changing dm environment 09282010
Staying relevant in todays changing dm environment 09282010Staying relevant in todays changing dm environment 09282010
Staying relevant in todays changing dm environment 09282010ERwin Modeling
 
Sneak peak ca e rwin data modeler r8 preview09222010
Sneak peak ca e rwin data modeler r8 preview09222010Sneak peak ca e rwin data modeler r8 preview09222010
Sneak peak ca e rwin data modeler r8 preview09222010ERwin Modeling
 
Monetizing data management 09162010
Monetizing data management 09162010Monetizing data management 09162010
Monetizing data management 09162010ERwin Modeling
 
Mastering your data with ca e rwin dm 09082010
Mastering your data with ca e rwin dm 09082010Mastering your data with ca e rwin dm 09082010
Mastering your data with ca e rwin dm 09082010ERwin Modeling
 
Integrating data process a roundtrip modeling using e rwin data modeler_erwin...
Integrating data process a roundtrip modeling using e rwin data modeler_erwin...Integrating data process a roundtrip modeling using e rwin data modeler_erwin...
Integrating data process a roundtrip modeling using e rwin data modeler_erwin...ERwin Modeling
 
Deciding to go cloud 09212010
Deciding to go cloud  09212010Deciding to go cloud  09212010
Deciding to go cloud 09212010ERwin Modeling
 
Data modeling for the business 09282010
Data modeling for the business  09282010Data modeling for the business  09282010
Data modeling for the business 09282010ERwin Modeling
 
Creating enterprise standards 09302010
Creating enterprise standards 09302010Creating enterprise standards 09302010
Creating enterprise standards 09302010ERwin Modeling
 
Ca e rwin state of the union 09082010
Ca e rwin state of the union 09082010Ca e rwin state of the union 09082010
Ca e rwin state of the union 09082010ERwin Modeling
 
Ca e rwin modeling global user communities_09232010 - webcast
Ca e rwin modeling global user communities_09232010 - webcastCa e rwin modeling global user communities_09232010 - webcast
Ca e rwin modeling global user communities_09232010 - webcastERwin Modeling
 
10 things to avoid in data model 09242010
10 things to avoid in data model 0924201010 things to avoid in data model 09242010
10 things to avoid in data model 09242010ERwin Modeling
 
5 physical data modeling blunders 09092010
5 physical data modeling blunders 090920105 physical data modeling blunders 09092010
5 physical data modeling blunders 09092010ERwin Modeling
 
Optimizing the design of your data warehouse 09222010
Optimizing the design of your data warehouse 09222010Optimizing the design of your data warehouse 09222010
Optimizing the design of your data warehouse 09222010ERwin Modeling
 

More from ERwin Modeling (15)

Zen of metadata 09212010
Zen of metadata 09212010Zen of metadata 09212010
Zen of metadata 09212010
 
Using ca e rwin modeling to asure data 09162010
Using ca e rwin modeling to asure data 09162010Using ca e rwin modeling to asure data 09162010
Using ca e rwin modeling to asure data 09162010
 
Staying relevant in todays changing dm environment 09282010
Staying relevant in todays changing dm environment 09282010Staying relevant in todays changing dm environment 09282010
Staying relevant in todays changing dm environment 09282010
 
Sneak peak ca e rwin data modeler r8 preview09222010
Sneak peak ca e rwin data modeler r8 preview09222010Sneak peak ca e rwin data modeler r8 preview09222010
Sneak peak ca e rwin data modeler r8 preview09222010
 
Monetizing data management 09162010
Monetizing data management 09162010Monetizing data management 09162010
Monetizing data management 09162010
 
Mastering your data with ca e rwin dm 09082010
Mastering your data with ca e rwin dm 09082010Mastering your data with ca e rwin dm 09082010
Mastering your data with ca e rwin dm 09082010
 
Integrating data process a roundtrip modeling using e rwin data modeler_erwin...
Integrating data process a roundtrip modeling using e rwin data modeler_erwin...Integrating data process a roundtrip modeling using e rwin data modeler_erwin...
Integrating data process a roundtrip modeling using e rwin data modeler_erwin...
 
Deciding to go cloud 09212010
Deciding to go cloud  09212010Deciding to go cloud  09212010
Deciding to go cloud 09212010
 
Data modeling for the business 09282010
Data modeling for the business  09282010Data modeling for the business  09282010
Data modeling for the business 09282010
 
Creating enterprise standards 09302010
Creating enterprise standards 09302010Creating enterprise standards 09302010
Creating enterprise standards 09302010
 
Ca e rwin state of the union 09082010
Ca e rwin state of the union 09082010Ca e rwin state of the union 09082010
Ca e rwin state of the union 09082010
 
Ca e rwin modeling global user communities_09232010 - webcast
Ca e rwin modeling global user communities_09232010 - webcastCa e rwin modeling global user communities_09232010 - webcast
Ca e rwin modeling global user communities_09232010 - webcast
 
10 things to avoid in data model 09242010
10 things to avoid in data model 0924201010 things to avoid in data model 09242010
10 things to avoid in data model 09242010
 
5 physical data modeling blunders 09092010
5 physical data modeling blunders 090920105 physical data modeling blunders 09092010
5 physical data modeling blunders 09092010
 
Optimizing the design of your data warehouse 09222010
Optimizing the design of your data warehouse 09222010Optimizing the design of your data warehouse 09222010
Optimizing the design of your data warehouse 09222010
 

Recently uploaded

Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomCzechDreamin
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlPeter Udo Diehl
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Julian Hyde
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...Product School
 
In-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT ProfessionalsIn-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT ProfessionalsExpeed Software
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityScyllaDB
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...Product School
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutesconfluent
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Alison B. Lowndes
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Product School
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Product School
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoTAnalytics
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeCzechDreamin
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfChristopherTHyatt
 
ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupCatarinaPereira64715
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024Stephanie Beckett
 

Recently uploaded (20)

Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
 
In-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT ProfessionalsIn-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT Professionals
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 

Effective capture of metadata using ca e rwin data modeler 09232010

  • 1. Effective Capture of Metadata Using CA ERwin Data Modeler Metadata –Data gets meaning.
  • 2. Abstract • In any data centric environment either Data warehouse or an OLTP data environment ,vital information anybody looks for is the purpose and content of the tables and columns. Its metadata of the data, provides more insight about the structure. In many organization, the lack of metadata has lead to redundant definition of table and columns , ignorance of real capability of your data centric system, inability to define standards and build the knowledge layer for the business. In the case of data warehouse its vital to capture the source and transformation rules along with dimensional model as it will help fixing incorrect mappings early in the life cycle, effective communication to ETL team and store the ETL rules close to the data model. Without metadata ,it will lead to individual's interpretation of data just like blind folded touching the elephant. In this webinar we will discuss about the various flexible features provided by CA ERwin Data Modeler for data warehouse and relational model. PAGE 2
  • 3. Speaker Bio • Sampath Kumar brings 11 years of experience in implementing small, medium and large scale data centric environments (both relational and data warehouse ) using CA ERwin Modeling suite of products. He is currently working for Infosys Technologies Limited as Technology Architect in their DW/BI practice group. Prior to that he was working with American Express Credit Cards as Sr System Analyst for their Worldwide Risk Information Management group. In all his experience he has worked extensively in various database products ,BI tools ,data modeling and related products offered by CA such as CA ERwin Data Modeler, CA ERwin Model Validator,CA ERwin Model Manager and CA ERwin Data Profiler. PAGE 3
  • 4. Agenda • Not going to focus on the known fundamentals and data jargons • Effective capture of metadata in Data warehouse environment – Case study using Customer_Dim • Other Flexible Options to capture metadata into data model – Using an example. PAGE 4
  • 5. Effective capture of metadata in Data warehouse environment Case study using Customer_Dim
  • 6. Problem Statement In any data warehouse development project, some of the major challenges include • Effective capture of metadata information in data model such as data source ,transformation, enrichment and data synchronization rules etc. • Keeping data model in synch with changing ETL rules and vice versa i.e. keeping ETL rules close to DW Data model (blueprint of your DW data) • Early identification of incorrect ETL mappings in the complete lifecycle. PAGE 6
  • 7. Problem Statement contd… • Effective communication of captured metadata information by data modeler to other teams such as ETL PAGE 7
  • 8. Background 3 Important pieces of information: • Source of data • Transformation rules-The method in which the data is getting extracted, transformed and loaded • Frequency: The frequency and timing of data warehouse updates. PAGE 8
  • 9. CA ERwin Features CA ERwin Data Modeler supports the following salient features to capture the metadata information effectively. • Data warehouse Sources Dialog • Columns Editor • Data Movement Rules Editor PAGE 9
  • 10. Customer_Dim Snapshot Communication – customer_SKID Email Address – snapshot_Begin_Date Phone – snapshot_End_Date Fax – current_ind Segmentation • Basic Information Shopping – Customer name Behavior – Customer Date of Birth – Driving License • Address – Mailing Address – Physical Address PAGE 10
  • 13. Make it Dimensional Model PAGE 13
  • 14. Make it Dimensional Model… PAGE 14
  • 16. Data warehouse Source Selector PAGE 16
  • 18. Data warehouse Sources The “Import other” provides three options to import the table structure • Flat File • Database/Script • Model Manager PAGE 18
  • 19. Importing table from CA ERwin® Model Manager Customer_Address customer_id (FK) mailing_address_line1 mailing_address_line2 mailing_city mailing_state mailing_county mailing_country physical_address_line1 Customer physical_address_line2 physical_county customer_id physical_city customer_first_name physical_state Opens Model Mart customer_last_name physical_country dob Library driving_license_nbr driving_license_state Behavioral_Segment behavioral_segment_nbr behavioral_segment_name Customer_Segmentation customer_id (FK) behavioral_segment_nbr (FK) Shopping_Segment shopping_segment_nbr (FK) shopping_segment_nbr shopping_segment_name PAGE 19
  • 22. Data warehouse Source Selector. Multiple sources can be added Transformation and business rules can be PAGE 22 added here.
  • 23. Source as Flat File PAGE 23
  • 24. ETL Mapping Template-using Data Browser PAGE 24
  • 27. Generate the Metadata Report PAGE 27
  • 28. ETL Mapping Sheet PAGE 28
  • 31. Attaching table to rule. PAGE 31
  • 32. Export as Report PAGE 32
  • 33. Other Flexible Options to capture metadata into data model Using simple example
  • 34. Import from MS Excel Using simple example
  • 35. Metadata Capture from MS Excel • When it would be useful – Import the definitions available already into data model – Import the definitions from business stakeholders for key columns to avoid wrong interpretation. • Step 1: Store the model locally in the hard disk • Step 2: Use the excel sheet “Import Definitions” or VBA macro provided by CA . • Step 3 :Import the metadata into the model by running VBA code. PAGE 35
  • 38. In this format PAGE 38
  • 39. Final Step Open the first sheet and click on “Update Entity Defns” which will update the definitions written for that particular table into the data model. Similarly click on the “Update Attribute Defns” which will update the attribute definitions. Note: • Keep the data model closed otherwise you will get error that it’s open. • Make sure table and column names are exactly same as in the data model. • It’s not only for business people but also for the data modelers who can enter the definitions in MS Excel and get the approval from the business or data management team, then it can uploaded separately using this utility. PAGE 39
  • 40. Capture metadata in Data Browser.
  • 41. Metadata Capture using Data Browser PAGE 41
  • 42. Conclusion • The metadata information such as “Data Source”, “Transformations rules” and “Data Movement rules” are very important for any Data warehousing efforts and it’s very critical to capture the correct information. • Metadata from data management standpoint , reduces considerable amount of time while consolidating the attributes or entities or databases during acquisition or merger. • Knowing the importance of metadata for the data model ,CA ERwin has provided these flexible options which can be leveraged to make the data model & data more meaningful. PAGE 42
  • 43. Questions? In case of any additional questions you can reach me in Sampath_Kumar01@infosys.com Sampath.k.kumar@gmail.com PAGE 43