MCE GeoProcessing Services for ADM(IE): Self Validation of Spatial Data Input from DND Bases
•Download as PPTX, PDF•
0 likes•2,808 views
The Department of National Defence (DND) is developing a central repository for land, building, and infrastructure data. This a joint project between the Assistant Deputy Minister Infrastructure and Environment (ADMIE) and the Mapping and Charting Establishment (MCE). This project involves managing DND real property and other spatial data provided by DND bases and wings across Canada through a unique, integrated and standardized Real Property Spatial Data Warehouse (RPSDW), hosted at MCE, containing a SQL Server database. Data provided by DND bases and wings must meet standards defined and documented by ADMIE, in terms of data formats accepted (GeoMedia MDB, ArcGIS FGDB, MapInfo MIF or AutoCAD SDF), geometry, schema and attribute data types, domains and accepted values for each feature class. An FME workspace and an equivalent tool contained within an ArcGIS Data Interoperability Toolbox were created to provide the geotechs from DND Bases using ArcGIS Data Interoperability or FME Desktop with a toolset, delivered together with a user’s guide, that allows them to perform a self-validation of the DND real property and other spatial data before these data is sent to MCE to be loaded into the RPSDW SQL Server database.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to MCE GeoProcessing Services for ADM(IE): Self Validation of Spatial Data Input from DND Bases
Similar to MCE GeoProcessing Services for ADM(IE): Self Validation of Spatial Data Input from DND Bases (20)
More from Safe Software
More from Safe Software (20)
Recently uploaded
Recently uploaded (20)
MCE GeoProcessing Services for ADM(IE): Self Validation of Spatial Data Input from DND Bases
- 1. ADM(IE) Real Property Spatial Information Management MCE GeoProcessing Services for ADM(IE): Self Validation of Spatial Data Input from DND Bases FME World Tour 2015 – Ottawa 15 April 2015 Martin de Zuviria (GIS Architect, MCE) Paul Churcher (DAES IMS GIS Coordinator, ADMIE)
- 2. 2 Introduction ADM(Infrastructure and Environment) Mission is to provide IE functional leadership and services to enable effective, efficient and sustainable management of DND infrastructure and environment in support of CAF missions and Departmental programs. ADM(IE) will have sole custodial authority by 01 Apr 2016 over all DND Real property (RP) assets. Responsible for: • Over 2 million hectares of owned/leased land • Over 20,000 buildings • Over 13,000 works items (docks, jetties, runways, etc) • Real property nationwide in every province and territory
- 3. 3 A proof-of-concept has been completed that shows the benefits of standardized and centralized real property spatial data. • A proof-of-concept collaboration with MCE, a centre of excellence for spatial data, has proven that open standards applied to a Real Property Spatial Data Warehouse (RPSDW) promotes integration and interoperability of CAD/BIM/GIS spatial data. • The data model supports portfolio-wide analysis and query. • Next steps − Expand the data model to include all real property spatial data. − Operationalize the warehouse through a graduated data maturity model. − Collaborate with bases/wings on data management best practice for keeping the warehouse current. − Provide documented schema for the integration of particular data to SAP and vice versa. − Provide information service to ADM(IE) stakeholders through web services, data delivery and integration. Real Property Spatial Information Management D A T A C E N T R A L I Z A T I O N
- 4. 4 Integration of spatial data between the RPSDW and corporate systems enables single-entry of data. • SAP holds master data (attribution) on built and natural assets required for mapping and site engineering. • Spatial data holds point-of-truth information such as measurement, age and geo-location for other corporate systems. • Integrating this attribute information to/from SAP will eliminate the current dual entry process. • Information will be updated automatically as changes are reflected in the originating systems. • Information required to provide spatial services will be integrated to the RPSDW from SAP. • Other corporate systems can benefit from data integration such as project delivery, plant maintenance, facilities management and energy auditing. Real Property Spatial Information Management D A T A I N T E G R A T I O N
- 5. 5 ADM(IE) manages real property spatial spatial data for site infrastructure & utilities, buildings and land registry which includes: • Civil Works Models • CAD Site and Building Plans • Building Information Models • GIS Views • Photographs • Land Imagery • Terrain Models • Environmental • Boundary Survey • Site Survey • LiDAR Survey Key attribute data for project delivery and ongoing real property management is exchanged between SAP and the spatial warehouse, enhancing overall accessibility. RP Spatial Data - Types
- 6. 6 RP Spatial Data Staged Maturity Land Registry The RPSDW data maturity model will provide a graduated, predictable transition from a decentralized to a centralized and integrated RP spatial data environment.
- 7. 7 RP Spatial Data Staged Maturity - Example
- 8. 8 RP Spatial Data Staged Maturity - Example
- 9. 9 9 Self Validation of Spatial Data Input from DND Bases Summary The FME workspace presented here is part of a joint project between the Mapping and Charting Establishment (MCE) and the Assistant Deputy Minister for Infrastructure and Environment (ADM(IE)). This project involves managing DND real property and other spatial data provided by DND Bases across Canada at MCE through a unique, integrated and standardized Real Property Spatial Data Warehouse (RPSDW) containing a SQL Server database. Data provided by DND Bases must meet the standards defined and documented by ADMIE, in terms of data formats accepted (GeoMedia MDB, ArcGIS FGDB, MapInfo MIF and AutoCAD SDF), schema and attribute data types, domains and accepted values for each feature class. An FME workspace and an equivalent tool contained within an ArcGIS Data Interoperability Toolbox were created to provide the GeoTechs from DND Bases using ArcGIS Data Interoperability or FME Desktop with a toolset, delivered together with a user’s guide, that allows them to perform a self validation of the DND real property and other spatial data before these data is sent to MCE to be loaded into the RPSDW SQL Server database. The FME workspace presented here outputs all errors, warnings and non-compliance issues to a destination FGDB, and is followed at MCE by other workspaces to ensure proper data management of the SQL Server database and communication with DND Bases.
- 10. ADMIE Standards: Feature Classes, attributes and properties defined and documented
- 11. 11 Overview of the Self-Validation ETL
- 12. Navigator: Input, Output and Bookmarks. Standards are used to validate schema compliance of any feature class input against them
- 13. 13 A similar sequence of bookmarks implemented for the P_Building feature class is being applied to the other 26 selected feature classes
- 14. 14 Standard Attribute Validation Custom Transformer (this validation is common to all feature classes)
- 15. Left and right screen captures of the P_Building schema and attribute validation bookmark with annotations indicating the type of validation implemented for a specific transformer
- 16. 16 Geometry Validation Custom Transformer
- 17. 17 Comparison between input from Bases and data on MCE Warehouse Bookmark (buildings geometry and all attribute values defined in the standard schema are compared)
- 18. 18 Errors and Warnings are reported in two FGDBs: In the first FGDB a single feature may be output several times showing only one error at a time (one error – one record) ; On the second FGDB a single feature may be output only once, reporting all errors together
- 19. 19 Overview of the QA_ETL_Aggregated_Output FGDB
- 20. 20 Data Inspection The example below shows on the ‘Table View’ that a ‘PRIN_TYPE’ value is missing for a building , that there are duplicate ‘Prin_No’ values, ‘Type’ values have not been provided, together with other 2819 errors identified for this dataset. These errors should be corrected in the Bases before loading these data to MCE warehouse
- 21. 21 Data Inspection The example below shows some of the errors reported. 15 errors are reported for the feature highlighted
Editor's Notes
- Standards were documented in an Excel Spreadsheet by Charlie Jessome, from ADMIE. Required Feature classes were defined (highlighted in yellow), together with their attributes and accepted values.
- FME Desktop 2013 SP3, from Safe Software, was used to build the Self-Validation ETL in May 2014 in order to allow to “copy and paste” all contents of this workspace to an ArcGIS Data Interoperability ETL
- As it may be seen on the left, an output feature class is generated for each input feature class every time errors and warnings are detected for each particular class. Feature classes of the output FGDB have the same name and schema of the input feature classes, but with the addition of the string ‘_QA’ at the end of each input feature class’ name (e.g. ‘P_Building_QA’) and with two new fields: ‘ETL_timestamp’ and ‘Type_Error’. These fields show the date and time when each individual feature was processed and the description of the type of error, warning or non-compliance issue found (over 160 types of errors and warnings have been typified and may be reported here). When this FME workspace is run at MCE, additional feature classes are generated with the string ‘_WH’ at the end (e.g. ‘P_Building_WH’). These features contain the input features and the result of comparing each individual input feature against features of the same feature class already loaded into the RPSDW SQL Server. These output feature classes include two new fields: ‘ETL_timestamp’ and ‘TypeMatchWH’. The last field shows one of the three following values: “Identical Geometry is NOT found in Warehouse”, “Identical Geometry and Different Attribute values are found in Warehouse” or “Identical Geometry and Attribute values are found in Warehouse”. This Self Validation FME workspace is followed at MCE by other workspaces to ensure proper data management of the SQL Server database and communication with DND Bases. These workspaces are not presented here. As it may be see on the right, bookmarks have been built for each feature class, to allow to refocus the main window on any feature after clicking on them and to allow the user to enable/disable all objects included in a bookmark (this can be very handy if the user would like to run the ETL for one or more selected feature classes)
- At he ‘Check Point Charlie’ the first bookmark on top (light green), it is first verified that input data provided in one of the formats accepted does not contain multi-part features (green colored custom transformer) and then it is verified that all input data lies within the corresponding DND Base data reporting area (this area is equal or larger than the respective DND Base legal area). At the ‘P_Building Non-Standard Attribute Validation Area’ bookmark, the defined schema and the accepted values for non-standard attributes (i.e. the attributes and accepted values that are unique to one specific feature class) are validated. At the ‘P_Building Standard Attribute Validation’ bookmark, the accepted values for standard attributes (i.e. the attributes and values that are common to many or all feature classes) are validated . At the ‘P_Building Geometry Validation’ bookmark, selected issues in input features are detected (each input feature is processed individually). These selected issues are those relevant to obtain OGC compliance, such as self-intersections, duplicate points and several others. At the ‘P_Building of input from DND Base with data existing on MCE Warehouse’ bookmark, a comparison is performed between each individual feature of the input P_Building data against the P_Building data already loaded into the MCE warehouse.
- Annotations are provided at different parts of the ETL in order to provide the user with additional information of a particular testing carried out by a transformer
- Since the data loaded to MCE Warehouse may be used to publish mapping services, this geometry validation (meeting Open Geospatial Consortium standards) and subsequent data repairs are required to prevent the discard of failed features at the time of publishing these services.
- This comparison generates an output feature class indicating for each feature whether an object with identical geometry and location is already loaded to MCE warehouse, and whether two matched objects have identical or different attribute values. This comparison is essential to prevent duplicated data and to manage data, changing the status of features updated.
- Errors, warnings and non-compliance issues are sent first to a visual interactive interface (FME Data Inspector) and then, if requested by the user, all errors and issues detected can be written to a destination FGDB. In the screen capture above, results for the ‘P_Building’ feature class are inspected. For all features, the location of the input data is reported, together with a time-stamp and a description of the type or error or warning found, like in the case of the examples indicated in the sub-title. More that 160 individual types of errors and warning have been typified and may be reported here.
- All errors and issues detected can be written to a destination FGDB in a new run of this ETL. Users with some basic knowledge of FME Desktop may request to send the output data to a format of their choice.