The Dark Art of Data Migration
•
2 likes•763 views
Presented at the SPATIALinfo conference in Denver on 9th June 2011.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (8)
Similar to The Dark Art of Data Migration
Similar to The Dark Art of Data Migration (20)
Recently uploaded
Recently uploaded (20)
The Dark Art of Data Migration
- 1. The Dark Art of Data Migration Presented by Graham Morgan MD Spatial Consultants Ltd
- 2. Tuning in …
- 3. 3 Expert independent advice
- 4. Is it that Difficult? Drawings with layout Features split Implied data (color) Multiple sources Choreography 4 Missing data Unstructured data Spatial formats Variability within Different meanings Inaccuracies
- 5. Target - SpatialNet All data is stored in an Oracle database 5 Data is: Feature based Highly organized Classified / Categorized Consistent Data integrity rules Standardized
- 6. Target - SpatialNet All data stored in Oracle database 6
- 7. 7 SpatialInfo data model
- 8. 8 Where do we start?
- 10. Data Challenges Where is the feature record? Different data sources / formats No single business identifier for features Which attribute values are correct? Multiple copies, alternatives Do we have all the attributes? Do we have connectivity information? Infrastructure Level Fiber level 10
- 11. System Challenges It isn’t just a data migration… System footprints Business functions Data (CRUD) Access security 11 Data migration is a component of a business improvement programme
- 12. Data Migration Process Requirements Estimate / Quote Design Implement Test Deploy Celebrate … 12
- 13. Data Migration Process 13
- 14. Data Migration Process - Prepare Map existing business functions to systems Existing / Planned Inventory data Data to migrate to SpatialNet / other systems? Prepare data documentation Develop baseline tests Feature counts Network trace tests Make local corrections 14
- 15. Data Migration Process - Develop Exchange data in situ (lossless) Load directly ‘As Is’ into Oracle Pre-process for load option Re-project Inventory / study / profile / explore / specify Migrate to intermediate model Validate Migrate to SpatialNet Validate Correct data at source … Repeat … 15
- 17. Profile / Explore Defend against duplicates – ensure unique keys Missing features Attribute value patterns / consistency Mandatory, Unique, Domain values Logical consistency Geometry types & consistency Connectivity: inferred vs. recorded 17
- 19. ESRI ArcGIS
- 20. MapInfo
- 21. Talend / Open Source
- 24. Promote to SpatialNet Standard load procedure Dictionary mapping Apply symbol rotation, generate annotation Validation reports 20
- 25. Shortcuts Pitfalls Data analysis in Oracle- rich toolset- neutral model Correct data at source- with familiar tools Train users on SpatialNet during migration- with familiar data Script everything Automate some clean up Square Peg / Round Hole- start migration before defining scope & context Migrate in isolation Let the consultants figure it out Plan for single migration 21
- 26. Thank You Graham MorganSpatial Consultants Ltd gmorgan@spatialconsultants.comwww.spatialconsultants.com
Editor's Notes
- SPATIALinfo conference, Denver, CO, 9th June 2011Planning for the migration of digital spatial data into SpatialNet
- NW England: Lake District, Britain’s Energy Coast
- GIS consulting for clients worldwideSpatial systems & data integrationData migrationInformation strategyApplication development
- Why should it be difficult – the source data is in a system that has probably successfully supported the business for a number of years.
- Relational tablesAttributes with data typesGeometry also stored in native OracleIntegrity rules to enforce consistencyStandards based – other programs can access it.Structure means that the data can be queried with confidence to support business decision making.
- All data, including geometries, can be accessed by Oracle tools – and many 3rd party tools.Geometry can be queried and manipulated through SQL.
- The SpatialNet data model is complex as it supports many purposes.
- Data held across different systems in different formatsHopefully with some form of documentationProbably some duplication with differentiationAs successful business applications often get more complex over time (as they take on additional responsibilities) then it becomes important to fully understand the match (and gaps) between the source and target data.Drawings are only fully intelligible to the trained human eye.
- Data associated by proximity on a drawingExtended entity data, blocks, etc.GIS systems can also be configured to store data as annotation (i.e. unstructured) ….
- Fiber connectivity is not represented spatially – it must be provided by attribution.
- Does the new system provide all the business functions of the existing system?Does all of the existing data map neatly into the model of the new system?Can the new system be accessed by all of the people that currently access the existing system?Existing system is likely to have evolved over time and may have assumed some additional responisbilities.An architect is required to have overall visibility of the entire system replacement project.The data migration is only one project within a larger programme.
- Dream on….Activities are generally correct, they are all required – but must expect several iterations to refine.
- Iterations generally get fasterFirst is the longest as there is more to put in place; later iterations are just refinements; final is very fast
- Magicians are really illusionists – there is always an explanation – Preparation is key.Master data may be in other systems!Must understand the CRUD matrix for the data being migrated – in which applications will each attribute be maintained? And what integration points are required (inbound and outbound) to maintain the data in the new system.Correct data errors in the existing system – greater levels of expertise and efficiency with the familiar system.Prepare for acceptance testing – how will we know when the data has been migrated correctly?What goes in is what came out.How to exchange data between source system/s and migration environment?
- Investigate Oracle loading options for spatial data.Standardize DictionariesSymbol Orientation, Annotation positioning, Feature placement (e.g. dist between cables)Pre-processing – minimum changes to prepare the data for loading into Oracle using the selected tools. E.g. drop constraints, change reserved words, protect Identity columns, etc.
- Load source into Oracle environment with minimal processing – use Oracle-based tools to process data with consistency.Bulk of the work happens in migrating from the staging environment to the neutral model.
- Geometry values – recorded Vs calculatedOracle may well provide a better environment for data profiling than source applications
- Every migration is unique.Need to be flexible with use of the most appropriate toolsYet try to be consistent in tool use.Script the migration as far as possible to enable efficient iterations.
- Neutral model is relatively simple, easy to read and review.
- Tom Ward presentation on afternoon of June 9th.
- Support for Just in Time training.Script as much of the migration as possible – there will be several iterations; there may be some time between them.Consider option of automated data cleanup (e.g. data formats for dates, phone numbers, etc.) if time allows.