The session discusses a case for best practice in FME Workspace design and process optimisation from Ornance Survey Great Britain . Presented by David Eagle of 1Spatial.

1. FME Process Optimisation at the Ordnance
Survey Great Britain
An exercise in best practice
david.eagle@1spatial.com
Principal Consultant
FME Certified Professional & Trainer
1Spatial

2. Agenda
• The Ordnance Survey
• The Object Editor Adapter
• Project Goals
• FME as a Test Harness
• Deploying ‘Best Practice’ in FME
• Tips and tricks
• Project Benefits
2

3. Ordnance Survey - Then
3
• The original ‘Board of Ordnance’, MOD of the day
• Started survey work in the late 1700s
• Comprehensively mapped the south coast to be able to effectively
repel invasion
• http://goo.gl/02tVFZ
Lands End,
Mount's Bay,
Cornwall
*

4. Ordnance Survey - Now
4
• Self financing part of the UK government
• £120-million-a-year civilian organisation
• At the forefront of the digital economy
• Remit covers Great Britain (England, Scotland
and Wales)
• 243,241 sq km / 93,956 sq miles
• Integrated data capture for field surveyors,
HQ-based editors, and external suppliers
*

5. Geospatial Data Management
System (GDMS)
• Seamless data maintenance solution
• Intelligent and integrated maintenance data model
• Allows separation of product data
• Includes automated data validation
• Platform for new product development
• 650+ data editors
• 500+ million features
• 4,000 data maintenance jobs per day
*

6. GDMS – Production
*

7. GDMS – Production Architecture
• Includes best of breed COTS components from
• 1Spatial
• BAE Systems
• ESRI
• Intergraph
• Microsoft
• Oracle
• Safe Software
• Snowflake Software
• Scalable infrastructure
• Disconnected editing thanks to the ‘Object Editor Adapter’
*

8. Object Editor Adapter
8
• Transforms data into a structure usable in the Object Editor map
application
• FME underpins the transformation
• Schema is manipulated into ‘understandable’ structure from the
database
GML
OEA
(FME)
PGDB
*

9. Project Goals
9
1. Document process and make maintenance practical
• Schema changes pending
• Component needed to be better understood
2. Upgrade process to FME 2013 (look beyond 2013)
• Built with FME 2009
• Utilise recent FME enhancements
3. Improve efficiency & performance
• Remove redundancy

10. “If it ain’t broke…”
10
• Huge amount of change in FME since 2009
• Attribute management particular interest
• OS needed to unpick the workspace to be
able to update it anyway
• Workspace opening time was slow,
impacting batch run time and FME Server
– 3 minutes
– c.60mb (most *.fmw don’t exceed a few 100kb)
• Quite a few transformers…

11. Transformers, transformers
everywhere
11
• 943
– Custom Transformer(s) = 255
– AttributeRenamer = 211
– Tester = 150
– AttributeRemover = 87
– AttributeCreator = 51
– ListExploder = 72
– CaseChanger = 21
– ListElementCounter = 18
– AttributeSplitter = 12
– AttributeSetter = 10
– ListIndexer = 9
– AttributeCopier = 8

12. What is a ‘Test Harness’?
12
In software testing, a test harness or automated test framework
is a collection of software and test data
configured to test a program unit
by running it under varying conditions
and monitoring its behaviour and outputs.
Wikipedia

13. Using FME as a ‘Test Harness’
13
• The process had to change but the data must not!
• FME to check the old process against the new
• Controlled via a spreadsheet, a WorkspaceRunner and Published
Parameters
GML
PGDB
OLD
PGDB
NEW
Detect
Change
Failures:
Spreadsheets and
Geometry
*

14. 14

15. Dynamic Schema
15
• Introduced around FME 2009
– Existing process didn’t use it
• Enable ‘Dynamic’ schema mode
– Whatever schema is read, gets written
– Decouple schema from format
• http://blog.safe.com/2010/02/fmeevangelist54/

16. Dynamic Schema
16
• 96 input layers can be reduced to 1!
– Magic Transformer = FeatureTypeExtractor
– Workspace infinitely less daunting

17. Reduce Transformer
Duplication
17
• Helped by Dynamic Schema Mode
• Try not to duplicate operations

18. Reduce Transformer
Duplication
18
• ‘Like’ operations carried out on all features
– Filter features off to unique processes when needed
• 20% of data had a ‘Level’
– 21 CaseChangers used to apply UPPERCASE
– Replaced with 1 StringCaseChanger
• 69 Custom Transformers with embedded Python used to
format Date attributes
– All features passed through a StringReplacer to remove hyphens
• 2013-01-01 to 20130101

19. Reduce Transformer
Duplication
19

20. Custom Transformers
20
• What?
• A container for many regular transformers
• Allows a common set of tasks to be reused in 1 or many
workspaces
• When exported becomes an *.FMX file for sharing

21. • The FME Store was introduced in 2012
– 2014 FME Store mature with 130+ transformers (and other resources)
• http://fmestore.safe.com
• http://fmepedia.safe.com/articles/Samples_and_Demos/Custom-Transformer-
KMLDiagrammer
• It’s increasingly likely your workspaces will contain Store resources
• FME now auto connects & downloads any resources required
Custom Transformers
21

22. • Support for versioning
• Ports are not disconnected when renamed
– http://blog.safe.com/2013/12/fmeevangelist121/
• Embed/link all instances of a given transformer
• Manage input and output schema on Custom Transformers
Custom Transformers 2014
22

23. • 14 unique custom transformers used 255 times
• PGDB had constraints, many tables did not allow null values
– Custom transformers tested and removed null attributes
– Thanks to Generic mode expose constrained attributes and pass through a
PythonCaller to remove the null attributes
• FME 2014
Use Custom Transformers
Effectively
23

24. Schema Mapping
24
• Attribute names are hardcoded into 211 AttributeRenamers
in the workspace
– Changes to target schema means attribute hunting!
– Biggest challenge for ongoing maintenance
• 1x SchemaMapper replaces all AttributeRenamers
– Non-FME user can alter the process by editing
AttributeMapping.csv

25. Documentation
25
• Always document your Workspace
• The OEA process was documented
– But didn’t include detailed explanations ‘why’
• You should be able to read a Workspace and understand it
– Highlight dependencies and use XML notation for naming

26. • In 2014 canvas annotations taken to the next level
Documentation – Self contained
26

27. Before and After
27

28. Benefits
28
Before After Improvement
Time to open (from local disk) 2mins 12secs 31secs 1min 41secs
File size 56mb 13.8mb 75% reduction
Lines of machine written code 450,332 111,334 75% reduction
Number of transformers 943 336 64% fewer
Sample run time 1hr 7mins 48mins 19mins
• Current, documented and ‘owned’ by the Ordnance Survey
• Robust production extraction routine
• Simpler to maintain and change
– Some schema changes possible without opening FME!

29. 29
david.eagle@1spatial.com
Principal Consultant
FME Certified Professional & Trainer
1Spatial