In our presentation, Michaël and I will present our solution to tackle a particular problem. Michaël will focus on explaining the specific requirements of his office, while I will focus on how we solved the problem. The client wanted to migrate specific data from one visualization platform to another platform in an automated way. The platform to migrate from communicating through a data streaming JSON message platform, with all new information being added to this platform (about 20k-30k messages daily). Relevant messages needed to be filtered out from this data stream and transformed in a specific way, as defined by an Excel spreadsheet. This spreadsheet went beyond a simple one-to-one transformation and defined which objects should be migrated and which attributes should be inherited. For instance, the spreadsheet even defined if an attribute should get a value from an external table based on a certain join condition. This was just one example of the complex mapping we were facing. Additionally, the spreadsheet needed to be dynamically adjustable, ensuring that it was generic and future-proof as the source system is in full expansion and offers customizable data models for each of its object classes. This means that the Excel mapping sheet for migration was not yet complete and would be expanded and modified in the future at minimal cost. We solved this problem by using a set of five workspaces that were all connected with each other through JobSubmitters and automations on FME Server. These ran on a scheduled basis. The mapping workspace dynamically analyzed which information needed to be migrated and how, based on the external lookup table. The complex mapping was solved by making custom transformers for the task. The message queue was also cleaned in the process, as well as producing multiple output data products, each fit for the customer’s needs, only after running validation tests to allow error handling and appropriate logging.

1. Beyond the SchemaMapper

2. Presenters
The
Peak
of
Data
Integration
20
23
Matthias
Holemans
FME Consultant
Nordend
Michaël
Ferré
GIS Analyst
District09

3. The
Peak
of
Data
Integration
20
23
Agenda
1. Topic
2. Specific problem
3. Solution
4. Conclusion

4. The
Peak
of
Data
Integration
20
23
Introduction

5. The
Peak
of
Data
Integration
20
23
Topic
Using simple spread sheets to extract specific data from a
big data platform.
Advantages?
• User friendly: separate configuration from scripting
• Transparency: paying attention to functional design
• Future-proof: allowing the source and target platforms to scale
• Limited to no adaptations needed to the script

6. The
Peak
of
Data
Integration
20
23
Problem
● New business processes shorten
information management cycle
● One source platform replacing x
different sources
● Data available through message
queue streaming JSON objects
● About 30k messages/day, highly
variable in size
● 1 message contains 1-n objects
● ‘last one wins’
● Automate ETL to uniformise and
shorten data updates
● Allow for efficient future scaling

7. The
Peak
of
Data
Integration
20
23
Difficulties
● How to set up a workflow fit for the job?
● Performant and robust in a non-controllable message stream
volume
● How to make this generic & future-proof?
● Source and target platforms have different stakeholders and
evolve independently
● How to do complex mapping?
● Finding the breakoff point of spreadsheet configuration

8. The
Peak
of
Data
Integration
20
23
Difficulties
● How to deploy this on FME Form and different FME Flow
environments?
● Creating a portable script on FME Form connecting to the source
platform environments on FME Form
● Scalability?
● Accounting for future scaling in the source as well as target
platforms

9. The
Peak
of
Data
Integration
20
23
How we tackled the problem

10. The
Peak
of
Data
Integration
20
23
2 workspaces connected with jobSubmitters
1. Filter the relevant information
2. Execute the mapping

11. The
Peak
of
Data
Integration
20
23
Workspace 1: Client input - excel sheet
Description KoppelingTypeCode KoppelingSubTypeCode TrajectDetailCode Target Table
Neighbourhoud bycicle
parking
1 58 71
Bicycle
Parking
Bicycle Parking Gent 1 58 72
Bicycle
Parking
Byciclin Parking Others 1 58 83
Bicycle
Parking
Parking meters 1 58 85
Parking
meters
Charging posts 1 58 81
Charging
posts

12. The
Peak
of
Data
Integration
20
23
Workspace 1: Theory

13. The
Peak
of
Data
Integration
20
23
Worspace 1: Reality

14. The
Peak
of
Data
Integration
20
23
Workspace 2: Input mapping spreadsheet
Source attribute Target attribute Prefix Reference ExternalTable
Join
Attribute
Join Attribute
Values
Attribute External
Table
Parse Seperator
Default
Value
Properties.Capacity Capacity
Capacities.Surface Surface SU_
ContourLabels Owner alfa.TrajectDetailLabel Code (353,354,355) Label # City Ghent

15. The
Peak
of
Data
Integration
20
23
Workspace 2: Input mapping spreadsheet
Source attribute Target attribute Prefix Reference ExternalTable
Join
Attribute
Join Attribute
Values
Attribute External
Table
Parse Seperator
Default
Value
Properties.Capacity Capacity
Capacities.Surface Surface SU_
ContourLabels Owner alfa.TrajectDetailLabel Code (353,354,355) Label # City Ghent
Simple one on one mapping
Input Data
Attribute A Properties.Capacity Attribute ..
.. 50 bikes …
15 bikes
39 bikes
Desired output data
Attribute A Capacity Attribute …
.. 50 bikes …
15 bikes
39 bikes

16. The
Peak
of
Data
Integration
20
23
Workspace 2: Input mapping spreadsheet
Source attribute Target attribute Prefix Reference ExternalTable
Join
Attribute
Join Attribute
Values
Attribute External
Table
Parse Seperator
Default
Value
Properties.Capacity Capacity
Capacities.Surface Surface SU_
ContourLabels Owner alfa.TrajectDetailLabel Code (353,354,355) Label # City Ghent
Simple one on one mapping with prefix
Input Data
Attribute A Capacities.Surface Attribute ..
.. Concrete …
Sand
Wood
Desired output data
Attribute A Surface Attribute …
.. SU_Concrete …
SU_Sand
SU_Wood

17. The
Peak
of
Data
Integration
20
23
Workspace 2: Theory in the most simple way

18. The
Peak
of
Data
Integration
20
23
But what with the more complex mapping?
Build your own solution with custom transformers

19. The
Peak
of
Data
Integration
20
23
Workspace 2: Input mapping spreadsheet
Source attribute Target attribute Prefix Reference ExternalTable
Join
Attribute
Join Attribute
Values
Attribute External
Table
Parse Seperator
Default
Value
Properties.Capacity Capacity
Capacities.Surface Surface SU_
ContourLabels Owner alfa.TrajectDetailLabel Code (353,354,355) Label # City Ghent
Complex mapping
Input Data
Attribute A Contourlabels Attribute ..
.. #314#354#312 …
#310#320
#355#367#398
alfa.TrajectDetailLabel
Attribute A Code Label Attribute ..
.. 353 Police ..
354 Municipality
355 Fire station
… …
Desired output data
Attribute A Owner Attribute …
.. Municipality …
City Ghent
Fire Station

20. The
Peak
of
Data
Integration
20
23
Workspace 2: Theory complex mapping
Handy tip: @Value(@Value(attribute))

21. The
Peak
of
Data
Integration
20
23
Worspace 2: Reality

22. The
Peak
of
Data
Integration
20
23
Conclusion

23. The
Peak
of
Data
Integration
20
23
Summary
● Always try to think as generic as possible
● FME is an ideal tool for data migration
● Always keep it simple for the client

24. The
Peak
of
Data
Integration
20
23
ListExploding may cause
the process to explode.

25. Call to Action
1. Think further than the standard transformers
2. Make your process future-proof by making it as generic
as possible
3. SchemaMapper cannot be configured with scripted
parameters
4. @Value(@Value(Attribute)) is a handy trick for generic
flows

26. ThankYou!
matthias@nordend.eu
michael.ferre@district09.gent