Under the hood of the Altalis Platform
•Als PPTX, PDF herunterladen•
0 gefällt mir•5,578 views
With the implementation of the new Altalis Platform, Altalis continues to distribute a variety of spatial data products in various formats that span Alberta and elsewhere. To fulfill translation requests, a number of factors determine the automated workflow, such as by a data update or delivery task, by product, by format, and more. We will showcase how we used FME Workbench and FME Server as a core component of the application platform to perform the translations and the optimizations we implemented.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Under the hood of the Altalis Platform
Ähnlich wie Under the hood of the Altalis Platform (20)
Mehr von Safe Software
Mehr von Safe Software (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Under the hood of the Altalis Platform
- 1. Under the hood of the Altalis Platform Janet Khuu Silvacom Ltd. 2020
- 2. Agenda 1. Introduction 2. Architecture 3. Altalis Products a. Cadastral b. Urban and Rural c. Area of Interest d. DIDs/DIDs+ 4. QC/Testing 5. Maintenance Data Detox
- 3. About me ● GIS Specialist at Silvacom for ~5 years ● GIS field for ~8 years ● FME Certified Professional
- 4. About Altalis ● Part of The Silvacom Group ● Trusted distributor of spatial data since 1998 ● In 2015, Altalis initiated a major refresh of their webstore platform ● Webstore enables customers to explore, view, and acquire spatial data products both paid and open data with the click of a button
- 5. Technologies in use ● Apache Tomcat ● Angular Javascript, Java ● PostgreSQL and PostGIS ● GeoServer, GeoWebCache, GeoTools And of course… FME Workbench and FME Server (2016) Altogether: ● 3 x application, 2 x FME Server, 2 x GeoServer
- 6. FME Server ● 2 engines x 2 hosts ● Scheduled jobs ● REST API ● Tagged jobs for products to run on dedicated engines
- 7. Reader and Writer formats ● CAD ○ Microstation DGN ○ AutoCAD DWG ● GIS ○ Shapefile ○ FileGDB (API and ArcObjects) ○ GeoJSON ● Databases ○ Access MDB ○ PostGIS/PostgreSQL
- 9. Workspace parameters ● workspaceStream ● action ● deliveryType ● aoi | areaList ● orderId ● packageId ● resourceDbLocation ● sourceFormat ● sourcePath ● sourceProjection ● outputFormat ● outputPath ● outputProjection
- 10. Chained workspaces Go through your proof points. A. B. C. A screenshot can go here. Or delete this box and use the text box beneath.
- 11. Chained workspaces Go through your proof points. A. B. C.
- 12. FME Server (Cadastral) Repository files: ● cadastral.fmw ○ cadastral_to_postgis ○ cadastral_to_pricing ○ cadastral_to_dates ○ cadastral_to_preview ○ cadastral_to_cad ○ cadastral_to_gdb ○ cadastral_to_gdb_update
- 13. Cadastral FME workspace for CAD formats:
- 14. Urban and Rural Cadastral
- 15. Urban and Rural Cadastral Product preview
- 16. Urban and Rural Cadastral Product preview with RuralUrban selection grid
- 17. Urban and Rural Cadastral Tile selection to file selection 1. areaList = lacombe;426040 2. 3. File list for DGN Reader - sourcePath/lacombe.10t - sourcePath/426040.10t
- 18. Urban and Rural Cadastral: Issue ● Read from all source files, then write ○ All collected in memory
- 19. Urban and Rural Cadastral: Optimized ● Use batch processing: Read from one, write out one ○ sourcePath/lacombe.10t → outputPath/lacombe.dwg ○ sourcePath/426040.10t → outputPath/426040.dwg ● Custom embedded transformers:
- 20. Area of Interest Cadastral Export merged (AOI) Cadastral file:
- 21. DIDs/DIDs+ DIDs ● Shapefiles ○ AB_APPL ○ AB_HIST ○ AB_LCON ● Discrepancy_report.mdb ○ Discrepancy_Report ○ ATS_Township ○ Dispositions_Not_Mapped DIDs+ ● Shapefiles ○ DAB_APPL ○ AB_HIST ○ AB_LCON ● Discrepancy_report.mdb ○ Discrepancy_Report ○ ATS_Township ○ Dispositions_Not_Mapped ● Access database ○ ACTIVITIES ○ ADDRESSES ○ CLIENTS ○ COMMENTS
- 22. DIDs/DIDs+: Issues ● Differences between DIDs and DIDs+ ● Multiple spatial and attribute tables ● Table joins ● Performance querying with complex AOIs
- 23. DIDs/DIDs+ FME workspace for data requests:
- 24. DIDs/DIDs+
- 28. Clipping with AOI ● Select, draw or upload shape on map ● GeoJSON format { "type": "Polygon", "coordinates": [ [-115.1276447834104, 53.28148915782895], [-115.12698179182546, 53.05749057020401], [-114.27640940758613, 53.05535810617569], [-114.27263182150816, 53.27933935656844], [-115.1276447834104, 53.28148915782895]] ... ] }
- 29. DIDs+ database querying ● AOI overlay with DAB_APPL WITH aoi AS ( --temp table of AOI geometry SELECT ST_Subdivide(ST_GeomFromGeoJSON(shape),10000) AS aoi_geom FROM Grid_Schema."AOI_Lookup" WHERE id=AOI ) SELECT string_agg(disp_num,',') AS disp_num FROM ( SELECT concat(E''',"DISP_NUM",E''') AS disp_num FROM ( SELECT DISTINCT "DISP_NUM" FROM sourcePath."DAB_APPL" INNER JOIN aoi ON ST_Intersects(aoi_geom, geom) ) AS intersected GROUP BY disp_num ORDER BY disp_num ) AS d;
- 30. DIDs/DIDs+ In summary ● Get list of all possible tables, apply flags ● Use Common Table Expressions (CTE) and subqueries ● AOI ○ Store in database lookup table ○ Read GeoJSON string in SQL as a temporary table ○ Simplify and break apart the AOI with PostGIS functions
- 31. Quality check methods 1. Automated count checks 2. Unit testing overlays 3. Error codes
- 32. QC: Deleting by grid tile Recall from the RuralUrban grid for Cadastral…
- 33. QC: Deleting by grid tile Compare count from before and after deletion:
- 34. QC: Updating by grid tile Compare count between database and product delivery:
- 35. QC: Unit testing overlays Overlaying with AOIs can be very complex. An AOI can be… ● Single polygon feature ● Multipart polygon feature ● Multipart polygon with donut holes feature Overlaying against product data… ● Product data features that are also complex features ● Data precision vs expected output
- 36. QC: Unit testing overlays Interject test values into workspace parameters:
- 37. QC: Unit testing overlays Switch workflow to use test values:
- 38. Error codes Error code system: Example: ALTERR-0111: Count mismatch with Cadastral. ALTERR- 0|1 0|1 0|1 0|1 FME workspace Data store Data product format QC activity
- 39. Quality Checking In summary ● Lots of count checks ● Testing method must be as optimal as the feature function ● Narrow fault points with error codes
- 40. Maintenance ● Gathering metrics ● ArcGIS license heartbeat
- 41. Comparing data volume vs runtime Pre-processed Cadastral FileGDB:
Hinweis der Redaktion
- So a little bit about me. I’ve been a GIS Specialist at Silvacom for about 5 years, in the GIS field for about 8 years. And thanks to all the work put into this platform, I became an FME Certified Professional.
- Now about Altalis. They are part of the Silvacom group of companies. … We’re based in Edmonton, Alberta, Canada. Altalis’s products serve mostly the province of Alberta, but also Canada.
- With the technology refresh, these are what’s in use. The web server is Apache Tomcat. Front end language is Angular/Javascript with GeoTools and backend language is Java. Database storage is PostgreSQL with the PostGIS extension The platform app has a major web mapping component, and that’s served by GeoServer with GeoWebCache extension. Lastly, we have FME Workbench and FME Server – note that they are 2016 versions. Altogether, for the purpose of load balancing, there are 3 application servers, 2 FME Servers, and 2 GeoServers.
- For FME Server in detail, There are 2 engines each for 2 hosts. There are scheduled jobs. The application makes use of the REST API to submit and check on jobs. And there’s certain types of jobs that are tagged for products to run on dedicated engines.
- GeoJSON serves multiple purposes for the Altalis platform It works on the website’s web map to track customer areas of interest. FME and PostGIS perform overlays with it depending on the translation.
- So I’m at the new site. This is where I order data. How does FME process order requests?
- The app sends a request of values to the following standardized parameters via FME Server with the FME REST API. I’ve grouped together some of those parameters. First group is workspaceStream, action, deliveryType, and aoi OR areaList. These tell FME what kind of translation it is Is it a delivery of data to the client or update to our database? Is there clipping to the product to just an area of interest? 2nd group helps us track orders and package files between FME and the app, while resourceDbLocation points to the central repository of asset files and custom transformers for FME to find. Last sets of groups define the data inputs and outputs of the translation.
- Pretty diagram of the trail of translations implemented in ATR. The main task of the master controller is to parse workspaceStream param to product-specific controllers.
- This is the main workspace the application calls via the API call, also called master controller. It’s called master because it controls the flow of translation requests.
- Let’s have an indepth look at all the translation files for the Cadastral product. In blue are all the translations that feed into our database, which stores the data and also auxiliary data for the webmap so that customers can interact with the product’s information. In purple are all translations to deliver in data formats for the customer.
- I’ll go over the workspace for Cadastral CAD products. Since the workspace is in 2016, I can’t collapse bookmarks, so I had to make a simplified diagram.
- There’s several parts to this translation First, the workspace reads in relevant job parameters and prepares global values for the workspace to use. With the is-AOI condition, it’s essentially checking for which Cadastral product it is. Here, we’re focused on the False state. Which leads us to the is-valid-areaList condition to create file paths. I’ll go indepth to this part in a bit. At this point, all the source data is known to the workspace to perform the translation. All it needs to determine next is the format output – Microstation or AutoCAD – and format layers accordingly. Now back to the valid-areaList check…
- Taking a step back, I want to show you what data and coverage we’re dealing with for the Urban and Rural Cadastral product. This is what it looks like – covering the full province of Alberta.
- Data is actually made up of 6000+ files. Each file coverage is represented by this selection grid.
- So through the web map interface, a customer can select from this grid of tiles of their interest. How do these tiles translate to how the workspace will fetch the right files? In this example, the customer selected Lacombe and township 426040. The app prepares this list of tiles as a delimited string for the areaList parameter. The workspace breaks apart this string to individual records that represent a tile. The name of the tile is the same as the filename of the DGN file it needs to read. Concatenate the filename with the sourcePath parameter – which should point to the storage folder directory – and DGN Reader can read in the data.
- Now what happens where there’s lots of tiles requested? FME would be reading all the source files before it writes anything out. All this data is stored in memory, or worse, onto hard disk.
- Use batch processing in the form of custom transformers. Read one file in, write it out, read the next one in, write that one out. What happens in these custom transformers? They run through similar processing as for the (Merged AOI) Cadastral product <next slide>
- Here’s the other Cadastral product served by the same workspace, where instead of selecting from a tile grid, the customer can request data based on their own area of interest polygon. If they had requested DGN format, set lineweight, seed file, cell file. If they had requested DWG format, set... Now for the AOI=all condition, sources are read in two ways for performance: For full province, just read in all the 6000+ files. For clipping, for each file read in, pass it through the clipper.
- For each table found, apply a flag indicators: Is it part of a discrepancy report db? Is it a DIDs or DIDs+ product?
- From DAB_APPL, get a list of disposition numbers Use the list in where-clause for each attribute table From each attribute table, get IDs found List of IDs to retrieve records
- An example with DIDs product showing clipping with an AOI. On the webmap, the customer creates the AOI, which is in the GeoJSON format. GeoJSON string is stored in a lookup table, with an ID. This ID is what’s passed in the AOI parameter.
- Overlay is done over SQL with PostGIS functions. AOI is brought into the overlay in a CTE expression – keeps AOI geometry in memory AOI geom is overlayed with DIDs geometry From the overlay, retrieve all the unique disp numbers of the disp polygons Format the list into a compact string that will fit into a where-clause for another SQL to query DIDs+ tables Essentially all querying is done by Postgres/PostGIS, not FME.
- Get the areaList into individual records Get a count of the tiles: count-before Get FeatureWriter to delete from the FileGDB by tile. Output summary stats as count-after Use FeatureMerger to match count-before and -after. Continue onward with insertion of new records.
- That was for updating data into our database. This QC is for deliverying data from our database. It again uses areaList… Formulate a SQL string to query the product’s database tables for a count. At the same time, get the summary output from FeatureWriter. Use FeatureMerger to match the count values of the processed output and stored data.
- Toggle the connections FeatureReader reads from the qc_aois table with an AOI ID lookup as a text file. Text string is converted to a SQL where-clause string. Wehere-clause is inserted into AttributeCreator. Why text string? A complicated AOI shape greatly reduces performance and increase runtime. Better to treat it as a string and let one system handle the geometry and overlay – PostGIS.
- ALTERR-0111 indicates there is no fault with the workspace. This is a failure by QC which, if not caught, would have allowed for a successful job. Check database data against the product data for possible resolution. No error code indicates there’s an exceptional use case.
- Hint at how you are going to prove your message, by telling the audience how your info will flow.
- Orange line: tile count Blue line: runtime