SlideShare a Scribd company logo
1 of 37
Making Maintenance Manageable
  Using FME to Maintain FEMA’s National Flood Hazard Layer

Rob Gaines
Senior GIS Analyst, Zimmerman Associates (FEMA Ctr)




                                                             April 18th, 2012
What is the NFHL?


 FEMA’s National Flood Hazard Layer, part of the
  National Flood Insurance Program.

 Nationwide regulatory GIS dataset used to
  communicate flood risk information.
The Numbers


 Compiled from 2081 individual community or
  countywide flood study datasets (and growing).

 Average 95 county/communities updated per
  month.

 32 spatial layers, 18 lookup tables.

 Over 85 million records in total.
How is the NFHL Accessed?


 MapViewer: 100,000 – 200,000 requests per
  month.

 KMZ/WMS: 3 – 6 million requests per month.

 NFHL Extract (via state shapefiles).
What is the NFHL?
What is the NFHL?
What is the NFHL?
What is the NFHL?
FIRM/NFHL Production Structure




FIRM Database        PTS rFHL          CDS NFHL
 County Dataset   Regional Dataset   National Dataset
FIRM/NFHL Production Structure
Data Going Effective 4/3/2012
Challenges


 Data produced at community level.

 Initial QA/QC process not intended for NFHL.

 Bureaucracy: 10 exceptions to every rule.

 Far removed from data producers.

 Inherited systems.
What Is FME Used For?


 QA/QC of data submissions.

 Data replication.

 Extraction.

 Report generation / tracking.

 LOMA Point creation.
Basic QA/QC Checks


 All records have DFIRM_ID value.
 All required layers are present.
 No duplicate geometry.
 Flood hazard lines & areas have valid flood zone
  values.
 Effective date information is correct.
 New data contains all currently effective LOMRs.
QA/QC FME Model
QA/QC: DFIRM_ID Value


 Uses FeatureMerger transformer to check that all
  records have DFIRM_ID value and all DFIRM_ID
  values match a record in the STUDY_INFO table.
QA/QC: Required Layers


 Uses ListPopulator and ListExploder along with
  FeatureMerger to create a list of all layers present
  in submission and then check them against the
  list of required layers.
QA/QC: Duplicate Geometry


 Concatenates a number of geometric attributes
  (such as coordinate count, length / area, etc)
  with flat attributes to detect duplicate features
  and filter them out.
QA/QC: Valid Values


 Uses the AttributeFilter to detect any invalid flood
  zone values.
QA/QC: LOMR Checker


 Compares submission LOMR layer with current
  NFHL LOMR layer to detect any missing
  information.
Sample QA/QC Sheet
Data Staging & Replication

            Staging
 NFHL
 Data
              DB


 NFHL         FME
 Data        Loader                NFHL SDE


 NFHL
 Data         FME       FME
             Logger   Replicator




             Log      MS Access
                      Interface
             DB
StagingLoad Model
StagingLoad Model: Logger
Replicator Access Query
Replicator Access Form
Extraction


 Extract all NFHL records into state-based
  shapefiles.

 Generate reports such as panel and LOMR count
  for each state.

 QC: Check current extract against prior extract to
  ensure completeness of layers.
Extraction: QC FME Model


 Runs through each state extract.

 Uses ListBuilder to build a list of each state’s
  layers and check them against previous month’s.

 Generates Excel report.
Extraction: QC Sheet
Reporting & Tracking


 Inventory of current NFHL holdings.

 Status of currently effective LOMRs.

 QC of existing NFHL data.
   Missing layers.
   Invalid values.
   Obsolete data.
NFHL Inventory Report
NFHL LOMR Status Report
LOMA Mapper


 Daily process which converts flat data with lat/lon
  attributes to geospatial data in SDE.

 Replaced custom-developed VB application.

 Original application: 18+ hour runtime.
  FME Model: <10 minute runtime.
LOMA Mapper Flowchart
LOMA Mapper FME Model
Into the Future… FME Server


 Internal
   Automated data submission & QA/QC process.

 External
   Clip/Zip/Ship via MapViewer.
   Custom extracts.
   Custom on-demand GIS services.
Thank You!


 Questions?

 For more information:
   Rob Gaines: robert.gaines@riskmapcds.com
    Zimmerman Associates Inc. (FEMA Contractor)

   FEMA Map Service Center: http://msc.fema.gov

More Related Content

Viewers also liked

Lightning Talk: FME 2012 Tips and Tricks
Lightning Talk: FME 2012 Tips and TricksLightning Talk: FME 2012 Tips and Tricks
Lightning Talk: FME 2012 Tips and TricksSafe Software
 
Using FME for the City of Palo Alto Topobase Implentation
Using FME for the City of Palo Alto Topobase ImplentationUsing FME for the City of Palo Alto Topobase Implentation
Using FME for the City of Palo Alto Topobase ImplentationSafe Software
 
What's Great in FME 2012
What's Great in FME 2012What's Great in FME 2012
What's Great in FME 2012Safe Software
 
FME Lightning Talk: Raster
FME Lightning Talk: RasterFME Lightning Talk: Raster
FME Lightning Talk: RasterSafe Software
 
FME Lightning Talk: LiDAR
FME Lightning Talk: LiDARFME Lightning Talk: LiDAR
FME Lightning Talk: LiDARSafe Software
 
Hydraulic Modelling with GIS Data
Hydraulic Modelling with GIS DataHydraulic Modelling with GIS Data
Hydraulic Modelling with GIS DataSafe Software
 
Managing Spatial Data for Telecommunications Using FME
Managing Spatial Data for Telecommunications Using FMEManaging Spatial Data for Telecommunications Using FME
Managing Spatial Data for Telecommunications Using FMESafe Software
 
FME Lightning Talk: Esri Geodatabase Templates
FME Lightning Talk: Esri Geodatabase TemplatesFME Lightning Talk: Esri Geodatabase Templates
FME Lightning Talk: Esri Geodatabase TemplatesSafe Software
 
GIS Data Management for Large Infrastructure Projects
GIS Data Management for Large Infrastructure ProjectsGIS Data Management for Large Infrastructure Projects
GIS Data Management for Large Infrastructure ProjectsSafe Software
 
FME Lightning Talk: FMEpedia
FME Lightning Talk: FMEpediaFME Lightning Talk: FMEpedia
FME Lightning Talk: FMEpediaSafe Software
 
Improving Productivity Using FME Workbench
Improving Productivity Using FME WorkbenchImproving Productivity Using FME Workbench
Improving Productivity Using FME WorkbenchSafe Software
 
FME Lightning Talk: Multiprocessing
FME Lightning Talk: MultiprocessingFME Lightning Talk: Multiprocessing
FME Lightning Talk: MultiprocessingSafe Software
 

Viewers also liked (14)

FME Store
FME StoreFME Store
FME Store
 
Lightning Talk: FME 2012 Tips and Tricks
Lightning Talk: FME 2012 Tips and TricksLightning Talk: FME 2012 Tips and Tricks
Lightning Talk: FME 2012 Tips and Tricks
 
Using FME for the City of Palo Alto Topobase Implentation
Using FME for the City of Palo Alto Topobase ImplentationUsing FME for the City of Palo Alto Topobase Implentation
Using FME for the City of Palo Alto Topobase Implentation
 
What's Great in FME 2012
What's Great in FME 2012What's Great in FME 2012
What's Great in FME 2012
 
FME & Governement
FME & GovernementFME & Governement
FME & Governement
 
FME Lightning Talk: Raster
FME Lightning Talk: RasterFME Lightning Talk: Raster
FME Lightning Talk: Raster
 
FME Lightning Talk: LiDAR
FME Lightning Talk: LiDARFME Lightning Talk: LiDAR
FME Lightning Talk: LiDAR
 
Hydraulic Modelling with GIS Data
Hydraulic Modelling with GIS DataHydraulic Modelling with GIS Data
Hydraulic Modelling with GIS Data
 
Managing Spatial Data for Telecommunications Using FME
Managing Spatial Data for Telecommunications Using FMEManaging Spatial Data for Telecommunications Using FME
Managing Spatial Data for Telecommunications Using FME
 
FME Lightning Talk: Esri Geodatabase Templates
FME Lightning Talk: Esri Geodatabase TemplatesFME Lightning Talk: Esri Geodatabase Templates
FME Lightning Talk: Esri Geodatabase Templates
 
GIS Data Management for Large Infrastructure Projects
GIS Data Management for Large Infrastructure ProjectsGIS Data Management for Large Infrastructure Projects
GIS Data Management for Large Infrastructure Projects
 
FME Lightning Talk: FMEpedia
FME Lightning Talk: FMEpediaFME Lightning Talk: FMEpedia
FME Lightning Talk: FMEpedia
 
Improving Productivity Using FME Workbench
Improving Productivity Using FME WorkbenchImproving Productivity Using FME Workbench
Improving Productivity Using FME Workbench
 
FME Lightning Talk: Multiprocessing
FME Lightning Talk: MultiprocessingFME Lightning Talk: Multiprocessing
FME Lightning Talk: Multiprocessing
 

Similar to Making Maintenance Manageable

SoftQL - Telecom Triage Services
SoftQL - Telecom Triage Services SoftQL - Telecom Triage Services
SoftQL - Telecom Triage Services Amar Uppalapati
 
Venturing into Hadoop Large Clusters
Venturing into Hadoop Large ClustersVenturing into Hadoop Large Clusters
Venturing into Hadoop Large ClustersVARUN SAXENA
 
Venturing into Large Hadoop Clusters
Venturing into Large Hadoop ClustersVenturing into Large Hadoop Clusters
Venturing into Large Hadoop ClustersNaganarasimha Garla
 
Application Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureApplication Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureVARUN SAXENA
 
Application Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureApplication Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureVARUN SAXENA
 
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords   The Stream Processor as the Database - Apache Flink @ Berlin buzzwords
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords Stephan Ewen
 
Centralized logging with Flume
Centralized logging with FlumeCentralized logging with Flume
Centralized logging with FlumeRatnakar Pawar
 
Mainframe Virtual User Group Summer 2013
Mainframe Virtual User Group Summer 2013Mainframe Virtual User Group Summer 2013
Mainframe Virtual User Group Summer 2013Serena Software
 
FME User Conference Presentation
FME User Conference PresentationFME User Conference Presentation
FME User Conference Presentationcwise
 
Fmeuc Presentation Hrsd3
Fmeuc Presentation Hrsd3Fmeuc Presentation Hrsd3
Fmeuc Presentation Hrsd3guest5c50492
 
Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...
Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...
Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...Yahoo Developer Network
 
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...Flink Forward
 
Environment Canada's Data Management Service
Environment Canada's Data Management ServiceEnvironment Canada's Data Management Service
Environment Canada's Data Management ServiceSafe Software
 
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017Till Rohrmann
 
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...Ververica
 
Training Slides: 153 - Working with the CLI
Training Slides: 153 - Working with the CLITraining Slides: 153 - Working with the CLI
Training Slides: 153 - Working with the CLIContinuent
 
Stephan Ewen - Experiences running Flink at Very Large Scale
Stephan Ewen -  Experiences running Flink at Very Large ScaleStephan Ewen -  Experiences running Flink at Very Large Scale
Stephan Ewen - Experiences running Flink at Very Large ScaleVerverica
 

Similar to Making Maintenance Manageable (20)

FME to the Rescue
FME to the RescueFME to the Rescue
FME to the Rescue
 
SoftQL - Telecom Triage Services
SoftQL - Telecom Triage Services SoftQL - Telecom Triage Services
SoftQL - Telecom Triage Services
 
Venturing into Hadoop Large Clusters
Venturing into Hadoop Large ClustersVenturing into Hadoop Large Clusters
Venturing into Hadoop Large Clusters
 
Venturing into Large Hadoop Clusters
Venturing into Large Hadoop ClustersVenturing into Large Hadoop Clusters
Venturing into Large Hadoop Clusters
 
Lee.stat
Lee.statLee.stat
Lee.stat
 
Application Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureApplication Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and Future
 
Application Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureApplication Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and Future
 
The Stream Processor as a Database Apache Flink
The Stream Processor as a Database Apache FlinkThe Stream Processor as a Database Apache Flink
The Stream Processor as a Database Apache Flink
 
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords   The Stream Processor as the Database - Apache Flink @ Berlin buzzwords
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords
 
Centralized logging with Flume
Centralized logging with FlumeCentralized logging with Flume
Centralized logging with Flume
 
Mainframe Virtual User Group Summer 2013
Mainframe Virtual User Group Summer 2013Mainframe Virtual User Group Summer 2013
Mainframe Virtual User Group Summer 2013
 
FME User Conference Presentation
FME User Conference PresentationFME User Conference Presentation
FME User Conference Presentation
 
Fmeuc Presentation Hrsd3
Fmeuc Presentation Hrsd3Fmeuc Presentation Hrsd3
Fmeuc Presentation Hrsd3
 
Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...
Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...
Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...
 
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...
 
Environment Canada's Data Management Service
Environment Canada's Data Management ServiceEnvironment Canada's Data Management Service
Environment Canada's Data Management Service
 
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
 
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...
 
Training Slides: 153 - Working with the CLI
Training Slides: 153 - Working with the CLITraining Slides: 153 - Working with the CLI
Training Slides: 153 - Working with the CLI
 
Stephan Ewen - Experiences running Flink at Very Large Scale
Stephan Ewen -  Experiences running Flink at Very Large ScaleStephan Ewen -  Experiences running Flink at Very Large Scale
Stephan Ewen - Experiences running Flink at Very Large Scale
 

More from Safe Software

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action:  Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action:  Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsSafe Software
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemSafe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISSafe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriSafe Software
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologySafe Software
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Safe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersSafe Software
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsSafe Software
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategySafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Safe Software
 

More from Safe Software (20)

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action:  Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action:  Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data Ecosystem
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GIS
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s Founders
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 

Recently uploaded

Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...apidays
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...Zilliz
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityWSO2
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistandanishmna97
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdfSandro Moreira
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...apidays
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDropbox
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MIND CTI
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century educationjfdjdjcjdnsjd
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodJuan lago vázquez
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...apidays
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWERMadyBayot
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxRemote DBA Services
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyKhushali Kathiriya
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services
 

Recently uploaded (20)

Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptx
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 

Making Maintenance Manageable

  • 1. Making Maintenance Manageable Using FME to Maintain FEMA’s National Flood Hazard Layer Rob Gaines Senior GIS Analyst, Zimmerman Associates (FEMA Ctr) April 18th, 2012
  • 2. What is the NFHL?  FEMA’s National Flood Hazard Layer, part of the National Flood Insurance Program.  Nationwide regulatory GIS dataset used to communicate flood risk information.
  • 3. The Numbers  Compiled from 2081 individual community or countywide flood study datasets (and growing).  Average 95 county/communities updated per month.  32 spatial layers, 18 lookup tables.  Over 85 million records in total.
  • 4. How is the NFHL Accessed?  MapViewer: 100,000 – 200,000 requests per month.  KMZ/WMS: 3 – 6 million requests per month.  NFHL Extract (via state shapefiles).
  • 5. What is the NFHL?
  • 6. What is the NFHL?
  • 7. What is the NFHL?
  • 8. What is the NFHL?
  • 9. FIRM/NFHL Production Structure FIRM Database PTS rFHL CDS NFHL County Dataset Regional Dataset National Dataset
  • 12. Challenges  Data produced at community level.  Initial QA/QC process not intended for NFHL.  Bureaucracy: 10 exceptions to every rule.  Far removed from data producers.  Inherited systems.
  • 13. What Is FME Used For?  QA/QC of data submissions.  Data replication.  Extraction.  Report generation / tracking.  LOMA Point creation.
  • 14. Basic QA/QC Checks  All records have DFIRM_ID value.  All required layers are present.  No duplicate geometry.  Flood hazard lines & areas have valid flood zone values.  Effective date information is correct.  New data contains all currently effective LOMRs.
  • 16. QA/QC: DFIRM_ID Value  Uses FeatureMerger transformer to check that all records have DFIRM_ID value and all DFIRM_ID values match a record in the STUDY_INFO table.
  • 17. QA/QC: Required Layers  Uses ListPopulator and ListExploder along with FeatureMerger to create a list of all layers present in submission and then check them against the list of required layers.
  • 18. QA/QC: Duplicate Geometry  Concatenates a number of geometric attributes (such as coordinate count, length / area, etc) with flat attributes to detect duplicate features and filter them out.
  • 19. QA/QC: Valid Values  Uses the AttributeFilter to detect any invalid flood zone values.
  • 20. QA/QC: LOMR Checker  Compares submission LOMR layer with current NFHL LOMR layer to detect any missing information.
  • 22. Data Staging & Replication Staging NFHL Data DB NFHL FME Data Loader NFHL SDE NFHL Data FME FME Logger Replicator Log MS Access Interface DB
  • 27. Extraction  Extract all NFHL records into state-based shapefiles.  Generate reports such as panel and LOMR count for each state.  QC: Check current extract against prior extract to ensure completeness of layers.
  • 28. Extraction: QC FME Model  Runs through each state extract.  Uses ListBuilder to build a list of each state’s layers and check them against previous month’s.  Generates Excel report.
  • 30. Reporting & Tracking  Inventory of current NFHL holdings.  Status of currently effective LOMRs.  QC of existing NFHL data.  Missing layers.  Invalid values.  Obsolete data.
  • 33. LOMA Mapper  Daily process which converts flat data with lat/lon attributes to geospatial data in SDE.  Replaced custom-developed VB application.  Original application: 18+ hour runtime. FME Model: <10 minute runtime.
  • 36. Into the Future… FME Server  Internal  Automated data submission & QA/QC process.  External  Clip/Zip/Ship via MapViewer.  Custom extracts.  Custom on-demand GIS services.
  • 37. Thank You!  Questions?  For more information:  Rob Gaines: robert.gaines@riskmapcds.com Zimmerman Associates Inc. (FEMA Contractor)  FEMA Map Service Center: http://msc.fema.gov