1. Department of Civil, Architectural & Environmental Engineering 1 June 2, 2011 Development of a Hydrologic Community Modeling System Using a Workflow Engine Committee BO LU Dr. Michael Piasecki Drexel University Dr. Jonathan Goodall Dr. Franco Montalto Dr. Mira Olson Dr. Ilya Zaslavsky 6/2/2011

2. Department of Civil, Architectural & Environmental Engineering 1 Let’s imagine… Model/Module Data Data Data Data Data Data Data Data/Data access Model Model Model Model Model Model Model Model Tools of transformation, analysis, display etc. Tool Tool Tool Tool Tool Tool Tool 6/2/2011

3. Department of Civil, Architectural & Environmental Engineering 1 Let’s imagine… Model/Module Data Data Data Data Data Data Data Data Data Data/Data access Model Model Model Model Model Model Model Model Model Tools of transformation, analysis, display etc. Tool Tool Tool Tool Tool Tool Tool 6/2/2011

4. Department of Civil, Architectural & Environmental Engineering 1 Let’s imagine… Model/Module Data Data Data Data Data Data Data Data Data Data/Data access Model Model Model Model Model Model Model Model Model Tools of transformation, analysis, display etc. Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011

5. Department of Civil, Architectural & Environmental Engineering 1 Let’s imagine… Model Model/Module Data Data Data Data Data Data Data Data Data Data/Data access Model Model Model Model Model Model Model Model Model Tools of transformation, analysis, display etc. Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011

6. Department of Civil, Architectural & Environmental Engineering 1 Let’s imagine… Model/Module Data Data Data Data Data Data Data Data Data Data/Data access Model Model Model Model Model Model Model Model Model Model Model Tools of transformation, analysis, display etc. Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011

7. Department of Civil, Architectural & Environmental Engineering 1 Let’s imagine… Objective: Develop a Hydrologic Community Modeling System(HCMS) that allows constructing seamlessly integrated hydrologic models with swappable and portable modules. Model/Module Data Data Data Data Data Data Data Data Data Data/Data access Model Model Model Model Model Model Model Model Model Model Model Model Tools of transformation, analysis, display etc. Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011

9. Lack of credibility of the algorithms or methods encapsulated in the codes

10. Poor documentation of source codes

11. Lack of “good coding practices”

12. Model Integration

13. Lack of modular model structure

14. Intertwining of user interfaces and computing kernels

15. Incompatible programming languages

16. Data Interoperability

17. Distinct input and output data structures of models

18. Distinct data models undertaken by disparate data sources

20. Facilities: tools that ease the development of component models or the migration of legacy models. Data analysis tools, transformation tools etc.

22. Supporting high-performance computations and provenance capture.

24. Will its run-time performance be affected when migrating a legacy model into the environment?

27. Composing, executing, monitoring and recording workflows

29. Scheduling workflow execution

30. Loading/running workflows from local/remote database

31. Loading/running workflows from local/remote database

32. Loading/running workflows from local/remote database

33. Running multiple workflows on different nodes of a server clusterTRIDENT SQL DATABASE 6/2/2011

34. Department of Civil, Architectural & Environmental Engineering 7 Why use TRIDENT in hydrologic modeling? Composing workflows with swappable activities via the drag-and-drop manner on a GUI. Flexible Model Setup Allowing automatic and holistic execution without any external intervenes, or alternatively, interactive execution with the control of users. Interactive/Non-interactive Execution High-performance Computing Allowing parallel or concurrent execution, distributed computations in the GRID environment. Recording who, how, what and which resources are used in a workflow, and the derivation flow of data products. It ensures repeatability of model executions. Provenance Capture Easy to Share Sharing workflow through publication mechanismsor repositories. 6/2/2011

35. 6/2/2011 Department of Civil, Architectural & Environmental Engineering Introduction of the libraries of HCMS Data Access Library Data Processing Library Hydrologic Model Library Post-Anaylysis & Utilities Library 8

36. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 9 1.Data Access Library Data Sources: Retrieving data from following data sources using SOAP/FTP protocols .

38. NLCD: 30m * 30m, GeoTIFF[Activity 1] — Access NED or NLCD data within a specified area via Application Services. [Activity 2] — Decompress downloaded data files.

40. NLCD: 30m * 30m, GeoTIFF[Activity 1] — Access NED or NLCD data within a specified area via Application Services. [Activity 2] — Decompress downloaded data files.

42. NLCD: 30m * 30m, GeoTIFF[Activity 1] — Access NED or NLCD data within a specified area via Application Services. [Activity 2] — Decompress downloaded data files.

44. Temperature, Precipitation, Long wave/Short wave radiation, Pressure, Vertical/Horizontal wind speed etc.[Activity 1] — Download hourly data files(GRIB) from NLDAS-2 data server. ftp://hydro1.sci.gsfc.nasa.gov/data/s4pa/NLDAS/NLDAS_FORA0125_H.002/ [Activity 2] — Make a choice of fields from a given field list, the activity then extracts data of selected fields from the downloaded data files via a decoder “WGRIB”. [Activity 3] — Cut gridded data set within a specified geospatial extent. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 11

46. Temperature, Precipitation, Long wave/Short wave radiation, Pressure, Vertical/Horizontal wind speed etc.[Activity 1] — Download hourly data files(GRIB) from NLDAS-2 data server. ftp://hydro1.sci.gsfc.nasa.gov/data/s4pa/NLDAS/NLDAS_FORA0125_H.002/ [Activity 2] — Make a choice of fields from a given field list, the activity then extracts data of selected fields from the downloaded data files via a decoder “WGRIB”. [Activity 3] — Cut gridded data set within a specified geospatial extent. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 11

49. It facilitates retrieving hydrologic and meteorological observation time series data from a central metadata catalogue (HISCentral located at the San Diego Supercomputer Center) which holds the richest metadata information in the world for water data. Variable Name (e.g. precipitation) Service ID (optional) Geographical Extent (watershed boundary or latitude/longitude ) Temporal Extent Get Web Services In Box Semantic Checking… Get Sites Web Service IDs Updated Variables Ontology Dictionary Sites Metadata Get Variables Verify Variable Catalog Get Time Series Data Variable Codes WaterML Time Series Data/Metadata Parse Output UI Processing Step Configuration Input Web Service 6/2/2011

50. Get Data via WaterOneFlow Web Services in TRIDENT [Activity 1] — Get web services within a specified geospatial extent. [Activity 2] — Get site and variable metadata based on given variable name. [Activity 3] — Get time series data of given variable within the given geospatial extent. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 14

53. Delineate watershed/sub-watershed boundary, Generate river network; Create Triangulated Irregular Network(TIN); Process Soil, Land Cover data; Create Hydrologic Response Unit (HRU).

54. Time Series Processing

55. Interpolation/Extrapolation, Unit Conversion.

56. Data processing customized for data sources

57. Aggregate NLDAS-2, MPE gridded data for sub-watersheds.6/2/2011

59. Locate the outlet6/2/2011

63. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 20 Creating Hydrologic Response Unit Step 2: Processing Land Cover Data Step 3: Create HRU

64. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 21 Processing Time Series Data

66. For NLDAS-2 gridded data

67. For MPE gridded data6/2/2011

69. For NLDAS-2 gridded data

70. For MPE gridded data6/2/2011

72. A physically based, semi-distributed watershed model that simulates hydrologic fluxes.

73. The VB version converted from 9502 FORTRAN version is migrated into the following workflow. [Activity 1] — Compute Topographic Index Histogram for the whole watershed or each sub-basin. [Activity 2] — Compute Area-Distance Histogram for routing flow. [Activity 3] — Interactive activity for inputting/modifying initial condition and parameters. [Activity 4] — TOPMODEL computation kernel. 6/2/2011

79. Encoded in C# and compiled into Dynamic Link Libraries (DLL).

80. Define input/output variables explicitly via a metadata-tagging approach.

81. Define “Execute” function that is invoked by the engine at run time.

82. Programming work

83. Scripting from the ground up.

86. physical computation elements: Sub-basin, River, HRU

88. apply the SWAT workflow to simulate daily runoff hydrographs over a 4 year period ranging from 2005 to 2008.

89. apply the TOPMODEL workflow along with a loosely coupled hydrologic model for the simulation of a flood event.

90. Sub-studies

91. conduct DEM processing via three types of workflows, and subdivide watershed under two schemes.

92. analyze precipitation data accessed from different data sources.

93. estimate potential evapotranpiration using activities encapsulating different approaches. 6/2/2011

95. Workflows: 1)Step by Step workflow, 2)Terrain Processing workflow, 3)Web service based workflow

96. Delineation: 1) 7 sub-basins: 500,000 cells as threshold 2) 33 sub-basins: 100,000 cells as threshold Total Flow Path Sink Filled DEM Flow Direction Flow Accumulation Raw DEM Watershed Grid Stream order Watershed and River Network (.shp) Stream Raster 6/2/2011

98. Merged Soil Groups

99. HRUS-- 7 sub-basin watershed contains 23 HRUs -- 33 sub-basin watershed contains 114 HRUs 6/2/2011

102. The NLDAS and NWIS precipitation data exhibit a good correlation.

103. The NLDAS precipitation data are adopted in the following modeling.

104. Temperature, Solar radiation, wind speed, pressure are accessed from NLDAS-26/2/2011

107. The estimates of Penman-Monteith method are adopted in the following modeling. 6/2/2011

110. Aggregate the water balance of finer sub-basins belonging to each coarse sub-basin.

111. The water balance accumulated from that of finer sub-basins is close to the one of corresponding coarse sub-basin. 6/2/2011

114. Inputs----NLDAS-2 hourly precipitation

117. Inputs----NLDAS-2 hourly precipitation

119. In general, it is remarkably straightforward to build up workflows in the HCMS for hydrologic modeling purposes.

120. It can save time and effort through the automated execution that the workflow sequences afford.

121. With the nationwide data coverage of incorporated data sources, the HCMS can be applied to anywhere in the US.

122. Performance of HCMS

125. TRIDENT workflow system provides a platform for designing the HCMS and for assembling hydrologic models as workflow sequences.

126. The HCMS was tested by carrying out several typical hydrologic modeling studies over Schuylkill watershed. It is proved to be used quite well as a modeling platform. While it is not computational cost free due to the middle ware layer, the additional time consumption is “affordable”, especially in the lengthy data preparation arena. 6/2/2011

127. Department of Civil, Architectural & Environmental Engineering 40 Future Work Data Data Data Data Data Data Data Data Data Data Data Data Model Model Model Model Model Model Model Model Model Model Model Model Model Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011

128. Department of Civil, Architectural & Environmental Engineering 40 Future Work Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011

129. Department of Civil, Architectural & Environmental Engineering 40 Future Work Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011