1. Department of Civil, Architectural & Environmental Engineering 1 June 2, 2011 Development of a Hydrologic Community Modeling System Using a Workflow Engine Committee BO LU Dr. Michael Piasecki Drexel University Dr. Jonathan Goodall Dr. Franco Montalto Dr. Mira Olson Dr. Ilya Zaslavsky 6/2/2011

2. Department of Civil, Architectural & Environmental Engineering 1 Let’s imagine… Model/Module Data Data Data Data Data Data Data Data/Data access Model Model Model Model Model Model Model Model Tools of transformation, analysis, display etc. Tool Tool Tool Tool Tool Tool Tool 6/2/2011

3. Department of Civil, Architectural & Environmental Engineering 1 Let’s imagine… Model/Module Data Data Data Data Data Data Data Data Data Data/Data access Model Model Model Model Model Model Model Model Model Tools of transformation, analysis, display etc. Tool Tool Tool Tool Tool Tool Tool 6/2/2011

4. Department of Civil, Architectural & Environmental Engineering 1 Let’s imagine… Model Model/Module Data Data Data Data Data Data Data Data Data Data/Data access Model Model Model Model Model Model Model Model Model Tools of transformation, analysis, display etc. Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011

5. Department of Civil, Architectural & Environmental Engineering 1 Let’s imagine… Model/Module Data Data Data Data Data Data Data Data Data Data/Data access Model Model Model Model Model Model Model Model Model Model Model Tools of transformation, analysis, display etc. Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011

6. Department of Civil, Architectural & Environmental Engineering 1 Let’s imagine… Objective: Develop a Hydrologic Community Modeling System(HCMS) that allows constructing seamlessly integrated hydrologic models with swappable and portable modules. Model/Module Data Data Data Data Data Data Data Data Data Data/Data access Model Model Model Model Model Model Model Model Model Model Model Model Tools of transformation, analysis, display etc. Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011

8. Lack of credibility of the algorithms or methods encapsulated in the codes

9. Poor documentation of source codes

10. Lack of “good coding practices”

11. Model Integration

12. Lack of modular model structure

13. Intertwining of user interfaces and computing kernels

14. Incompatible programming languages

15. Data Interoperability

16. Distinct input and output data structures of models

17. Distinct data models undertaken by disparate data sources

19. Facilities: tools that ease the development of component models or the migration of legacy models. Data analysis tools, transformation tools etc.

21. Supporting high-performance computations and provenance capture.

23. Will its run-time performance be affected when migrating a legacy model into the environment?

26. Composing, executing, monitoring and recording workflows

28. Scheduling workflow execution

29. Loading/running workflows from local/remote database

30. Loading/running workflows from local/remote database

31. Loading/running workflows from local/remote database

32. Running multiple workflows on different nodes of a server clusterTRIDENT SQL DATABASE 6/2/2011

33. Department of Civil, Architectural & Environmental Engineering 7 Why use TRIDENT in hydrologic modeling? Composing workflows with swappable activities via the drag-and-drop manner on a GUI. Flexible Model Setup Allowing automatic and holistic execution without any external intervenes, or alternatively, interactive execution with the control of users. Interactive/Non-interactive Execution High-performance Computing Allowing parallel or concurrent execution, distributed computations in the GRID environment. Recording who, how, what and which resources are used in a workflow, and the derivation flow of data products. It ensures repeatability of model executions. Provenance Capture Easy to Share Sharing workflow through publication mechanismsor repositories. 6/2/2011

34. 6/2/2011 Department of Civil, Architectural & Environmental Engineering Introduction of the libraries of HCMS Data Access Library Data Processing Library Hydrologic Model Library Post-Anaylysis & Utilities Library 8

35. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 9 1.Data Access Library Data Sources: Retrieving data from following data sources using SOAP/FTP protocols .

37. NLCD: 30m * 30m, GeoTIFF[Activity 1] — Access NED or NLCD data within a specified area via Application Services. [Activity 2] — Decompress downloaded data files.

39. NLCD: 30m * 30m, GeoTIFF[Activity 1] — Access NED or NLCD data within a specified area via Application Services. [Activity 2] — Decompress downloaded data files.

41. NLCD: 30m * 30m, GeoTIFF[Activity 1] — Access NED or NLCD data within a specified area via Application Services. [Activity 2] — Decompress downloaded data files.

43. Temperature, Precipitation, Long wave/Short wave radiation, Pressure, Vertical/Horizontal wind speed etc.[Activity 1] — Download hourly data files(GRIB) from NLDAS-2 data server. ftp://hydro1.sci.gsfc.nasa.gov/data/s4pa/NLDAS/NLDAS_FORA0125_H.002/ [Activity 2] — Make a choice of fields from a given field list, the activity then extracts data of selected fields from the downloaded data files via a decoder “WGRIB”. [Activity 3] — Cut gridded data set within a specified geospatial extent. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 11

45. Temperature, Precipitation, Long wave/Short wave radiation, Pressure, Vertical/Horizontal wind speed etc.[Activity 1] — Download hourly data files(GRIB) from NLDAS-2 data server. ftp://hydro1.sci.gsfc.nasa.gov/data/s4pa/NLDAS/NLDAS_FORA0125_H.002/ [Activity 2] — Make a choice of fields from a given field list, the activity then extracts data of selected fields from the downloaded data files via a decoder “WGRIB”. [Activity 3] — Cut gridded data set within a specified geospatial extent. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 11

48. It facilitates retrieving hydrologic and meteorological observation time series data from a central metadata catalogue (HISCentral located at the San Diego Supercomputer Center) which holds the richest metadata information in the world for water data. Variable Name (e.g. precipitation) Service ID (optional) Geographical Extent (watershed boundary or latitude/longitude ) Temporal Extent Get Web Services In Box Semantic Checking… Get Sites Web Service IDs Updated Variables Ontology Dictionary Sites Metadata Get Variables Verify Variable Catalog Get Time Series Data Variable Codes WaterML Time Series Data/Metadata Parse Output UI Processing Step Configuration Input Web Service 6/2/2011

49. Get Data via WaterOneFlow Web Services in TRIDENT [Activity 1] — Get web services within a specified geospatial extent. [Activity 2] — Get site and variable metadata based on given variable name. [Activity 3] — Get time series data of given variable within the given geospatial extent. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 14

52. Delineate watershed/sub-watershed boundary, Generate river network; Create Triangulated Irregular Network(TIN); Process Soil, Land Cover data; Create Hydrologic Response Unit (HRU).

53. Time Series Processing

54. Interpolation/Extrapolation, Unit Conversion.

55. Data processing customized for data sources

56. Aggregate NLDAS-2, MPE gridded data for sub-watersheds.6/2/2011

58. Locate the outlet6/2/2011

62. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 20 Creating Hydrologic Response Unit Step 2: Processing Land Cover Data Step 3: Create HRU

63. 6/2/2011 Department of Civil, Architectural & Environmental Engineering 21 Processing Time Series Data

65. For NLDAS-2 gridded data

66. For MPE gridded data6/2/2011

68. For NLDAS-2 gridded data

69. For MPE gridded data6/2/2011

71. A physically based, semi-distributed watershed model that simulates hydrologic fluxes.

72. The VB version converted from 9502 FORTRAN version is migrated into the following workflow. [Activity 1] — Compute Topographic Index Histogram for the whole watershed or each sub-basin. [Activity 2] — Compute Area-Distance Histogram for routing flow. [Activity 3] — Interactive activity for inputting/modifying initial condition and parameters. [Activity 4] — TOPMODEL computation kernel. 6/2/2011

78. Encoded in C# and compiled into Dynamic Link Libraries (DLL).

79. Define input/output variables explicitly via a metadata-tagging approach.

80. Define “Execute” function that is invoked by the engine at run time.

81. Programming work

82. Scripting from the ground up.

85. physical computation elements: Sub-basin, River, HRU

87. apply the SWAT workflow to simulate daily runoff hydrographs over a 4 year period ranging from 2005 to 2008.

88. apply the TOPMODEL workflow along with a loosely coupled hydrologic model for the simulation of a flood event.

89. Sub-studies

90. conduct DEM processing via three types of workflows, and subdivide watershed under two schemes.

91. analyze precipitation data accessed from different data sources.

92. estimate potential evapotranpiration using activities encapsulating different approaches. 6/2/2011

94. Workflows: 1)Step by Step workflow, 2)Terrain Processing workflow, 3)Web service based workflow

95. Delineation: 1) 7 sub-basins: 500,000 cells as threshold 2) 33 sub-basins: 100,000 cells as threshold Total Flow Path Sink Filled DEM Flow Direction Flow Accumulation Raw DEM Watershed Grid Stream order Watershed and River Network (.shp) Stream Raster 6/2/2011

97. Merged Soil Groups

98. HRUS-- 7 sub-basin watershed contains 23 HRUs -- 33 sub-basin watershed contains 114 HRUs 6/2/2011

101. The NLDAS and NWIS precipitation data exhibit a good correlation.

102. The NLDAS precipitation data are adopted in the following modeling.

103. Temperature, Solar radiation, wind speed, pressure are accessed from NLDAS-26/2/2011

106. The estimates of Penman-Monteith method are adopted in the following modeling. 6/2/2011

109. Aggregate the water balance of finer sub-basins belonging to each coarse sub-basin.

110. The water balance accumulated from that of finer sub-basins is close to the one of corresponding coarse sub-basin. 6/2/2011

113. Inputs----NLDAS-2 hourly precipitation

116. Inputs----NLDAS-2 hourly precipitation

118. In general, it is remarkably straightforward to build up workflows in the HCMS for hydrologic modeling purposes.

119. It can save time and effort through the automated execution that the workflow sequences afford.

120. With the nationwide data coverage of incorporated data sources, the HCMS can be applied to anywhere in the US.

121. Performance of HCMS

124. TRIDENT workflow system provides a platform for designing the HCMS and for assembling hydrologic models as workflow sequences.

125. The HCMS was tested by carrying out several typical hydrologic modeling studies over Schuylkill watershed. It is proved to be used quite well as a modeling platform. While it is not computational cost free due to the middle ware layer, the additional time consumption is “affordable”, especially in the lengthy data preparation arena. 6/2/2011

126. Department of Civil, Architectural & Environmental Engineering 40 Future Work Data Data Data Data Data Data Data Data Data Data Data Data Model Model Model Model Model Model Model Model Model Model Model Model Model Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011

127. Department of Civil, Architectural & Environmental Engineering 40 Future Work Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011

128. Department of Civil, Architectural & Environmental Engineering 40 Future Work Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Model Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool Tool 6/2/2011