Arc Hydro - Stormwater Processing
Arc Hydro - Stormwater Processing
Arc Hydro - Stormwater Processing
i
5.2 Downstream trace ............................................................................................ 52
5.3 Create network connectivity ............................................................................. 54
6.0 Tools and process automation ............................................................... 56
6.1 Stormwater data development tools ................................................................. 56
6.2 Stormwater analysis tools ................................................................................ 58
7.0 Database design ....................................................................................... 60
7.1 Raster data structures ...................................................................................... 61
7.2 Vector data structures/network/relationship ..................................................... 62
7.3 Tabular data structures .................................................................................... 64
Appendix 1. Stormwater preprocessing automation results ........................ 65
ii
1.0 Executive Summary
Arc Hydro concept and base data model have supported stormwater analyses from the inception
in 2002. Stormwater representation is combination of surface drainage and connectivity defined
by built infrastructure that often redirects and alters surface drainage. Early development of Arc
Hydro tools in cooperation with Southwest Florida Water Management District focused on
detailed representation of such complex drainage systems based on availability of high-quality
DEMs and asset inventory with ultimate goal of supporting detailed integrated hydrologic and
hydraulic models operating on such systems.
The biggest problem with stormwater implementation was historically the level of detail that is
needed to capture the intricacies in stormwater flow, both for the overland and built flow
patterns. For a reasonable representation, high quality DEM and stormwater asset inventory is
needed. Without both, attempts to build a viable stormwater system representation are not
feasible.
Availability of high resolution DEMs (1m cell size or better) and better quality of stormwater
infrastructure assets make automation and analysis more viable. The Arc Hydro team has
supported several efforts in the past few years focusing on stormwater representation within Arc
Hydro context focusing on representation of general flow patterns as opposed to detailed flow
modeling. In a nutshell, the focus is on establishing the stormwater system representation that
allows the user to perform basic upstream (watershed area) and downstream (flow path) analysis
from any point in the landscape based on minimal stormwater and terrain informational content.
This document describes the implementation of stormwater representation within Arc Hydro
framework based on these more general requirements and additional Arc Hydro tools and
workflows that have been developed to handle stormwater special cases.
When establishing initial stormwater analytical dataset (network), the key is to identify
stormwater infrastructure that participates in the water collection and conveyance. This includes:
1. Stormwater inlets that allow the surface water to enter the pipe system.
2. Pipe system that conveys water but does not collect it.
3. Open channel system that both collects and conveys water.
Often, stormwater infrastructure is not organized from this perspective. For example, a typical
manhole feature class can include both manholes that are completely covered and do not allow
water to enter the stormwater system as well as manholes with grated cover that do allow entry
of the surface water. Similarly, inlet features that allow water to enter the system can be stored
in multiple feature classes.
One of the first steps in building the representative Arc Hydro stormwater infrastructure is to
identify which feature types play the three identified roles in Arc Hydro and organize those
features into the three specific feature classes (part of the same feature dataset) for further use.
This is performed to streamline the use of the tools and clearly separate data roles. The original
infrastructure data will also be augmented during the terrain processing, so keeping the original
infrastructure data separated from the “analysis” data is an important design consideration. It is
important to note that this “duplicates” some of the infrastructure data, so appropriate data
maintenance workflows need to put in place so that the data are in “sync” (through well-defined
data editing and update procedures).
Initial infrastructure data are organized into three feature classes (and one optional one):
1. HydroJunction – combined inlet features.
2. Pipe – combined closed conveyance features.
3. StreamOrig – combined open conveyance features.
4. PipeOutlet (optional) – known structures that let water onto the surface
Note that at this point a single database design (attribute) structure was not enforced for these
three feature classes – the attributes from individual contributing feature classes can be
“combined” when the features are merged. This generates an inefficient data model that can be
streamlined later. Once the improved data model is established, when moving original
infrastructure data into this model a proper attribute transformation/population should be
performed. The most important aspect of this attribute transformation is inclusion of key
identifier that allows for linking of the data between the original infrastructure database and the
Arc Hydro analytical geodatabase.
This section presents basic discussion of data quality requirements and some of the methods and
tools for their evaluation and adjustments. While the requirements are not complicated, getting
the typical data to conform to those requirements will represent the bulk of the effort in
stormwater implementation and will require an iterative data development and review process.
Experiences from this exercise should be incorporated in the data maintenance and collection
workflows so that new data comply with the requirements and do not require additional
adjustments.
Note that significant effort can go into snapping and orienting open channels, pipes, and inlets
into a topologically “clean” dataset that can be used for tracing after the initial data are imported
into the three basic stormwater feature classes. Even larger effort might be needed if complete
infrastructure data are missing and fieldwork must be performed to collect them. The level of
effort will also be end goal driven. For example, if the end requirement is to establish drainage
areas draining to outfalls from 48” pipes or higher as opposed to hydraulic modeling of the full
system for design purposes, a different data content design/evaluation/augmentation strategy can
be implemented.
There are three key elements that must be correctly represented in the final stormwater system:
1. Feature content. Line and point features representing built infrastructure need to be in
proper places (fill in the gap in missing features).
2. Connectivity.
a. Inlets need to be snapped on the from node of a pipe.
b. Pipes and streams need to be snapped onto each other.
3. Directionality. Pipes and streams need to be oriented “downstream” (for predominant
flow direction in the system – disregard potential surcharging of the system).
Figure 1 presents some of the issues with the typical dataset. Thin black lines represent lateral
pipes, thick black lines represent main pipes, blue lines represent open channels/streams, points
represent inlets, and arrows are placed at the downstream end of the linear features indicating the
direction of “flow”.
Any standard data evaluation and editing tools can be used to build a topologically clean dataset.
Ideally, these requirements would be introduced early in the data development cycle, so that the
asset data brought into this system do not have “problems”. Following is a summary of types of
issues addressed in this document (a 1m cell size DEM is assumed to be available to support the
analysis, but this is not a “hard” requirement).
1. Remove all inlets that are not connected to a pipe (unless the inlet represents a know
terrain sink).
2. Add inlets to all head pipes that do not already have an inlet.
A document focusing on typical stormwater data issues and fixes will be presented separately.
This section presents a general overview of the process.
This is a quality control step that should be performed early on when the data are received, and
their quality is uncertain. The final geometric network representing full stormwater system will
be built later on after the overland connectors are built (if necessary), so this is performed here to
do a “coarse” check on the stream and pipe data before detailed processing is started.
Note that building geometric network using snapping and simple features will modify geometry
of the underlying input data. These changes CANNOT be reversed. It is strongly suggested that
this operation is performed on the copy of the data until all the quality issues are resolved.
To build a good geometric network, initial data screening should be performed to eliminate
known issues before building the network. Some of the more glaring issues include:
1. 0 (or very short) length features.
2. Multipart features.
3. Coincident/overlapping/duplicate features.
These issues should be reviewed and fixed using any editing means. The following steps are
presented here as an example of what typical operations might include but are not exhaustive by
any means and do not include any manual editing that might be required in a production
situation. They include a mix of standard and Arc Hydro geoprocessing tools.
1. Multipart to singlepart conversion.
2. Removing of pseudonodes.
3. Splitting and connecting pipes and streams.
Review location of default junctions (QC_Net_Junctions). You will notice that they are placed
at the ends of all the lines, but not at their intersections, indicating that the pipes are not splitting
(connecting to) the streams.
Place the network flag at the most downstream stream segment and trace upstream.
Place a network “flag” on points of interest and trace upstream. This can be done interactively
using the Utility Network Analyst toolbar or through geometric network GP tools. Typical
points of interest should include major outfalls. Lack of connectivity indicates “missing”
elements in the system (missing pipe/stream segment, elements not properly snapped, and wrong
digitized direction). After the above fixes to the data, the same outlet point shows much larger
connectivity and includes many pipe sub-systems contributing to the open channel flow.
Note that later in the process we will generate “overland flow connectors” that will connect
isolated subsystems into a connected system based on DEM-derived connectivity.
3.2.2 Review and remove all inlets that are not connected to a pipe
Unconnected inlets will generate “sinks” that are not connected to anything and as a consequence
that will exclude area contributing to those features to contribute anywhere in the system. That
IS possible, so such inlets should not be deleted. In most cases though, these indicate missing
pipe information.
In a typical data scenario, several inlet features might not be connected to a pipe. Note that some
of the inlets are probably in a right place, but the connection pipes are missing. These instances
should be investigated and fixed.
This is a data augmentation step – if not performed, these pipes will not carry any flow (which
might be correct, but makes them redundant in the analytical system).
Assigning HydroID is needed only if not already performed as part of other data processing.
It is possible to generate coincident inlet features, so they will be deleted. These are cases where
two pipes have the same origination point. In a normal data QC workflow, these cases should be
identified (you can use tool “Find Identical” for that), reviewed, and edited manually if
necessary.
Stream outlets (end points of streams not connected to other streams) can happen in three
circumstances:
1. Error in data – the connecting element is missing. If that is the case, it should be noted
during initial quality control of linear features and fixed by digitizing the connector (see
option for automated fixing discussion later on).
2. The stream segment actually ends in a sink (depression) and does not flow further
downstream.
3. The stream segment ends in a pipe structure (e.g. culvert). In this case, the end of the
stream is the beginning of a pipe, and those are modeled as sinks, so effectively, this is
the same as case #2.
In this exercise, we will work under the assumption that these locations are represented as sinks
in the system (case #2 or #3). See text inserts if case #1 is to be implemented.
Several stream “outlets” are identified. They are at expected places. Few might require addition
of overland paths but in this example, they will be treated as “real” sinks.
Select stream outlets that are not represented in the HydroJunction feature class and add them to
it.
In this case none of the outlets coincide with pipe inlets and all will be added to the
HydroJunction feature class.
Inlets in the current data processing workflow are modeled as sinks (to “attract” water to flow
into them). Sinks are defined as polygons and constructed using a ~2-cell buffer around the inlet
point. Proximity of inlets to a stream thus can generate unexpected behavior. The inlet that is
in/too close to the stream will be treated as if the inlet defines a pipe divergence that is in the
stream. The consequence will be that the “sink watershed” for that inlet will include upstream
Similar issue exists when two inlets are close to each other since they will be collapsed into a
single sink polygon.
Note that in the following example, the optional search distance is set to 10m to limit the
operation time but also look at few more features than just the ones within the sink polygon
buffer distance. This can be controlled based on QC requirements.
The result provides the fields with measurement of distance (NEAR_DIST) and proximity match
layer name (NEAR_FC). Distance field will contain the distance to the closest stream or inlet
feature (value of -1 indicates that the point is NOT within a search distance from any feature).
Note that the distance is in layer units (e.g. feet), not in the “query” units (e.g. meters).
3.2.6 Check all explicit outlets that are not connected to a pipe
This is just a quality control step to identify if the pipes that are supposed to end at those outlet
locations are potentially missing. It is possible that the pipe outlet features are not explicitly
known (pipe outlets is an optional feature class), so this check might not be possible to perform.
Note that in this example we will not really review which end of pipes have and do not have an
outlet structure. We will work under the assumption that any pipe outlet needs to be connected
to the stream, either by already being on the stream or by creating an overland stream connector.
In a normal QC process, a more thorough investigation and possibly manual editing should be
performed.
The resulting feature class will have an attribute called “OnStream” where value 0 indicates that
the outlet point is not on the stream and 1 indicates that it is. To ensure that the stream segments
are split on the connection points with pipes (to ensure the consistency of the geometric
network), the streams will be split at the connector points. In case you performed manual editing
of the linear network elements, and you used “Connect” tool in geometric editing tools to “snap”
stream and pipe network elements, the following two steps are not needed since the GN editing
process will split the streams at pipe ends. Performing these two steps though will not do any
“damage” so it does not hurt to include them into the process regardless of the previous editing
effort.
Steps presented in this chapter were just some of the possible techniques for data review,
cleanup, and augmentation. They were presented here not as something that MUST be
performed or the ONLY way of performing these activities, but rather as some of the viable
options to perform them and to highlight Arc Hydro tools that were developed to support these
activities. The actual work that will need to be performed on the data will vary greatly
depending on the provenance of the data and the robustness of the data editing workflows used to
develop them. Each local dataset is unique and specific steps will have to be developed to
accommodate nuances of the specific dataset.
This section identifies individual terrain processing steps after the initial data QC, editing, and
creating has been performed. The following layers are used in the process:
# Description QC process result name Preprocessing workflow name
1 DEM demaoi
2 Pipes PipeQC Pipe
3 Streams StreamQC3 StreamInit
4 Inlets HydroJunction HydroJunction
5 Outlets SyntheticPipeOutlet PipeOutlet
Column “QC process result name” are names of the layers if you followed steps described in
chapter 3. Column “Preprocessing workflow name” has the layer names that will be used as
initial inputs in this chapter.
To ensure consistency of the layer naming in the next steps, current layers that do not match
“workflow name” in the table above will be renamed into it. While not necessary, a new folder
and geodatabase is created that contains only the relevant renamed layers. To make this as
generic as possible, there is not assumption that any previous Arc Hydro processing has been
done with the data (so the layers do not need to have HydroID, etc.). Since no specific attributes,
besides the ones generated during the process are used, all current attributes on the four feature
classes can be deleted if desired. As discussed before, it is suggested that the original unique
identifiers of the data elements are preserved to enable link back to the original feature if need
arises.
This step is used to generate artificial sinks at inlet locations to “force” the terrain to flow into
the inlets.
There are several ways to generate “sink polygons” associated with inlets. A buffer approach is
presented here. A buffer of 1.5 the cell size around the inlet is used (in this case 1.5m). The
presented buffer distance is a minimum that can be used. It is possible that such a small buffer
will not generate a “sink” big enough to resolve DEM inaccuracies around inlets and a larger
buffer might be appropriate. This can be evaluated once the sink watersheds are established. It
is possible to have a point-specific buffer distance that varies throughout the system. In that
case, establish a field in the HydroJunction feature class and populate it with the buffer distance
to be used in the buffer function. Then use that field to define distance value in the “Buffer”
function.
Assign HydroID to HydroJunction FC if necessary (if not performed somewhere earlier in the
data development process).
This step generates sink representations (vector and raster) matching underlying DEM resolution
that will be used in the processing.
Notice that several points fall into a single sink polygon (the tool message will identify if
multiple points fall within a single sink polygon and how many in total were deleted). This
happens due to the point proximity with respect to the DEM cell size and buffer distance being
used to generate sink polys. The tool will select one of the points within sink polygon and ignore
the others. While not necessarily an error, this might generate unexpected results (e.g. one inlet
collects all the water from the sink) and a detailed QC of those points is suggested (but not
needed if the end results are acceptable).
This step generates stream representations (vector and raster) matching underlying DEM
resolution that will be used in the processing.
The following section deals with evaluating the possibility that a sink point resides on a stream.
Upon review, if data need to be fixed (e.g. remove the inlet point or stream), they should be
fixed, and previous steps re-run. How many steps need to be re-run will depend on the changes
made. In this example, no fixes were made.
Note that “Input Stream Raster” is a result of the “Create Drainage Line Structure” function.
The following function is used to generate “missing” overland flow connectors (usually ends of
pipes discharging onto a surface) and connect them to the existing stream features and/or
downstream inlets/sinks.
Sink watersheds represent total directly contributing surface contributing area to a sink. They
will be the same as the sink catchment unless there is a stream draining directly into the sink.
This section presents several steps for completing the stormwater system (building final
connected network) and generating supplemental layers to facilitate stormwater delineation.
The following two steps are performed to establish a flow accumulation raster and a snap stream
raster (to facilitate precise placement of points of interest when performing watershed
delineation). These are optional steps but are recommended.
A threshold of 10,000 cells (10,000 m2) was used here. This can be changed to get more or less
dense (bigger threshold) snap density.
Once the stormwater system is built, it can be used to support different analyses. Two specific
tools are built to leverage the stormwater system data representation.
This tool allows watershed delineation along the stormwater system. Three types of watershed
delineation are possible: full delineation, direct surface contribution, and simulated inlet location
(check tool help for detailed discussion on how each delineation type operates). To use the tool:
1. Open the “Stormwater Delineation” tool.
4. Click on “OK” in the tool interface. The tool will run and delineate the watershed.
During the run, tool will generate an extensive log (message) on what it is doing. Review
it to get the familiarity with the activities and processing times during delineation
process. Process scales well, so if you pass multiple points, each point will take roughly
the same time to delineate.
A custom tool is not developed for this functionality. Standard geometric network tracing can be
easily used to identify downstream path from a point of interest. Follow this procedure:
1. Turn on “Utility Network Analyst” toolbar if necessary. The “AHStormwater” network
should already be active.
7. Use other tracing options and general ArcMap capabilities if needed, depending on your
end requirements.
8. You can also use “Trace Geometric Network” gp tool to perform the trace.
This tool establishes network element connectivity in tabular form that can be useful for specific
custom applications that need simpler connectivity representation. This form is also easy to
export to formats that can be used by different network solvers operating outside of ArcGIS
environment.
This section provides a list of tools used in the Arc Hydro stormwater data development and use
workflow. Most of the tools used in the stormwater workflows have been part of Arc Hydro
toolset for many years. The recent, stormwater specific tools are highlighted here.
Sample time of tool execution is provided for relative reference only (comparison of tool
execution on a single dataset on a single computer). Actual time will vary depending on the
complexity of the data (mostly DEM size) and the hardware used for processing. For reference,
the sample dataset had:
1. DEM: 8,149 by 16,166 cells
2. Pipe: 8,721 features
3. StreamInit: 117 features
4. Stream: 1,333 features
5. HydroJunction: 5,515 features
6. PipeOutlet: 1,028 features
All of the tools used in the workflow are available in the Arc Hydro Tools Python toolbox
(highlighted in green are the stormwater specific tools). The following toolset abbreviation is
used:
• TP: Terrain Preprocessing
• TP-DM: Terrain Preprocessing -> DEM Manipulation
The 19-step workflow is captured in tool “Stormwater Processing” (in Terrain Preprocessing
Workflows -> Stormwater toolset). Output location (based on input data) and naming of all
oputput layers is controlled by the tool and not accessible to the user.
This section provides database structures generated during the process. It follows the standard
Arc Hydro data organization and naming conventions.