Are you sure Intersect is the right tool for the job?

Are you sure Intersect is the right tool for the job?

UPDATE

ArcGIS Pro 1.0 introduced a PairwiseIntersect tool which emulates the pairwise tool discussed in this blog post.
ArcGIS Pro 1.1 has an additional pairwise tool, Pairwise Dissolve.
By default, starting in ArcGIS Pro 1.1, both the PairwiseIntersect and PairwiseDissolve tools run in parallel mode. This will allow these tools to distribute the work to all (or a portion of) the logical cores on the machine. The performance benefit of parallel processing varies from tool to tool and depends on the input data being processed.

Please see the following for more info: PairwiseIntersect and PairwiseDissolve

(Note: There are no system Pairwise tools provided in 10.x. Continue to use the provided methodology and scripts included with this post.)

=========================================================

I often talk with people using ArcGIS Geoprocessing who find themselves surprised about the amount of time the Intersect tool takes to run, or confused about the output results.  Much of the time this confusion comes from a misperception about what tool to use for the analysis, or from a lack of understanding of what the Intersect tool does.

Here’s an example… A user contacted me about the Intersect tool because the tool would run for hours and then fail. They perceived their data as small and really thought Intersect would only take a few minutes.

My first question, and one you should ask too is, “What do you really want for a result?”

The answer in this, and many cases was “I want to know how much of each feature in input 1 is in each feature in input 2.”

Cool!  That means Intersect is not the right tool to use!

Intersect does not compare each feature in the first input to each feature in the second input.  It works in a much more complex, thorough way. The intersect tool is for creating a topologically correct fabric across the entire extent of the dataset and return those newly created features that were created due to overlap from a feature in each input.  The Intersect tool throws all the features from all the inputs into one bucket, and passes this to the underlying topology engine so it can create this topological fabric. This means all relationships, between all features, including those relationships within each input are determined and new features are created for every interaction. The Intersect tool then selects just those features from this topologically correct fabric.

So, for the Intersect tool if you have 2 inputs the output will only contain those features that were created where a feature from each of the 2 inputs overlapped.  If you have 100 inputs the output will only contain those features that were created where a feature from each of the 100 inputs overlapped.

Here is what looks like a very simple example of intersecting one polygon layer with 10 overlapping polygons with a single polygon from a second feature class (features are labeled with their OID):

Input 1 (10 input polys that overlap)

Input2 (One polygon)

Intersect output will contain new features created from the intersection of the overlapping polygons in Input1 and their intersection with the polygon in Input2:

As you can see, there are now a lot more polygons in the output (167!) than there were in the inputs combined.  Imagine if Input1 was actually complex:

Ouch!

Using the output of Intersect you now have to perform further analysis to put together polygons that were broken apart in Input1 due to their intersections with one another. Seems like a lot of work to figure out how much of the polygon with OID 9 in Input1 intersects with the polygon in Input2 right?

So what should they, and you, be doing if you want to know how much of each input feature intersects with each feature in the second input?  Here are two possible workflows, one good, and one much better, that will give you exactly what you want, and may get you the results of your analysis much faster.

Good… Clip and Transfer Attributes workflow

This is a simple approach that uses existing tools and can be setup by anyone familiar with scripting. The only ‘advanced’ part, is the use of a cursor, but it should be fairly easy to understand.

You add code to transfer as many of the attributes from the second input as you want. If you want this method to go a little faster, you could play with sending the intermediate data to in_memory featureclasses.  Trouble is, you’ll have to manage how many outputs you send to in_memory if your data is very large.  You may use up all your memory storing the intermediate data and cause the workflow to fail.

There are other possible problems with this workflow. The main ones are SelectLayerByLocation ‘INTERSECT’ doesn’t use the exact same rules to select features as Intersect does (see here) and creating and managing all that intermediate data is painful and slow.

So, here’s a much better way to do this…

Better… Intersect geometry method

We can get rid of the creation and maintenance of all that intermediate data, use a method that mimics the Intersect tool, and make this all run tons faster on larger, more complex data.

Let’s use a geometry method instead!

Here is a simplified code snippet explaining the core basics of the workflow:

For a fully functional (I hope!) script tool and script that you can try out, download the Pairwise Intersect Tool here.

Taking a look at the same example I used above except this time I use the PairWiseIntersect tool:

That’s more like it!

In testing we’ve done, the “Better” method is multitudes faster with large and/or complex data and it also gets through cases the Intersect Overlay tool appears to have trouble with (and for the most part these ‘trouble’ cases do not make sense to process using the Intersect tool).

Many thanks to Jason Pardy for reviewing the code and blog contents.