There have been many deep learning capabilities that were added to the Image Analyst extension for ArcGIS Pro over the last few releases. The emphasis for the ArcGIS Pro 3.1 release was to improve user workflows and to increase productivity. Here are our top three new deep learning features in the 3.1 release:

• Pixel classification using time-series imagery
• Pixel classification on features with image attachments
• Interactive segmentation for labeling

Want to learn more? Continue reading.

Pixel classification using time-series imagery

In this release, we added support for the Pixel-Set Encoders and Temporal Self-Attention (PSETAE) architecture to train models for time series classification. PSETAE is a transformer based deep learning model, originally developed by Garnot et al for agricultural parcel classification into different crop types using time-series satellite images.

To support this workflow, we have added support for multidimensional inputs and the PSETAE architecture for the following tools:

• Export Training Data for Deep Learning tool – to support multidimensional dataset (as a CRF format or mosaic dataset).
• Train Deep Learning Model tool – to support the PSETAE architecture.
• Classify Pixels using Deep Learning tool – to support multidimensional inputs (as a CRF format or mosaic dataset) and the PSETAE architecture.

Example workflow: Crop classification on time-series imagery

We have aerial images for different time periods, and we want to perform crop classification. Our first step is to prepare the data for training, where we will use the Export Training Data for Deep Learning tool. The input time-series imagery can be either be a composite of rasters or a multi-dimensional raster, from the required time periods. To create training labels for a multi-dimensional raster, use the Label objects for Deep Learning tool (available on the Imagery tab within the Image Classification group). Pixels are labelled into different classes, based on the available information. The figure below shows labeling of different crop types. The tool will write out stacked images as training chips. The training chips will have all the information from all the time slices.

Once we have our training data, we can run the Training Deep Learning tool with the new PSETAE model architecture. Finally, we can use the Classify Pixels using Deep Learning tool to perform the inferencing. The Classify Pixels using Deep Learning tool also supports multi-dimensional datasets, which can be a CRF or a mosaic dataset.

The final output will be a classified raster of the different crop types.

Pixel classification on features with image attachments

Features with attachments is an information model that spans across the ArcGIS platform. Many of the Esri apps, such as ArcGIS QuickCapture and ArcGIS Survey 123, use this approach to collect data and photographs. We enhanced the Classify Pixel using Deep Learning tool to allow features classes with image attachments as an input. The output will be a feature class with images, whose pixels are classified or transformed as per the input deep learning model. You can learn more about adding attachments to a feature in this video.

Example: Blurring sensitive information on features with image attachments

Let’s consider that we collect street view images for various purposes, such as asset inspections, landscape design, or traffic management. This information can be collected using car mounted cameras. However, the images may contain human faces and vehicle license plates. For privacy purposes, we want all faces and license plates to be blurred out before allowing this data to be public. We can use  ArcGIS pre-trained models (Face Blurring and License Plate Blurring) to address these privacy concerns.

We will use the GPS locations to create points for each image and attached the images to the corresponding points features. And of course we will make sure to blur the faces and license plates in the image attachments.

The image above shows an example of a point features with image attachments. We will use this point feature class as an input in the Classify Pixels using Deep Learning tool.  The final output is a feature class with images having faces and license plates blurred out. We can use ModelBuilder or a Python script to streamline all the steps together in the workflow.

This is just one example of how pixel classification using features with image attachments can help your workflows. We are confident that this enhancement can help your workflows too.

Interactive segmentation for labeling

And finally, another productivity enhancement we like a lot is interactive segmentation for labeling. We like this enhancement because data labeling is one of the most tedious and time-consuming aspects of deep learning workflows. Organizations spends large amounts of resources to get data labeled. In ArcGIS Pro 3.1, we have added a new feature called Auto Detect in our Label Objects for Deep Learning pane. This feature can detect objects and automatically draw the polygons or segments on the target image. You can just select your label and click on the features.

In below animation, we are clicking on the houses and pools, and the tool is creating polygons around the objects automatically.

What do you think about these enhancements? Do you have any use cases that can benefit from these new features? Please share your ideas to improve deep learning workflows in ArcGIS.