Overview

One of the most frequent questions I get from developer is “What is the best way to integrate with the Utility Network”?  My answer is always the same: “The best way to integrate with the Utility Network is the way that works”. The truth is there is no one, right answer. You may prefer one strategy or API over another depending on the nature of your interface and your preferred language. This article is an overview of integration strategies and the APIs commonly used to develop behind-the-scenes interfaces and applications with the Utility Network.  If you’re interested in learning more about extending the utility network through client-side customizations I recommend you watch the “Extending the Utility Network” presentation provided by Esri during the Esri Developer’s Summit and Esri User Conference, to access this content you will need to attend the conference or get digital access to conference content afterwards.

I’ve supplied a matrix below that you can use to quickly view the API or workflow you’re most interested in. I encourage you to read the details of all the workflows and APIs before you get too far into your implementation, especially if you’re using an approach that has some known limitations.

Integration Requirement	ArcPy Python	ArcGIS API for Python Python	ArcGIS Pro SDK C#	ArcGIS Enterprise SDK C#	ArcGIS Data Interoperability FME	REST Any
Basic Transaction	✅	✅	✅	✅	✅	✅
Versioned Transaction		✅	✅	✅
Network Transaction		✅	✅	✅
Analyze Network Data	✅	✅	✅	✅
Extract Network Data	✅	✅
Import External Datasets	✅				✅

✅ Best Practice

Some Limitations

SQL

You may notice that SQL is not listed as one of the available APIs, this is because direct SQL read/write access is not available for features in a utility network dataset.  If you have a strict requirement to create SQL-based reports against the data stored in your utility network dataset, I recommend you read this Branch Versioning and SQL article which describes the best practices for working with branch versioned data using SQL or consider an alternative approach like setting up a reporting instance or data warehouse.  The ArcGIS Solutions team provides an Export Reporting Data tool as part of their Utility Data Management Support Tools that is designed to export a utility network geodatabase to a new geodatabase for reporting purposes.

Integration Requirements

Basic Transaction

This workflow should be used when the integrating the current state of the GIS with an external system like a Data Warehouse or Customer Information System (CIS). These workflows are read-only (Data Warehouse) or only interact with non-versioned features (CIS). If the interface needs to edit versioned features (Asset Management System), you will need to use a versioned feature data workflow since this will allow you to perform the edits in an isolated transaction that can be committed or rolled back without blocking other processes from reading/writing the database.

Note: If you are planning on editing large amount of data on a regular basis (hundreds of thousands of rows) you should ensure that your interface is designed to edit non-versioned classes to ensure good database health and performance.

return to top

Versioned Transaction

This workflow is primarily used when making a large volume of edits to versioned features in the GIS or when versioned edits require user validation or input before they are committed to the default version of the system like when importing the parcels for a proposed subdivision from a contractor.  If these edits result in dirty areas in your network, you must use a Network Transaction.  These the edits are made in a version where a user can review and act upon them before committing them to the system. If the edits do not require user interaction, an automated process can commit the changes to the system.

Note: Regularly making large numbers of edits against versioned classes will negatively affect database health and performance and should be avoided at all costs. As an example, instead of populating customer name directly on a service point (network point), maintain a related non-versioned table to store all the customer details.

return to top

Network Transaction

These are workflow where you need to write information to the GIS that will affect the state of the network like importing the features associated with a design from an external design tool. An example of this includes updating the open/closed status of a device, because this will affect flow in the network.  For more information refer to the Dirty area management with the Utility Network blog post. In addition to the steps required for versioned feature data, you must also validate the topology for any dirty areas created by your edits and update subnetworks that are marked as dirty.  The network will remain in an inconsistent state until all dirty areas and dirty subnetworks are resolved.  Inconsistent network prevent users from tracing the network or other systems from exporting networks.  The recommended workflow is to do the following:

1. Create a version
2. Apply all your edits to the version
3. Validate topology for the resulting new dirty areas
4. Update subnetworks on any dirty networks
5. Reconcile/post your version
6. Delete your version

return to top

Analyze Network Data

These are workflows when you need to answer questions about your data that require you to trace the network like when a CIS wants to validate customer connections to the network.  You should keep performance in mind when designing these interfaces because a single trace may take one or more seconds to complete depending on the complexity and scope of the analysis.  Executing individual traces may be suitable for incremental or near real-time interfaces but batch interfaces will require additional logic to ensure they can meet performance requirements.

return to top

Extract Network Data

These workflows involve extracting all the feature information associated with a Network for use in engineering analysis, planning, or operations. Before you develop your own extractor you should see if your vendor or a business partner has a solution that has been certified by Esri to be compatible with the ArcGIS Utility Network.

return to top

Import External Datasets

These are workflows where you are importing data into the GIS external non-Esri systems using either non-Esri formats or data that requires reprojection. This often takes the form of Shapefiles, CAD, BIM, or other spatial 2D/3D datasets into the GIS from external systems.  It is also combined with other workflows for importing versioned feature data or network feature data.

return to top

APIs

The Utility Network runs on top of ArcGIS Enterprise, and anyone who has been involved with the technology over the past decade knows that there has been massive growth in the number and capabilities of methods of integrating GIS data with other systems. In this section I’ll be describing some of the most common technologies used for integration

ArcPy

The ArcPy library is a python API developed by Esri for performing analysis using the ArcGIS Desktop and ArcGIS Pro applications and architecture. It is a library of coarse-grained data management, spatial analysis, and data interoperability tools. These APIs are only capable of interacting with the default version of a database and lack many of the fine-grained functions required to maintain a valid utility network dataset. ArcPy has coarse grained operations for reading data, but write operations are applied to the server for individual rows, meaning it not suitable for updating large sets of data.  This is different from the ArcGIS API for Python which can request and update features using collections of features, resulting in less client/server requests. Finally, because Esri Geoprocessing tools are known to have overhead with each call you should avoid calling them in a tight loop.  You can mitigate this by only using GP tools for bulk operations, only using non-gp methods and classes in tight loops, or by using one of the other APIs.

You can find additional information in the online help: ArcGIS Pro Python Reference

return to top

ArcGIS API for Python

The ArcGIS API for Python is a series of python APIs developed for interacting with ArcGIS Online and ArcGIS Enterprise services. They have access to a large library of fine-grained administrative, analytical, and data management tools. These APIs have access to most of the capabilities of the service based functionality for the Utility Network including all the major capabilities of the transactional and network model.  These APIs use feature sets to minimize the amount of client/server interactions which allows for more scalable read/write operations.

You can find additional information in the online help: ArcGIS API for Python Reference

return to top

ArcGIS Pro SDK

C# APIs developed for interacting with all forms of ArcGIS data using ArcGIS Pro. When you use this SDK to develop headless console applications you don’t have access to any of the APIs that require the application or a map.

You can find additional information in the online help: ArcGIS Pro SDK Reference

return to top

ArcGIS Enterprise SDK

C# and Java APIs developed for extending the capabilities of ArcGIS Enterprise.  It provides the most comprehensive access to the APIs for interacting with the utility network dataset but is also one of the most complicated solutions to develop and deploy.  It requires developing a custom server object extension (SOE) to expose new functionality / APIs or a server object interceptor (SOI) to inject new behaviors into existing services.  You can learn more about creating our own SOE or SOI by reading the Extending Utility Network Services with the ArcGIS Enterprise SDK blog.

You can find additional information in the online help: ArcGIS Enterprise SDK Reference

return to top

ArcGIS Data Interoperability

This extension for ArcGIS Pro and ArcGIS Enterprise include desktop and client-based tool for manipulating data. It is capable of reading/writing hundreds of different spatial and non-spatial datasets using a variety of different operating systems, you can find a full list of supported file formats here. While it supports interacting with both local dataset and service-based datasets using readers/writers it does not have access to the Utility Network or Branch Versioning APIs. You can work around this by using a REST reader/writer. This extension is built on top of Safe Software’s Feature Manipulation Engine (FME), so you can find more about the processing capabilities of FME by referring to the online help for FME

You can find additional information in the online help: ArcGIS Data Interoperability

return to top

REST

All the functionality of the Utility Network is available via the REST APIs, therefore they are the most powerful way of integrating or extending your implementation.  Directly accessing the REST APIs is a way that any language or application can integrate with ArcGIS Enterprise and Utility Network data. These APIs supply fine-grained access to all the functions needed to read/write data as well as access to all the utility network and branch versioning APIs.

In the table above, I show that you should use caution when implementing a REST interface. This is because there is no formal REST contract to compile or build against. Any interfaces built directly against the APIs will be harder to support through upgrades and can be more difficult to troubleshoot than the structured / managed APIs. Customers who have a dedicated GIS developer will find this to be a powerful way of developing interfaces.  IT departments that support many applications without a dedicated GIS developer may have more difficultly keeping up with the technology required to develop and maintain these interfaces.

You can find additional information in the online help: ArcGIS Rest APIs Documentation

return to top

Conclusion

Hopefully, you’ve found your integration requirement(s) and API, I recommend that you bookmark this blog and revisit it as you are developing your solutions to ensure you’re following best practices. Once you’ve started with the API, I recommend that you begin searching for examples from conferences, blog posts, videos, or other online resources to get you started. Finally, over the coming weeks I will be releasing additional blog posts providing specific examples of how to architect interfaces between the utility network many of the major systems at a utility: customer information systems, asset management systems, outage management systems, and more!