Cache your map and image services in the cloud

In our last blog post, we talked about running ArcGIS Enterprise in the cloud. Now, let’s take a look at a specific example of using cloud-native functionality to improve an existing workflow: tile caching.

Caching map and image services is a time-honored approach to improving performance for basemaps and other map layers where the underlying data changes infrequently. We have multiple options today for how to cache our data. Raster tile caches are the traditional option, going back for over a decade; more recently, we introduced the vector tile cache option.

We recommend that when possible, you should cache vector data (point, line and polygons) using vector tiles and cache raster data (imagery or hillshades) using raster tiles. However, you cannot migrate your vector data-based services and apps to vector tiles overnight, so let’s focus on the traditional raster tile cache, which can be used with both map and image services.

By creating a raster tile cache, you pre-generate or “pre-cook” all of the images for a given area (extent) and scale. Map caches that cover large areas at large scales (zoomed in very close to the ground) can become big very quickly, easily going into tens or hundreds of gigabytes. For organizations with several global cached basemaps, it’s not uncommon to have terabytes worth of map caches.

So, how do we store all this data? Traditionally, data like this would be stored on a file server or a dedicated NAS device. But NAS devices can often be expensive to purchase and require ongoing maintenance. By comparison, cloud-native storage often costs less than USD $0.02 – $0.03 per gigabyte per month, and the underlying storage is fully managed with no need for scaling out hardware or tuning as storage demands increase or fluctuate.

Cloud storage support in ArcGIS Enterprise

ArcGIS Enterprise has supported cloud storage for map and image caches for several releases:

ArcGIS Enterprise 10.5

• added support for serving out caches from Amazon S3 buckets

ArcGIS Enterprise 10.5.1

• added support for serving out caches from Microsoft Azure Storage
• added support for Export Map and WMTS operations for both Amazon S3 and Azure Storage

ArcGIS Enterprise 10.6

• added support for generating caches directly into cloud storage

ArcGIS Enterprise 10.6.1

• added support for export and import cache workflows
• added support for taking caches offline when stored in cloud storage
• added support for Alibaba Cloud OSS and Huawei Cloud OBS cloud storage

Get started with cloud storage

Now, we’ll describe how to make use of cloud storage as part of several key caching workflows.

To work with a cloud store as a cache directory, you first register the cloud store as a data source with your ArcGIS Server site.  To take advantage of cloud stores, your ArcGIS Enterprise deployment itself should be running on infrastructure within that same cloud region. While it is technically possible to use cloud storage from infrastructure running elsewhere, this impedes performance and is not an officially supported deployment pattern.

Here is a snapshot of the steps to register a cloud store. You can find the full details in our cloud caching documentation.

The key here is to ensure that the cloud store being registered – that is, your S3 bucket, Azure blob store, Alibaba OBS bucket, or Huawei Cloud OSS bucket – has a folder named arcgiscache. That folder name is required for registering a cloud store as a cache directory with ArcGIS Server.

Once registered as a cache directory, all the management clients such as ArcGIS Pro, ArcMap and ArcGIS Server Manager and server caching tools will work with it regardless of its type, providing a seamless user experience independent of the location of the cache.

Raster tile cache cloud storage in action

Let’s take an example. When creating a cached Map Image Layer using ArcGIS Pro, you can set the Layer configuration to use the cloud store cache directory for its cache content (see the snapshot below). The workflows for cache management – such as cache generation/recreation/deletion using full/custom extents, or area of interest – are supported using the Manage Map Server Cache Tiles geoprocessing tool.

The notable difference when working with cloud store directories is that the cache bundles will be generated in the local cache directory first and then uploaded to the cloud store directory regardless of the number of machines participating in the Server site. Hence it will be important to allocate enough disk space to the temporary cache folders.

Once the cache is copied or generated in place in the cloud store, it can be kept in sync with data updates using the Manage Map Server Cache Tiles geoprocessing tool.

Let’s say you are managing a cached image service for the state of California, and you have received updated imagery content for the area affected by the Carr Fire. You would first update the imagery content of your mosaic dataset and then compute dirty areas of the updated imagery. The output of this tool should be provided as an input to the Manage Map Server Cache Tiles tool to update cache tiles at a detailed scale for the areas with updated imagery content. This will reduce the time taken to update the cache with the new imagery content. Your caches can then be exported for offline usage using the Export Tiles REST API operation or the Export Map Server Cache Tiles geoprocessing tool.

Using cloud storage for raster tile caches is a great example of how cloud-native functionality can help you go beyond what you could easily do with on-premises infrastructure. We hope this will help alleviate disk space constraints, while getting excellent performance (with native high availability built in) when hosting large raster tile caches for your map and image layers.