In ArcGIS Enterprise 11.4, the ArcGIS Data Store tile cache data store is deprecated and replaced by the object store. Consequently, the object store is now a component of a base ArcGIS Enterprise deployment.
What does deprecated mean? Deprecated means we no longer recommend using the tile cache data store. Existing hosted scene layers with their caches stored there will continue to function, but organizations are encouraged to migrate to the object store after upgrading. In the near future, the tile cache data store will be fully removed and no longer supported.
The object store is the new home for hosted scene layer caches in ArcGIS Enterprise. All newly published hosted scene layers will have their caches written to the object store. It also stores data for hosted 3D tiles layers and cached query responses for hosted feature layers. This change consolidates multiple functions into a single data store, making data storage more efficient.
Note: The ArcGIS Data Store tile cache data store only stores hosted scene layers caches. Other layer types, such as map image layers, imagery layers, tile layers, and elevation layers, are not impacted by its deprecation.
This blog discusses the purpose of the object store, the available architectural options, and how organizations using ArcGIS Enterprise 11.4 can migrate existing hosted scene layer caches to it today.
Quick links
Use the links below to jump to a section of interest:
- The purpose of the object store
- Architectural flexibility for the object store
- How do I determine which data stores I have?
- Identify the scene services eligible for migration
- Performing the migration
- After you migrate
The purpose of the object store
Object storage solutions are designed for storing large volumes of unstructured data. In ArcGIS Enterprise 11.0, the ArcGIS Data Store was enhanced to include an option for configuring an object store. Through the 11.3 release, its primary purpose was to store cached query responses for hosted feature layers, improving their performance and scalability in ArcGIS Enterprise.
With the release of ArcGIS Enterprise 11.4, the purpose of the object store expanded, becoming the new home for hosted scene layer caches and other hosted 3D layer types, such as the new hosted 3D tiles layer. This means that in addition to boosting the performance of your organization’s most in-demand hosted feature layers, the object store also stores caches in support of hosted scene layers and other 3D layer types.
Next, we will dive into some more specifics about the object store, highlighting the option for cloud deployments to take advantage of more cloud services.
Architectural flexibility for the object store
Beginning with ArcGIS Enterprise 11.4, organizations running ArcGIS Enterprise in the Amazon Web Services (AWS) or Microsoft Azure cloud can use a cloud object storage service in place of the ArcGIS Data Store object store.
The following diagram represents these two options:
The ArcGIS Enterprise portal’s hosting server only supports having one object store registered. It can be provided by the ArcGIS Data Store object store or a cloud object storage service.
The next two sections describe these options in greater detail, beginning with the ArcGIS Data Store.
Using the ArcGIS Data Store object store
ArcGIS Enterprise deployments installed on-premises, or running on clouds other than AWS or Microsoft Azure, will use the ArcGIS Data Store object store. This type of ArcGIS Data Store supports a single-machine mode or, for customers who require a highly available and scalable object store, it supports deploying in cluster mode. Cluster mode requires a minimum of three machines.
If your organization anticipates publishing many hosted scene layers or hosted 3D tiles layers, deploy the ArcGIS Data Store object store in cluster mode. This allows for horizontal scaling and protects against single points of failure by duplicating data across multiple machines.
The ArcGIS Data Store object store also requires large amounts of memory and disk space. For this reason, it is recommended to install it on its own machine, separate from other data store types or ArcGIS software.
For instructions to install and configure the ArcGIS Data Store object store, see the installation guide appropriate for your operating system:
Next, let’s discuss the alternative option: using a cloud object store.
Using a cloud object store
Cloud services are optimized for performance, scalability, resiliency, and reliability across multiple regions. In ArcGIS Enterprise 11.4, two cloud object storage services are supported: Amazon S3 and Azure Blob Storage.
Organizations running ArcGIS Enterprise 11.4 on AWS or Azure can register a cloud object store with their hosting server, as an alternative to the ArcGIS Data Store object store. This option serves same purpose but leverages a cloud service instead of local disk storage. Cloud object storage services deliver superior scalability, cost efficiency, and high availability, making them a smart choice for data storage.
Refer to the documentation for steps on configuring a cloud object store as the hosting server’s object store:
- Use the ArcGIS API for Python to Add an Object Store to an ArcGIS Enterprise Deployment (AWS or Azure)
- Use the ArcGIS Server Administrator Directory to Configure the object store for ArcGIS Enterprise on Amazon Web Services
- ArcGIS REST API doc: Add an Object Store to an ArcGIS Enterprise Deployment (AWS or Azure)
- Use ArcGIS Enterprise Cloud Builder for Microsoft Azure to Add an object store to an ArcGIS Enterprise deployment on Microsoft Azure
Note: This cloud storage location should not be used for storing backup files or any user managed data, such as caches for map and image services. Instead, register and maintain a separate cloud storage location for those purposes.
How do I determine which data stores I have?
If you’re the ArcGIS Enterprise administrator, you likely already know what data stores are configured for your deployment. But maybe you inherited the job of ArcGIS Enterprise administrator recently. In such cases, you can verify whether the hosting server has the tile cache data store and object store registered using ArcGIS Server Manager.
Checking for the object store
To check to see if your hosting server has an object store registered, sign in to ArcGIS Server Manager for the hosting server and click Site > Data Stores. If an entry shows Managed Object for the type, then your deployment has it.
You can verify that the hosting server and the object store can communicate with one another by checking the box next to that data store entry and clicking Validate. If all is well, a green check mark appears in the Status column.
The file name for the object store identifies the type of object store. If the file name for the Managed Object entry shows ArcGIS_Data_Store, the deployment uses the ArcGIS Data Store. If any other file name is listed, it is using a cloud object store. Again, using a cloud object store is only recommended for deployments running in the cloud.
If the hosting server has no managed object store registered, use the information and links shared in the previous sections to add one.
Checking for the tile cache data store
Next, check for the presence of the tile cache data store. When you log in to ArcGIS Server Manager, you’ll see a Tile Cache data store type listed on the Site > Data Stores page if the deployment has a tile cache data store.
If the tile cache data store is not listed as a registered data store with the hosting server, organization members could not publish hosted scene layers. Therefore, you will not need to migrate any scene layer caches. However, if the deployment does have a tile cache data store, you likely do need to migrate. Validate that the tile cache data store can communicate with the hosting server, then proceed with the migration.
Migrating from the tile cache data store
After upgrading to ArcGIS Enterprise 11.4, administrators should verify the hosting server has the object store configured. This will allow new hosted scene layers and hosted 3D tiles layers to be published. In addition, it will prepare them to migrate hosted scene layer caches from the tile cache data store to the object store.
Migrating to the object store is important, because ArcGIS Enterprise 11.5 will be the last release to support the tile cache data store. Following that release, the tile cache data store will be fully removed, and all hosted scene layer caches must be stored in the object store. Use the specialized tooling included in the ArcGIS Enterprise 11.4 release to migrate all eligible caches to the object store.
How can you verify whether you have caches to migrate? Let’s talk about that next.
Identify the scene services eligible for migration
It is easy to determine which scene services have their caches stored in the tile cache data store. This is thanks to the new MigrateSceneServices utility, which is included as part of ArcGIS Enterprise 11.4. This utility, which comes installed with ArcGIS Server, has two modes: analysis mode and execution mode.
When run in analysis mode, the utility provides a list of all services that are eligible for migration. It lists details about all hosted scene services that have caches in the tile cache data store. These details include the service name and the estimated size of each cache in megabytes. The size estimate helps you ensure that you have ample disk space and memory on the object store machine if you’re using the ArcGIS Data Store object store. (If you use a cloud object store, such as Amazon S3 or Azure Blob Storage, disk space isn’t a concern.)
Knowing the cache sizes also helps you decide when to run the migration and whether to migrate certain services separately. For example, if all the caches are gigabytes in size, migrate them when the system is not under heavy load. If you have a few caches that are terabytes in size, migrate those separately, perhaps outside of normal business hours.
When caches are migrated, you may see a spike in CPU on the machines running the tile cache data store and object store. Because of this, we recommend you perform the migration outside of business hours to minimize resource contention-related issues. This is especially important if you have caches that are extremely large.
The following is a screenshot of running the MigrateSceneServices utility in analysis mode:
If running the utility in analysis mode reports there are no services to migrate, then you can proceed with removing the tile cache data store. If the utility returns a list of services, your deployment has caches to migrate.
At this point, ensure you have a full, usable backup of the ArcGIS Enterprise deployment. This will allow you to restore your system from a backup if any unexpected issues arise.
Also, test your hosted scene layers to ensure they work as expected before migrating their caches. Otherwise, it will be difficult to pin down the cause of the issue if a scene layer doesn’t work as expected after the migration.
Now you can proceed with the migration.
Performing the migration
The MigrateSceneServices utility reports information about scene services prior to migration, when run in analysis mode. To perform the migration, you can use the utility in execution mode. Alternatively, scene layer owners can migrate each of their hosted scene layers from the item’s page in the Enterprise portal.
No matter which method you use, there will be little to no service downtime. The hosted scene layer remains accessible during the migration.
We will discuss both approaches in greater detail below.
Migrate caches in bulk using the utility
To perform the scene cache migration using the MigrateSceneService utility, run it in execution mode. When run in execution mode, the utility iterates through the eligible services, one at a time. It migrates each service’s cache to the object store and then deletes the cache for that service from the tile cache data store.
Using the utility to migrate the caches means you can…
- Migrate all, multiple, or even single scene service caches at a time, because the utility takes scene service names as input.
- Schedule the migration using tools on the hosting server.
- Perform the migration as the Enterprise administrator.
- Adjust the logging settings, including the log location and level of detail.
Other benefits to using the utility include:
- It reduces effort on your part when your organization has many hosted scene service caches to migrate.
- You’re already running it to analyze what to migrate and to ensure no caches remain in the tile cache data store after you complete the migration process. That means you’ll use a single tool and won’t have to switch environments.
Once more, hosted scene layers will remain available while the migration is in progress. At the end of a successful migration, there will only be a brief moment of downtime as the service updates to point to the newly copied cache in the object store.
To learn about utility syntax and how to run it, review the MigrateSceneServices utility documentation.
Migrate caches for individual layers from the Enterprise portal
Alternatively, you can migrate caches by initiating the migration from the Settings tab of the hosted scene layer’s item page in the Enterprise portal. Migrating caches from the item page is a good approach if you do not have many services to migrate.
The option to migrate the cache is present on the item’s Settings tab if both of the following are true:
- You sign in to the Enterprise portal as the owner of the hosted feature layer.
- The hosted scene layer’s cache has not been migrated yet.
If the option to migrate the cache is not shown, verify you are the item owner. Although this approach does not create a log file containing migration information, it does allow you to view the progress of the migration. Clicking the Migrate Cache button initiates the process, and you can view the job status and its progress, as shown below.
To learn more about migrating caches from the Enterprise portal, refer to the Manage scene layers documentation.
Things to check if migration isn’t successful
Sometimes, things don’t go as expected. If any service caches fail to migrate successfully, consider these tips:
- Run the utility in execution mode: The utility supports logging, which can be helpful for troubleshooting. If any services fail, run the utility in execution mode with the log level set to trace. This will provide you with the most detailed information. Additionally, use the optional select parameter to choose one or more specific services to migrate at a given time. Rest assured, re-running the utility will not result in any duplicate files in the object store.
- Verify server URL: When using the utility, provide the URL of the hosting server site using either the Web Adaptor URL or third-party network load balancer URL. At this time, using a URL that bypasses the Web Adaptor and uses the fully qualified domain (FQDN) of the machine and port directly will not work.
- Verify credentials: When migrating caches from the Enterprise portal, you must be signed in as the item owner. The Migrate Cache option is not available to other users at this time. When using the migration utility, provide the username of a built-in account that is a member of the default Administrator role. Using an account from an organization-specific identity store or the primary site administrator credentials for ArcGIS Server will not work.
- Adjust the amount of memory allocated to the object store: If the hosting server’s object store is provided by the ArcGIS Data Store, and large scene caches are failing to migrate, you may need to increase the heap size of the object store. This adjustment will provide more memory to the object store, which is essential when migrating (or publishing) large hosted scene layer caches. To verify the current heap size that is set for the object store, use the describedatastore utility.
Should these tips fail to resolve any issues you encounter, contact Esri Technical Support.
After you migrate
You’ve migrated all hosted scene layer caches to the object store. Now you can complete post-migration testing and clean up.
Verify that each hosted scene layer continues to function as expected
To ensure the layers are working as expected after the migration, test each hosted scene layer. Use the same verification process that you used before you migrated. For example, if you checked the hosted scene layer functionality by interacting with it in Scene Viewer before migrating, perform that same test after migration. This way, you can feel confident that the service functions the same before and after migration, because you’re making an apples-to-apples comparison.
Confirm that the tile cache data store contains no caches
You must confirm that your deployment no longer relies on the deprecated tile cache data store. To do this, an organization administrator can run the MigrateSceneServices utility in analysis mode. If the utility returns any services, migrate those now. If the utility reports that zero services are eligible for migration, then all caches have been migrated.
Here’s an example of what you the utility returns when no migratable services exist:
With all hosted scene layer caches migrated and tested, you can remove the tile cache data store from your deployment.
Removing the tile cache data store
After verifying that the tile cache data store is no longer in use, disable it. To do this, the ArcGIS Data Store administrator runs the disabledatastore utility on a tile cache data store machine. Running the utility on any machine in the tile cache data store disables the entire tile cache data store. For example, if you deployed the tile cache data store in cluster mode and it runs on three machines, you only have to run the utility from one of those machines.
The disabledatastore utility is installed with ArcGIS Data Store 11.4. It unregisters the tile cache data store from the hosting server and deletes all relevant files. This frees up resources on the machines where the tile cache data store is installed.
If the tile cache data store is installed on dedicated machines separate from other ArcGIS Data Store types, you may be able to uninstall it. This will allow you to repurpose the machines or, in the case of virtual machines, delete them.
To learn about when and how you can remove the tile cache data store, refer to this technical article: Remove the ArcGIS Data Store tile cache data store from an ArcGIS Enterprise deployment.
In summary…
The object store provides ArcGIS Enterprise 11.4 deployments with storage for hosted scene layer caches, hosted 3D tiles layer caches, and more. Its functionality replaces the deprecated tile cache data store, and organizations should migrate all hosted scene layer caches to it after upgrading to 11.4. We’ve introduced migration tooling in the 11.4 release of ArcGIS Enterprise to ease this transition, and we hope that this blog has provided a comprehensive overview about the migration workflow for your reference.
If you have questions about the migration to the object store, please post them to the ArcGIS Enterprise Community.
Commenting is not enabled for this article.