Authors/Contributors: Caroline Wright, Eva Mui, Garima Tiwari, Gayathri Vijay, Jeff Smith, Jonathan Quinn, Laurence Clinton, Markus Walker, Michelle Bush, Moginraj Mohandas, Nidhi Arora, Shreyas Shinde, Subrat Bora, Trevor Seaton, Vaibhav Singh
Thank you to everyone who attended our Kubernetes sessions at Dev Summit 2021! While we missed being able to see everyone in person at Palm Springs this year, we enjoyed talking to such a highly engaged audience. We were happy to showcase our work to build ArcGIS Enterprise on Kubernetes, the cloud native architecture which enhances scalability, resilience, and manageability for ArcGIS Enterprise customers who choose to adopt this new deployment option. We received many questions from you — too many to respond to in the time allotted — so this blog is our chance to respond to them all.
By the way, if you registered for Dev Summit 2021 you should have access to the recorded sessions on the conference platform for 30 days. And on our Esri Events Youtube channel you will find recordings from the summit plenary sessions and technology demonstrations.
Please reach out to your Esri representative for more information. Thanks again, from the ArcGIS Enterprise team!
General
Q: What’s in the initial launch?
A: This is a complete architectural redesign, with new technology being integrated into this deployment option for ArcGIS Enterprise. To achieve our goals of scalability, resilience and manageability, and most importantly to meet our quality standards, we have had to make some scope tradeoffs. At launch, functionality will be similar to a base deployment of ArcGIS Enterprise, with the portal experience, support for publishing, data analysis, hosted data, federating ArcGIS Server and Image Server, and more. However, we have had to defer support for certain apps or extensions, customizations, the ability to federate some server roles, and more. This will be documented by the time we launch, and we encourage you to reach out to your Esri representative if you’re considering Kubernetes and need more details. ArcGIS Enterprise remains one product, and our intent for the Kubernetes deployment option is to provide the same GIS capabilities you get with Windows and Linux.
Q: When will ArcGIS Enterprise on Kubernetes launch?
A: The launch is planned for May 2021, as part of the ArcGIS 2021 release in Q2.
Q: How can we get involved with testing the software before launch?
A: Please reach out to your Esri representative, if you have an active Enterprise Agreement and are interested in deploying the software into your Kubernetes environment.
Q: How can I learn more about Kubernetes?
A: You can think about learning Kubernetes as similar to becoming familiar with a new operating system. Our team can recommend several places to look for help. Cloud providers such as AWS or Azure have learning paths related to cloud native architecture and related services and tools, with extensive courses available. The Cloud Native Computing Foundation (CNCF) is a great resource. Also, check out kubernetes.io where you will find plenty of learning resources and tutorials. For ArcGIS Enterprise on Kubernetes specifically, we are developing resources and will post them to our site at launch.
Q: How will ArcGIS Enterprise on Kubernetes be licensed or priced?
A: Eligible customers will require an active Enterprise Agreement with Esri, and will need their own Kubernetes environment that meets our system requirements. In addition you will need to procure ArcGIS Server (ECP or PRVC) and portal user authorization (JSON) license files from MyEsri, just like today with Windows or Linux, in order to create an organization. Please see your Esri representative for more about eligibility details.
Q: How might the pricing and licensing change in the future, for ArcGIS Enterprise on Kubernetes?
A: At launch, our eligibility expectations are simple. We need to ensure a licensing arrangement with Esri and that our customers can receive the support they need, and we’ve determined that an EA is appropriate to that end. We are evaluating licensing and access alternatives and will share these plans when they are available. We do not see significant changes to eligibility or pricing at least over the coming year.
Q: How will ArcGIS Enterprise on Kubernetes compare to other deployment options in terms of expected cost?
A: It will depend on your individual ArcGIS Enterprise deployment, overall organization’s IT strategy, and architecture vision. A medium to large deployment with High Availability / Disaster Recovery requirements and multiple servers may see better return on investment with Kubernetes, versus what would be achieved with smaller deployments.
Architecture and Data
Q: How will physical servers or virtual machines be represented in the new architecture?
A: ArcGIS Enterprise on Kubernetes leverages the modern paradigm of containerization to build, maintain and scale applications at ease. Containers package together a piece of software with everything it needs to run, such as libraries and dependencies. Kubernetes orchestrates container workloads by placing them into basic deployable object called pods, which helps run the a containers with assigned storage and resources. The portal or services experience for GIS users remains similar to that on Windows or Linux. For more details about how the Kubernetes architecture is different from one based on physical servers or virtual machines, please review the documentation at kubernetes.io for further details.
Q: What Kubernetes engines will you support at the initial launch, and what other engines are you expecting to include?
A: When we launch we will support the Amazon Web Services Elastic Kubernetes Service (EKS), Microsoft Azure Kubernetes Service (AKS), and Red Hat OpenShift. Our roadmap includes expanding to more engines, including GCP, vSphere, and others, as quickly as possible.
Q: How did you arrive at the three supported Kubernetes engines at launch? What is your criteria for expanding to other Kubernetes engines?
A: Early on we talked with customers and partners that expressed a specific need for cloud native architectures. We decided to target a limited number of Kubernetes engines that were well represented among our customers, including those already working with the technology or those who planned to adopt it early. We wanted to include both cloud and on-premises patterns.
Q: What enterprise databases will be supported with ArcGIS Enterprise on Kubernetes?
A: At launch, we will support both PostgreSQL and Microsoft SQL Server databases as registered enterprise geodatabases.
Q: Will the new architecture be inclusive of components such as portal admin, server admin, and rest/sharing interfaces?
A: There will be several endpoints provided. One will help you manage environment logging, security, licensing and so forth. Another will help you manage sharing, and a third endpoint will be for geospatial services, wherever your services are deployed. In general we are preserving the REST API to maintain consistency for administrators, even with the new user interface available (such as the Enterprise Manager).
Q: How will the new architecture support multiple tenants or deployments within my organization?
A: Kubernetes enables this pattern through the use of separate namespaces. Each deployment will be installed into a unique namespace with its own portal capabilities and services. Multiple namespaces can exist within a single Kubernetes cluster. In this way you could have the functionality of multiple Enterprise portals, while still taking advantage of centralized resource management and infrastructure scaling.
Q: What restrictions are there for using Oracle databases or data files in ArcGIS Enterprise on Kubernetes, at the launch and beyond?
A: We will not be supporting Oracle databases out of the box at initial launch. This means you cannot register an Oracle database directly with your Kubernetes deployment. But you can federate an ArcGIS Server site that includes services backed by data from Oracle with the portal deployed on Kubernetes. Our internal container distribution is Ubuntu and Oracle has not committed to supporting that combination, so we ask that you talk with your Oracle representative about this.
Q: How are content files stored in ArcGIS Enterprise on Kubernetes?
A: Content files uploaded or added to the portal are stored in a hosted object store. The object store writes its files to persistent volumes configured in the Kubernetes cluster.
Q: Can ArcGIS on Kubernetes be configured to work with only certain components, such as ArcGIS Server?
A: You cannot choose which components to deploy, but you can choose which functionality to use. An ArcGIS Enterprise on Kubernetes deployment includes the functionality similar to an ArcGIS Enterprise base deployment and includes an Enterprise portal as a result. The architecture of ArcGIS Enterprise on Kubernetes allows you to federate a separate ArcGIS Server site (Windows or Linux) with your Kubernetes organization.
Q: What more can you tell me about security, blob storage, or resource management?
A: ArcGIS Enterprise on Kubernetes doesn’t require Active Directory but you can connect it to AD, as your identity provider for users within your deployment. For storage, it uses blob storage serviced by your cloud provider or on-premises vendor and can be managed using Enterprise Manager or the API. You can manage resources and adjust them to achieve the service level appropriate for you. You can manage individual microservices, but they are all necessary to provide the functionality of a base deployment of ArcGIS Enterprise.
Q: How are ArcSOC processes implemented with shared or dedicated instances, as services are provided?
A: Services such as map services, along with extensions, and geoprocessing system and utility services do have ArcSOC processes embedded into their respective pod(s). Shared map services run inside a special pod, a shared map server that hosts all shared instance map services. This pod contains ArcSOC processes that are used to power the GIS functionality for the services. Thus, one pod or a set of replicas can essentially host any number of your shared map services. In the case where you need a dedicated pod to run a map service, it will have its own ArcSOC process that runs inside the dedicated pod that is only serving out requests for the one service. All of your System and Utility services are also dedicated pods running to fulfill certain capabilities in the deployment. There are no ArcSOC processes running for the Shared Feature Server pod (or its replicas), which hosts all your Hosted Feature services.
Capabilities and Workflows
Q: Can I publish map services or image services with ArcGIS Enterprise on Kubernetes? Are there any limitations to doing this?
A: Yes, you can publish map services from ArcGIS Pro to ArcGIS Enterprise on Kubernetes, and there is an option to select a federated server and publish to it. There is no set limit in the Kubernetes environment, other than what is limited by your system resources when publishing.
Q: How are map, scene or tile layers built with differently, between desktop and the cloud?
A: You can build precooked tile/vector/scene packages in desktop, or reference tile/scene content from object stores in the cloud. With the next release of ArcGIS Enterprise, you will have options to create cache for your map services in Kubernetes and generate content in object stores in Enterprise or cloud.
Q: Which server roles will be able to be federated?
A: ArcGIS Enterprise on Kubernetes supports federating ArcGIS Server and Image Server. GeoEvent Server, GeoAnalytics Server and Raster Analytics Server will not yet be supported for federation.
Q: How is Backup and Restore implemented to protect applications, configuration settings, and persistent storage, and does it incorporate WebDR?
A: In ArcGIS Enterprise on Kubernetes, backups are managed and initiated internally as opposed to a separate tool to call into backup APIs, such as the WebGIS DR tool for ArcGIS Enterprise on Windows and Linux. The process will create backups of all items, users, groups, services, configuration settings, and any infrastructure objects like deployment information, that are required in order to re-hydrate a deployment. The backup is stored on a persistent volume that can be restored in the event your deployment incurs data corruption or a more significant outage event. Everything required to bring your deployment back to a working state will be included in the backup and restored.
Q: How much of the functionality in the new Enterprise Manager interface will made available to the other deployment options or infrastructure other than Kubernetes?
A: The Enterprise Manager interface is designed to help administrators manage the software running on Kubernetes, specifically. Certain features may be integrated into other deployment options at a later date but there are no specific plans to do so at this time.
Q: How will the ArcGIS Python API be supported in ArcGIS Enterprise on Kubernetes, including importing python scripts?
A: ArcGIS API for Python will be a first class client API to Enterprise on Kubernetes for consuming GIS services, administration and content/data management.
Q: How will customizations such as SOI/SOE be facilitated by ArcGIS Enterprise on Kubernetes?
A: At the initial launch we will provide the base deployment functionality and you can also federate an ArcGIS Server site on Windows or Linux into the cluster. SOIs and/or SOEs could be running on that federated server, however we do not support SOEs or SOIs running directly within the Kubernetes cluster. This is on our roadmap for future development.
Q: How will the product capabilities of ArcGIS Enterprise running on Kubernetes diverge from the Windows or Linux deployment options, over time?
A: For the launch of ArcGIS Enterprise on Kubernetes, we had to make choices about functionality to include or defer until the future. Over time, the capabilities on Kubernetes will converge toward the functionality available on the deployment options of ArcGIS Enterprise on Linux or Windows.
Deployment and Migration
Q: What steps are required to deploy ArcGIS Enterprise on Kubernetes, and how has Esri streamlined the process?
A: Once a customer has met eligibility requirements and has a Kubernetes environment that meets the system requirements, the process of getting the software running follows three steps: Deploy => Configure => Use. You will run a single script to deploy ArcGIS Enterprise on Kubernetes to AWS Elastic Kubernetes Services (EKS), Azure Kubernetes Service (AKS), or Red Hat OpenShift. For your convenience, the script can also create and configure a cloud provider’s load balancer automatically as a part of the deployment. Once the initial pods are deployed successfully, you can then run another script or use the new Enterprise Manager interface to configure your organization. ArcGIS Enterprise on Kubernetes is now ready for you to use.
Q: What is the expected speed of a typical deployment?
A: It depends on your individual environment, the resources available, the download speed for the ArcGIS Enterprise containers and your configuration. In general, our testing reveals that less than an hour is needed to complete a deployment, but this may vary for you.
Q: To what degree can deployment be fully automated?
A: We have designed the deployment script to write parameters into a properties file after the first time it is is run. Then, the deployment script can be run in a “silent” automated mode with the properties file itself passed in as a parameter.
Q: Are deployment automation tools like Cloud Formation or Terraform supported?
A: These tools help you work with virtual machines. With Kubernetes the underlying structure is a cluster, and it’s relatively easy to configure a cluster using EKS or AKS. So you don’t need automation tools because this is already packaged for you. ArcGIS Enterprise on Kubernetes is deployed using a single script plus a few configuration steps.
Q: How would an administrator optimize their deployment and choose between shared and dedicated instances for services?
A: GIS services can be designated as shared or dedicated as part of optimizing resources and balancing availability requirements. Typically, if an administrator believes a service is at risk for being under-resourced or is a bottleneck, they can convert to a dedicated instance which insulates it from losing resources within the cluster. Kubernetes can be told to generate additional pods, subject to the namespace overall resource quota limitations set by the admin, and Kubernetes will generate containers and assign compute or memory as needed to meet those settings. The administrator should consider the potential impact to uptime and traffic to a service. For high availability and high traffic requirements, the dedicated instance model is recommended.
Q: Are there any restrictions for migrating or adjusting content when moving to Kubernetes?
A: We are still working on a complete migration story for our customers. There will not be a fully automated migration utility provided at launch, but most tools built by administrators to assist with migrating content between environments should still work. This is because the REST API is not changing, so any such tools using the API should still work.
Q: How will Esri assist with migrating from Windows or Linux to Kubernetes?
A: Our ability to assist with individual migrations depends on services available through Esri support or professional services. We will continue to provide documentation and content related to migration. However there will not have a fully automated migration process at launch. Migrating between environments is generally a complicated process with many configurations involved. This is something we will continue to design and develop, in order to ease the transition to Kubernetes. A likely pattern will be customers creating a Kubernetes deployment alongside their existing one and moving data and content over time.
Q: Can we install ArcGIS Enterprise on Kubernetes into local environments such as minikube?
A: You may be able to set up your own Kubernetes environment that conforms to the open standard and run the software there, but this would not be supported by Esri.
Q: What will the upgrade process be like and how fast will it be?
A: While we cannot guarantee upgrades will always be faster, several advantages with the Kubernetes architecture should bring improvements. A streamlined process to execute an upgrade, from the API or from the Enterprise Manager requires fewer actions by an administrator. Being able to target more specific code within individual containers should reduce the volume of software needing an update, and rolling updates will be possible for certain types of upgrades (which may reduce or even eliminate downtime for users). This also means that upgrades can be more granular and eliminate the need to upgrade the entire system (or cause the entire system to be unavailable) when only certain functionality or components need to be upgraded.
Q: Does ArcGIS Enterprise on Kubernetes require or work with an Azure Application Gateway?
A: No. A layer4 Azure load balancer in front of the agent nodes. You can use an Application Gateway if you wish but you will have to configure it manually. Azure allows you to create a layer4 load balancer as a part of automated service deployment. We leverage Azure’s capability to automatically deploy our Ingress Controller service and configure the load balancer for you.
High Availability and Scaling
Q: How will ArcGIS Enterprise on Kubernetes enable high availability?
A: Kubernetes provides high availability as a characteristic of its own orchestration capabilities and will include two architecture profiles you may choose from to meet your high availability requirements. Autoscaling is another means by which administrators can prepare their ArcGIS Enterprise deployment to meet unpredictable demand and protect system availability and performance.
Q: How will Kubernetes provide high availability, instead of having to set up multiple portals or use other design patterns to increase resilience?
A: High availability is achieved by taking advantage of Kubernetes’s orchestration and pod lifecycle monitoring. By implementing liveness and readiness probes, Kubernetes can detect and reschedule pods automatically. When any pod’s replica sets or stateful sets are found to be unhealthy, Kubernetes is going to recreate them automatically and handle routing traffic within the cluster, abstracting all of this from administrators via public and private ingress controllers and coordinating with the load balancers. We did not want to reinvent the wheel when it came to making ArcGIS Enterprise highly available; we took full advantage of Kubernetes’ capabilities for orchestrating the cluster environment.
Q: To what degree is autoscaling implemented, horizontally or vertically, and how will administrators manage it?
A: We do not have autoscaling logic built into the software at launch, but we expect to add this in future releases. However, we will provide you with an intuitive interface in ArcGIS Enterprise Manager to scale out your GIS service pods, as well as update memory and CPU minimums and maximums. This scaling is also possible for some of the foundational or system pods like the portal directory or the administrative API. We also provide you valuable service usage statistics using our Metrics Viewer and Metrics API in order to study historical usage and trends and use the information when considering how to scale resources.
Q: What are the minimum memory and CPU requirements for the cluster and individual pods, and how do these change with different services being provided or levels of demand?
A: The pod specifications will differ depending on the functionality they provide: foundation pods providing functionality for administrative or sharing API; GIS Services pods; and data store components all have different requirements. Other pods such as those running GP services may have additional requirements. Based on internal testing we have optimized the memory and CPU mins and max for each of these categories of pods as well as for individual service types for the system to be able to run without issues, but we know there is no one size fits all default. You know your requirements, and we want to learn from you what is working and help you optimize your deployment. You are encouraged to take a look at the existing values, put your deployment through appropriate testing and tweak the configuration as needed.
Networking and Security
Q: What authentication methods, such as SAML, will be available on the portal?
A: The same authentication methods that we currently support for ArcGIS Enterprise on Windows or Linux will be supported on Kubernetes. This includes SAML, OpenID Connect, Windows Active Directory, and LDAP. For Windows AD and LDAP, communication between the portal and domain controller or LDAP server is required. Web-tier authentication (IWA, Basic, client certificate authentication, etc) will also be supported when the ArcGIS Web Adaptor is released.
Q: Is ArcGIS Enterprise on Kubernetes able to integrate with Windows Active Directory?
A: Yes; we will integrate fully with Windows Active Directory just as we do on Windows or Linux. This means you can use users and groups in the same way that you can in the non-Kubernetes environments.
Q: What is the relationship between the web adaptor and the Kubernetes architecture?
A: ArcGIS Web Adaptor will be a part of ArcGIS Enterprise on Kubernetes. It will be installed on servers outside the Kubernetes cluster and will route traffic to the cluster. ArcGIS Web Adaptor will specifically be used where Integrated Windows Authentication or client certificate authentication is desired. In all other scenarios, a load balancer or reverse proxy in front of the cluster nodes will be enough.
Q: How will external load balancers be configured with Kubernetes, and how does this impact WebContextURL?
A: The WebContextURL parameter capabilities are now implemented as the organization URL that are defined during the deployment stage. The organization URL once defined cannot be changed at the initial release.
Q: What are our options to deploy Portal for ArcGIS, ArcGIS Server, ArcGIS Web Adaptor and data storage (e.g. blob) across virtual network groups in the cloud?
A: This question is more related to a non-Kubernetes architecture; this is possibly with proper firewall and port configuration. But this is technically not something relevant to the Kubernetes architecture.
Article Discussion: