City of Calgary Monitors System Health with Enterprise GIS Optimization Tool

Calgary is the most populous city in the western Canadian province of Alberta. Known for its more than 300 days of sunshine that make it the sunniest place in the country, and for its annual rodeo, the Calgary Stampede, the City of Calgary has 1.3 million residents that call this municipality home. With the motto, Making Life Better Every Day, the City of Calgary offers a range of services—from fire and emergency response to affordable housing and social programs—designed to support and enrich residents' lives. 

The Geospatial Business Solutions (GBS) division, part of the Corporate Analytics & Innovation business unit in the City of Calgary, connects people with location-based data, analysis tools, and technology that enable quicker service and better information. The GBS division provides geographic information system (GIS) and IT infrastructure to support city departments like planning, assessment, roads, and facilities management.

The GBS division has a large number of resources and processes that rely on the overall enterprise GIS working well each day. The current system to monitor the health of the GIS did not satisfy the division's needs, as it performed only one task. As such, staff went in search of a new solution to help them better manage resources and monitor their systems to keep things running efficiently.

The GBS team deployed ArcGIS Monitor, a product designed to collect data and information on the status, usage, and resource utilization of the enterprise GIS, which has yielded an array of benefits such as better troubleshooting and reporting. The improved monitoring of the health of the GIS has led to less lost work time and allowed workers to focus on their job of serving the city.

The various departments in the city rely on the GIS platform for their operations, so enhancing the unit's operational awareness of its ArcGIS system was critical, explains Eddie Fung, GIS data coordinator. The GBS team wanted an automated method to test and check the health of its systems.

"We need to identify potential issues as soon as we can. We need to be informed so we can address the issue before it impacts our clients' operations," says Fung. "Essentially, we don't want to be reactive anymore; we want to be more proactive."

He explains that the GIS has a relatively large footprint, which made it challenging to manage all the resources associated with the enterprise system and identify any daily operational issues. Examples include the division's nine stand-alone GIS servers and five separate enterprise portals as well as development, testing, and production environments.

Previously, the GBS team employed an application capable of only one function: REST call checks. REST calls were sent to the city's server, and the GBS team checked for responses. Fung says that, because of the way the software was configured, staff would manually set up hundreds of these checks for each web service URL.

Each time a new web service was published, the team would add the line in, which ultimately overloaded the system because of the large number of web services. Fung explains that it would take 10 minutes to load the page, then additional time for the form to show up and for staff to enter the URL. With more than 370 services, this process was time-consuming and created challenges for the team.

"[This process] was useful, but only to a certain extent," says Fung. "You know a service is down, but it doesn't help you pinpoint, Is it a database issue? Is it our web adapter? Is it the server? It's basically trying to find a needle in a haystack at that point. So it is hard to identify on certain services if it's actually our system, or if it is something wrong with a part of IT."

With the GBS team supporting so many departments, Fung says it became increasingly important that the workflows performed efficiently because so many processes are built on it and are intertwined. Fung went in search of a new solution from Esri to help GBS staff streamline their work and better manage their system.

The City of Calgary became an early adopter of ArcGIS Monitor, a tool designed to monitor the health of an ArcGIS Enterprise deployment by providing awareness of system usage and performance. Simon Biickert, a Senior Solutions Architect at Esri Canada, told the GBS division about the tool and referred the team to Michelle Williamson, senior consultant for the Prairie Region Professional Services department in Esri Canada, for assistance. Williamson went to the GBS office to conduct a demo and provide training for the team. Fung says learning the tool was very easy.

"We kind of explored hands-on with Michelle [Williamson]. She walked us through the whole installation process, all terminology, and each of the functions," he says. "We were sold instantly because this is exactly what we needed. We are a big Esri shop, and Monitor does all the scanning that we wanted plus more."

The GBS team set up Monitor to help ensure that the system in place is performant and working optimally. Using the information collected by ArcGIS Monitor, GBS implemented a two-part solution: a near real-time display and messaging component for monitoring and report production on consumption and growth.

The GBS team is monitoring its six ArcGIS Server sites and Enterprise portals for use and consumption with ArcGIS Monitor. It is also running health checks on the full stack of technology that is supporting the Esri system, including the hosting servers for the GIS servers and Enterprise portals, the published services, and databases.

GBS now has several tools available to monitor system health. One example is the HTTP counter the team set to collect data every minute, increasing their availability to approximately 99 percent each quarter. HTTP counters allow tracking of the HTTP response code, response time, content length, server time, and network time for specific URLs.

From fewer outages to tracking down poorly performing web services, the GBS team has seen many positive benefits with ArcGIS Monitor. According to Fung, ArcGIS Monitor helps him and another team member care for everything, and they can determine a problem before the client realizes there is one. He adds that there are also fewer outages for web services, which translates to less lost work time.

"The best is no question, no problem, right? If no one is [coming to us], I think we did a pretty good job. And we couldn't achieve that without ArcGIS Monitor," says Fung.

ArcGIS Monitor has also helped the GBS team better troubleshoot problems. Fung says that troubleshooting before was challenging because it was hard and often time-consuming to determine what the root cause was, like whether it was a database issue or machine related. Now, he says, as soon as an issue is flagged in Monitor, the team can react immediately to address it.

Previously, issues occurred in the GIS server if the RAM was beyond 80 percent, but the team didn't have a way to know which machine had the issue, Fung explains. Staff would have to hit every machine manually or check logs. Now, if there are two machines on a site, Monitor instantly identifies which machine is having an issue, which, he says, has been a huge time-saver.

"The time saved in figuring out what's wrong is, I think, the best part of [Monitor]. Managing the health of our ArcGIS Enterprise system was more reactive in the past, and we only addressed issues as clients notified us," says Fung. "Troubleshooting is easier [because] we can see [the] history of the counters and identify the pattern on the issues. We can anticipate what issue we might run into, and mitigate accordingly."

Another valuable tool is the ability to configure and send notifications with Monitor. The GBS team members developed a Python script to parse alerts. Monitor sends a health check summary email every four hours by default, but because the system is so large, they wanted to only see "alerts that matter" to better pinpoint issues, says Fung. As such, the customized Python script parses the cross-platform database program they use to grab the alert and see if it is persistent before sending a notification.

Fung explains that if the issue resolves itself, then it moves into an email, which significantly reduces the number of notifications and delivers alerts for only the most important issues. Also, a Python script was created to look at all scheduled tasks on the batch server, and results are written directly into ArcGIS Monitor. Monitor also allows the team to get alerts if a batch job has failed.

"We really liked the Python extension function in ArcGIS Monitor because we were able to customize pretty much anything we want, says Fung. "So it was very useful and allowed us to customize to our heart's content. It's definitely resulted in less scripting time to collect the same data. Otherwise, we would have put in a lot of effort into writing the code to get everything."

Statistics about the enterprise GIS are viewable in Monitor, which, Fung says, allows the team to see the big picture. Monitor has a functional web page to view collected statistics, but the team took it a step further and decided to create a dashboard to display vital data. This helps avoid extra steps like refreshing the page to check individual servers.

ArcGIS Monitor collects the events and stores them in the database repository. Then, staff filter the events and create a message stream directly to their custom JavaScript-built dashboard. The custom solution shows real-time metrics, enabling the team to be more proactive.

"[The dashboard] gives management a visual of what is happening and the amount of work we do to keep the system online," says Fung.

The GBS team members create monthly, quarterly, and annual reports for management with Monitor. The report includes uptime and usage and how many services they have each quarter and annually. Fung says the report is a great way to show their key performance indicators (KPIs), which helps justify further investment in GIS technology and increases transparency for city officials.

"It's helped our director, Bruce [Cullen, director, Corporate Analytics & Innovation for the City of Calgary] to communicate to [city] council how important ArcGIS technology is at the City of Calgary," says Fung. "That's a huge thing to show management that the public money that we spend on the technology has very good value for its return."