Full-Service Real Estate Advisory Firm KBC Advisors Connects Systems and Automates Data Integration from External Systems with ArcGIS Data Pipelines

For businesses of any size, the decision to expand can help increase profits and reach new customers. This may involve increasing physical locations or obtaining new spaces to produce goods. KBC Advisors assists clients with managing and executing the best real estate outcomes for their businesses using data-backed guidance. With a focus on premier industrial space, KBC’s full-service platform caters to occupier, landlord, and investor needs.

For internal staff and external clients, the best way to show the availability of a commercial site and its location is through a visual platform. Prior to implementing a new process, the location intelligence team at KBC Advisors had a manual process in place to track existing assets and land sites. This process involved gathering data from an external application and exporting files, then uploading the information into a separate system for visualization. 

This process was time-consuming and required focus to avoid errors while moving information from system to system. In search of a better solution, KBC’s location intelligence team implemented ArcGIS Data Pipelines for more seamless data integration with ArcGIS. This has improved the accuracy of data and automated processes—saving time, enhancing efficiency, and enabling the location intelligence team to more effectively deliver clients and real estate brokerage professionals the data they need. 

James Drumm, director of services and technology at KBC Advisors, explains that the location intelligence team created data and tracked projects and business deals, and then used ArcGIS Online—Esri’s web-based mapping software—to visualize data and show clients available site locations.

This data visualization process was a challenge because there was a disconnect from external platforms to the Esri environment, shares Drumm. The previous workflow included creating and exporting a report of the locations and managed sites. A geographic information system (GIS) analyst would then use ArcGIS Pro, desktop GIS software from Esri, to plot the coordinates of the location and then publish it to ArcGIS Online. 

The published web layers showing the data were then showcased in client-facing applications, but would quickly become out-of-date. The disconnected systems translated to too many manual steps, creating inefficiencies and errors in the process. Information was extracted dozens of times a week, so the GIS team needed a more efficient process for data visualization. 

“[The process] was too slow . . . edits in either application weren't always being tracked in the other application,” says Drumm. “By definition, disconnected systems are not as accurate . . . so that was a major problem that we were running into.”

One important item impacted by this process was the status of a site, whether available or unavailable. The most current status was not always reflected in the system because the data was based on the static spreadsheet downloaded at the time and not based on real-time data.

Drumm began the search for a new solution that would help integrate outside applications with ArcGIS. After receiving email notifications about a new Esri product, Drumm and the team became early adopters of ArcGIS Data Pipelines. ArcGIS Data Pipelines streamlines data integration in ArcGIS Online by offering an efficient way to ingest, prepare, and maintain data. Drumm says Data Pipelines offered an effective way to connect their separate systems, including one that was a non-Esri-based environment. 

However, the group still needed to use a mediary data storage solution. When Drumm saw that Azure Blob was a supported input data source for Data Pipelines, he decided they could use that as the intermediate connector between ArcGIS Online and an outside platform. Azure Blob is a storage solution for the cloud from Microsoft, and Drumm says they have an expert on staff who helped with implementation. 

Drumm and the team of GIS analysts began using Data Pipelines in late 2023, and he notes that the use “exploded” after they established a connection to their Azure Blob Storage container. Drumm describes Data Pipelines as easy to use and learn, and the learning curve for Data Pipelines was minimal because of the group’s familiarity with Esri technology.

Also, he notes that connecting the cloud storage container was very straightforward. Drumm began the implementation process by working with their internal Microsoft Azure specialist, who set up a job that would push the data from an external platform to Azure Blob. From there, Data Pipelines can access the latest data.

The integration process is now automated and set to a schedule. Azure Blob is refreshing multiple times per day and the respective data pipelines are scheduled to run at the same cadence, kicking off after Power Automate jobs. Using this process, the data in ArcGIS Online is updated every few hours, ensuring the latest changes are reflected throughout the day. 

The revamped workflow with Data Pipelines has streamlined operations and offered many benefits to the location intelligence team’s daily workflows, according to Selena Lawson, GIS manager at KBC Advisors. The use of Data Pipelines has saved time, and many of the data pipelines run multiple times per day, Monday through Friday. Lawson explains that instead of spending time on the previous manual ETL (extract, transform, and load) processes the team used, the data updates are reflected in ArcGIS Online without manual effort.  

“This real-time reflection of the data ensures our users have the most up-to-date information we can provide,” says Lawson. 

Drumm says Data Pipelines has vastly improved their speed and reliability. He says the scheduling feature that lets the location intelligence team schedule based on the time frame they need is critical and helps ensure the availability of current data. The recurring Azure Blob refresh takes only seven minutes, and then the data pipelines that are connected automatically kick off shortly after to update the feature services inside ArcGIS Online. In total, the updates take less than 20 minutes. 

“Due to the scheduling function, we can redirect much of our focus from repeat tasks to more intricate analyses and deliverables,” says Lawson. 

The accuracy of data has improved with Data Pipelines. Lawson explains that with this solution, the team can rely on living resources where the data is housed. She says this ensures “the schemas match, data is formatted correctly, and no time is spent correcting transformation errors.” However, if there is an error made when data is entered, the data pipeline will catch and correct it, and the cleaned data will be reflected in ArcGIS Online. 

“The data update capability enables us to no longer rely on multiple sources and users for data input; because of this there are virtually no human errors or manual time spent on the Esri side of things,” says Lawson. “This saves an average of 75% of our analysts’ total time spent on data update requests.”

Drumm adds, “The data movement is no longer a concern or a need because it's automated. This saved time allows our analysts to focus on location analysis, serving our clients, and building custom applications instead of data cleanup and updates.”

According to Drumm, the basic component of plotting coordinates of commercial sites is critical because it comprises 99 percent of the location intelligence team’s workflows. The new workflow means that the team is no longer converting tabular locations into a feature service to show it on a map. The Create geometry tool in Data Pipelines is used instead to create a point geometry field from a tabular dataset that contains longitude and latitude fields and then update an existing feature service automatically.

Drumm says the GIS team has now created hundreds of feature layers with Data Pipelines. Now that all data is connected, accurate, and available in ArcGIS Online, deliverables such as a labor study or a site survey comparison can be done in a day, not weeks, helping them better serve their clients. 

“One of the most important things in our industry . . . is speed combined with accuracy. So, if I can cut out manual steps, if I can cut out extra people doing things, if I can cut out you having to enter data in four different systems, that just vastly improves our speed to a solution . . . ,” says Drumm. “My personal goal for the team is speed and accuracy, and we can meet that.”