Southern California Association of Governments Supports Sustainability, Transportation Initiative with Workflow Management System

With over 39 million people, California is the nation's most populous state, and its population is projected to reach 45 million people by 2050. The growing population means increased demand for housing, food, and transportation, and this consumption can contribute to environmental degradation and an increased risk of disasters like floods or wildfire. One dedicated organization convenes to address these regional issues that may impact residents. 

Founded in 1965, the Southern California Association of Governments (SCAG) is the largest Metropolitan Planning Organization (MPO) in the nation in terms of land and population size. Southern California consists of 191 cities and six counties—Ventura, Los Angeles, Orange, San Bernardino, Riverside, and Imperial—encompassing more than 38,000 square miles and almost half (48 percent) of California's population. Much of SCAG's work supports local jurisdictions with their planning through several critical initiatives. 

SCAG's main purpose is to produce Connect SoCal, the long-range Regional Transportation Plan (RTP) and Sustainable Communities Strategy (SCS). Connect SoCal outlines approaches for Southern California to grow efficiently and sustainably. To prepare for Connect SoCal, SCAG initiated the Local Data Exchange (LDX) process to gather the most up-to-date information (e.g., land-use and socioeconomic forecasts) available from local jurisdictions. This data helps SCAG better understand how the region is developing and meeting its targets in terms of mobility, housing, and the environment. 

To modernize the association's current data contribution process and increase efficiency in the data collection process, SCAG wanted a new solution to monitor input. SCAG deployed ArcGIS Workflow Manager to revitalize the data exchange process. The use of Workflow Manager yielded positive benefits for the association, including increased flexibility in data sharing and automated communications, and helped SCAG better manage and track contributions from local jurisdictions.

The SCAG Regional Data Platform (RDP) is a group-designed system for collaborative data sharing to enable better planning at the city, county, and regional levels. According to Tom Vo, principal regional planner at SCAG, one of the goals of the RDP is to provide the association's regionally significant datasets and resources to local jurisdictions to facilitate transparency and collaboration in the region.

An important part of this initiative is the exchange of local data that takes place when SCAG connects with local jurisdictions to collect and review data, and then provide it back to SCAG. Local jurisdictions are asked to provide feedback on an estimated 30 data layers in six different categories, primarily land-use and socioeconomic forecast data. Due to lack of standardized processes, historically data from each jurisdiction was sent to SCAG in different formats, such as marked-up maps or spreadsheets. Feedback was also received via mail, email, or paper maps.

This data exchange occurs every four years, and input is incorporated into SCAG's Connect SoCal and its database. Mengdi Li, a senior regional planner at SCAG, explains that the land-use data is important for not only SCAG but also jurisdictions, so an efficient system is necessary to update datasets.

Vo says the original challenge was tracking and organizing the input, which, due to limited resources, became difficult to manage. For example, each jurisdiction has a different set of land-use data with its own unique codes. Upon receipt, this collected data had to be translated into standardized SCAG codes, a time-consuming process that involved manual review by staff.

“Before, in my opinion, the process of collecting data took a little bit longer. In the past, when we received input, we would have staff to keep track and put that into the system,” says Vo.

Over the past several years, SCAG began to explore and leverage its new geospatial infrastructure to evolve Connect SoCal. With large volumes of data going back and forth between SCAG and local jurisdictions, the SCAG team needed a new enterprise solution to standardize this input process.

The team members at SCAG selected ArcGIS Workflow Manager to help them transform their data exchange. Workflow Manager is scalable enterprise geographic information system (GIS) software that automates and simplifies the performance and management of location information-enabled work. Because the RDP was built on Esri infrastructure and technology, Esri Professional Services suggested using Workflow Manager for the data exchange process.

According to Sean Tucker, lead GIS administrator at SCAG, the SCAG planning team works heavily in ArcGIS Pro, Esri's desktop GIS software. Consequently, ease of integration was an advantage to using Workflow Manager. Additionally, Tucker says the ability to set up email notifications was a bonus.

"After looking at what planning [staff] wanted to do with the RDP, it became obvious that we would have a lot of processes that would require consolidation . . . and people being able to look back and know what they did," explains Tucker. "Workflow Manager was a good choice to have because it works with the other Esri elements, and we would have that collaboration."

Now, using Workflow Manager, input on the data layers is received through different input tools such as an LDX web application and geospatial file uploads. Each input method is powered by Workflow Manager, meaning there is a standard input and review process regardless of the way a local jurisdiction provides the input.

Regional planners, including Vo, gave details on what they wanted out of working with Workflow Manager, and then SCAG collaborated with Esri for the initial implementation. Tucker says that because of the work SCAG does for the region, the planners wanted the inaugural setup to be the industry standard.

"I think that was part of the reason why we really leaned on Esri. [Esri has] the understanding of the know-how and these nuances of how to collect data, how to collaborate with data, how to make sure that there's continuity," says Tucker.

SCAG created its Local Information Services Team to provide technical assistance to jurisdictions in the region that want to learn how to use Workflow Manager and tools and resources from SCAG's Regional Data Platform. The use of Workflow Manager has streamlined the input collection process, according to Vo.

"The overall framework is that we get inputs from many different formats. Workflow Manager is one of them, and we are trying our best to encourage our jurisdictions to use Workflow Manager integrated in Regional Data Platform because it helps us. [Workflow Manager] has a better management system," says Vo. 

The use of Workflow Manager has streamlined the input process for SCAG, leading to improved transparency and efficiency. Vo says Workflow Manager has proved beneficial for SCAG and external stakeholders. SCAG can more easily keep track of what's in the system and how it has been incorporated into the database, and cities and jurisdictions can now keep track of their input and see their status.

"This will enhance our communication and trust with our local partners because now, when they provide data to us, it's not a black box anymore. They know exactly where their data is going, and they can keep track of that," says Vo. "And that will also increase engagement and help us build consensus with different partners in the region."

ArcGIS Workflow Manager offers APIs for extensibility. These APIs, coupled with the functionality of ArcGIS API for Python, provide flexibility to integrate custom geoprocessing scripting tools into workflows. The software also gives users the ability to have certain steps be completed outside the Workflow Manager application while continuing to track job progress in one central system.

Li says the SCAG team is currently using Workflow Manager in ArcGIS Pro to view the details of specific jobs and see the edits users have made to existing datasets. For example, if the user has added a specific code, Workflow Manager highlights it so that the team can easily see it.

Li says the team members are currently looking for an alternate way to view job details and review submissions with Python scripting. She says the team was recently able to generate job details using ArcGIS API for Python and ArcGIS REST API for more comprehensive tracking. Team members are also using APIs to extract the difference between the default version and a version created by a user.

"This way, it doesn't bring us the entire database. [Team members] only extract the parcels that a user makes modifications to. The dataset is very compact and easy to handle," explains Li.

Vo says this solution helps SCAG with system management because the team can see where the data comes from by using Python scripting. By generating a summary table of all data-contribution jobs in the system that can be converted into statistics, like the percentage of jurisdictions that have provided input, the solution streamlines various internal tracking and reporting purposes.

"Because we have limited resources right now, I saw opportunities using Python to streamline the process. In our GIS team, we only have several people—but we have hundreds of jobs to deal with," says Li. "So how to batch process jobs is very important for us. We kind of rely on those ArcGIS APIs to help us get access to all the information and do the spatial analysis."

The SCAG team receives email notifications sent by Workflow Manager when an external user submits input. This helps the team keep track of submissions and ensure accuracy. Also, external users have steps to review their contributions. SCAG can view all contributions in ArcGIS Pro and incorporate the appropriate changes.

With the successful implementation of Workflow Manager, SCAG is looking to use it in other applications. SCAG has the ability to configure or extend Workflow Manager in a way that aligns with the association's workflows and user needs.

"I think the idea is to keep this system alive so that cities and counties can always use the system and contribute data to us on the fly. My hope is our stakeholders in the region take this opportunity to leverage this system," says Vo. "If they are using our system to do their local planning activities, . . . that will also benefit SCAG in return because we use their data and we don't have the local expertise like they do."

“Communication and trust lead to stickiness. Folks, once they realize something is standardized, it's set, and they get used to that way of working. It is the industry standard, it's repeatable, it's familiar,” adds Tucker. “Even when they're in the future, this will be the way to do it. And the region needs that trust.”