The US Forest Service spearheaded an effort to increase access to its trove of more than a petabyte of aerial imagery and raster datasets. This culminated in a cross-agency geospatial data sharing service that provides cost savings, increases capacity, and delivers more efficient data management of high-resolution aerial imagery and raster data.
For 20 years, the US Forest Service has served high-resolution aerial imagery to more than 12,000 people who use GIS technology to analyze images to answer questions about forest health.
Imagery has helped the Forest Service revolutionize the way forests are monitored and managed, including detecting pests and diseases earlier, assessing habitats for endangered and threatened species, conducting research of forest dynamics and ecosystem processes, and monitoring and managing wildfires.
A recent move to a new data center resulted in an unsustainable 1,300 percent cost increase, which left Forest Service administrators searching for a way to continue to use imagery to enhance operations. Instead of searching for ways to lower the cost, imagery experts had an idea. What if they could team with other federal agencies to make the leap to the cloud?
They reasoned that a cloud approach could eliminate data duplication across agencies while unlocking new capabilities. It could also help agencies make the best use of the pipeline of high-resolution imagery the federal government procures from a network of providers.
“At the start of this, we had a feeling of skeptical optimism,” said John Gillham, a contractor working as a project manager for the Forest Service. Gillham and Dave Vanderzanden, enterprise data and services program leader at the Forest Service, originated the project at the US Forest Service’s Geospatial Technology and Applications Center. The duo reached out to geospatial administrators at various federal agencies and received guarded enthusiasm for the concept. Their queries revealed a widespread dissatisfaction with current imagery hosting options and a desire to use more imagery to understand landscape changes across the country.
With support from the Forest Service, the US Fish and Wildlife Service, and the US Geological Survey (USGS), the first cross-agency geospatial data sharing service—the Interdepartmental Imagery Publication Platform (IIPP)—launched in May 2024.
Rob Dollison, The National Map (TNM) delivery services lead at USGS, related how IIPP has provided a learning opportunity for all. “We’ve been able to share the process of building an image service in the cloud, and the Forest Service has done a lot to organize and host historical imagery,” he said. “We each brought expertise to the table.”
Advantages of the Cloud
The government’s cloud-first pledge, and innovations in cloud-native technologies, finally made it feasible to create a shared data source for imagery. With the cloud, each agency can maintain control of its own data while gaining access to a larger trove of imagery for its users in this collaborative system.
“Interestingly, the cloud-first strategy wasn’t our top consideration when we started four years ago,” Gillham said. “We did an exhaustive analysis of 10 different hosting environments and scored those against eight criteria. The cloud popped out as the topmost viable option for cost-effectiveness, capability, and expandability.”
The IIPP site is hosted on Amazon Web Services using Esri’s ArcGIS Enterprise on Kubernetes. Its cloud-native architecture scales quickly based on demand.
“Kubernetes helped us design a system that doesn’t cost more than necessary when it’s not in use, and it can handle an onslaught of anywhere from 12,000 to 50,000 users,” Gillham said. In a recent test of its dynamic scaling capabilities, a new server was automatically added within seven minutes to meet a spike in demand.
The containerization approach of Kubernetes reduces the technical complexity of the system by bundling the code and dependencies needed to run it, which makes provisioning a new server automatic and quick.
“There’s just nothing else out there that can do that,” Gillham said. “We’re going from a system that would take two months to get another server added compared to under 10 minutes now for as many servers as we need to meet the demand, all automatically.”
Kubernetes also allows agencies to maintain full autonomy in a shared system. “We come from a publishing group ourselves that hosts data, and the last thing we want is to lose access and control,” Gillham said. “When we reassured everybody that IIPP partners maintain full autonomy over their data, it helped relax some concerns and more agencies became interested.”
A Platform for Sharing and Consuming Imagery
The typical pattern has been for each agency to host its own copy of the same data. During the scoping process for IIPP, the team found that six agencies hosted duplicate sets of the high-resolution images that the US Department of Agriculture collects for the National Agriculture Imagery Program (NAIP). Since 2004, the Forest Service has served all NAIP imagery, which amounts to half a petabyte of imagery.
Discussions have been ongoing with the Farm Production and Conservation (FPAC) Business Center, which processes and shares images through its Geospatial Enterprise Operations (GEO) division. FPAC GEO had coincidentally been making a migration to Amazon Web Services too, which makes further efficiencies possible.
“When their NAIP data becomes cloud optimized, we’re hoping FPAC GEO will be able to make it available to us, at which point we can eliminate our copy,” Gillham said. “When we’re connected directly to FPAC source data, then the minute they update that data it’s instantly updated for all of our users. That’s an efficiency government should always strive for.”
This cloud-to-cloud data sharing switch illustrates the efficiency gains that increase imagery sharing. It not only removes the cost of storing imagery more than once but also reduces labor costs to manage the data at each agency.
The IIPP team has been talking with nine federal agencies to answer questions and address past data- and cost-sharing challenges. An IIPP governance charter has been established to define roles and responsibilities, and a working group sets standards for how data is published and used. IIPP has a publishing tier for agencies that want to host data, and a consumer tier for agencies that want access to published datasets.
The three founding agencies have begun publishing data to IIPP and are planning rollouts to their users. By the end of 2024, the aim is to have all users on board and to begin to welcome more agency partners.
In addition to unlocking access to streaming imagery for all agencies, the hope is that more agencies will be compelled to publish and share their own datasets with the wider community. “With this joint shared system, we’ve lowered the cost and learning curve to get into publishing, especially in the cloud.” Gillham said.
Gaining Interest Among Agencies
The first goal of reducing the cost of hosting imagery has been achieved. Next, IIPP aspires to expand its imagery serving service to all interested federal agencies. State government partners may be next, and public access has been discussed.
“The minute we cut over to the new system and turn the old system off, the Forest Service stands to save about $5 million per year,” Gillham said. “We hope that each partner should also see a cost reduction to some degree, and the more partners that join this imagery hosting co-operative, the cheaper it becomes for everybody. This is just the beginning.”
Given its status as the first interagency data hosting solution, IIPP has gained a lot of attention. “There were some skeptics in the beginning,” Gillham said. “And as we keep progressing and getting over hurdle after hurdle, I think we’ve garnered more support and excitement.”
Continuous effort will be made to refine and enhance the system based on user feedback and technological advancements. The promise of this cloud approach goes well beyond an easier and cheaper imagery serving system. Ultimately, it’s about better answers.
“For years, these systems have been siloed so one agency couldn’t access the other agency’s data or images or each other’s expertise,” Gillham said. “I foresee a time when somebody applies a deep learning model or processing algorithm that another agency developed that suddenly solves a pressing problem they struggled with. That’s the power of a collaboratively shared hosting system.”
