Automating the migration to a new data model
Using ArcToolbox with Python tools, the Air Force (AF) created a repeatable process in ArcGIS for Desktop that enabled the translation of a very specific data model to a more generalized one. The AF solution transitioned the Spatial Data Standard for Facilities, Infrastructure, and Environment (SDSFIE) 2.6 geospatial data to the Air Force Adaptation of SDSFIE 3.0.
The AF GeoBase program supports the AF Civil Engineer (CE) and broader AF missions by providing accurate geospatial data of real-world features for all AF installations, ranges, and property. The GeoBase mission involves tightly integrated cross-functional collaboration across the AF to provide geospatial visualization and analysis of CE business data. Its mission includes managing geospatial data and serving as the authoritative repository of all installation geospatial data.
Per the direction of Executive Order 12906, Coordinating Geographic Data Acquisition and Access: The National Spatial Data Infrastructure, published April 13, 1994, many government organizations implemented the SDSFIE logical data model. This model is based on standards from the Federal Geographic Data Committee (FGDC). Within the United States Department of Defense (DoD), SDSFIE is now the geospatial data standard for all the armed services and the Washington Headquarters Services. It is governed by the Defense Installations Spatial Data Infrastructure (DISDI) Group. The Department of Defense Real Property and Installations Lifecycle Management (RPILM) Investment Review Board (IRB) approved SDSFIE 3.0 on November 15, 2010.
Following this approval, the Office of the Under Secretary of Defense (Acquisition, Technology & Logistics) (USD[AT&L]) submitted the SDSFIE 3.0 to the DoD Information Technology Standards Registry (DISR) for listing as a mandated standard. To comply with the standard, the AF developed the AF Adaptation of SDSFIE 3.0, approved in January 2012.
The New Standard
The new standard is significantly different from the prior version, SDSFIE 2.6. The data model for SDSFIE v2.6 represents facilities, infrastructure, and environmental features in a broad, thematically based, hierarchical manner. Items are organized first by a broad theme, then to a more detailed description or type for a given theme, then to a specific class of a thematic type. Simply put, items are organized into entity set, entity type, and entity class.
The data model for SDSFIE v3.0 represents features similarly to v2.6 except that it is more general. For example, in v2.6 transportation features are broken out into several modes, whereas in v3.0, all forms of transportation are in one entity type or dataset.
In some cases, there is a simple one-to-one relationship between features in v2.6 and v3.0. However, this generalization can require multiple v2.6 feature classes to be merged into one v3.0 feature class. Also, one v2.6 feature class can relate to many v3.0 feature classes. Additionally, some v2.6 feature classes are not in v3.0, and some v3.0 feature classes do not have a corresponding 2.6 version. The variety of relationships between features in v2.6 and v3.0 complicated the task of automating the migration process.
The Migration Process
Successful data migration required meeting two key goals: rapid conversion and no data loss. During the transition process, each AF user had to stop editing the current SDSFIE 2.6 geodatabase while it was migrated to the AF Adaptation. Consequently, rapid data migration was essential. Significant differences in the two versions necessitated considerable data manipulation and required a carefully developed methodology that ensured no data would be lost.
Because migrating from SDSFIE v2.6 to v3.0 was a significant effort, automating as much of the process as possible was essential. Once the migration methodology was envisioned, Python tools for use in ArcToolbox were developed to assist GeoBase personnel with the task. Using these tools, the transition process was accomplished more efficiently and resulted in minimal downtime for GeoBase. While creating the tools was not without difficulties, in the end the tools were used to migrate well over 1,000 sites in only a few months.
The Process Step by Step
ArcToolbox Python tools developed by the AF saved critical man-hours while ensuring rapid and lossless data migration. The method the AF developed requires an interim (transition) geodatabase. The interim geodatabase contains v3.0 feature classes, but each feature class includes both v2.6 and v3.0 attributes. Consequently, it was named the v2.6-3.0 merged geodatabase.
To automate the process, the AF developed a series of tools that were separately applied. This multistep approach was adopted because initial tests using a single script required long run times (as much as 18 hours in one case). A lengthy run time might cause users to conclude the tool had stopped in midprocess. In this multistep process, each tool prepares the geodatabase for the next step. This approach has the additional benefit of giving users an opportunity to check the results of one operation before moving to the next—a safeguard against data corruption or loss. Each tool and the processes associated are listed in “An Overview of the SDSFIE v2.6 to v3.0 Migration Process.”
Automating the Process
For more information, contact Kyle D. Turner.
