In our previous blog post, Metadata – Data Interoperability’s Hidden Talent (Part One), we looked at how Metadata can play a vital role in ensuring the context, accuracy, and integrity of your data. We discussed the first of three use cases for managing your metadata and how to populate feature class and table metadata in ArcGIS Pro. In this article, we will focus on the other two uses cases and how to create and maintain metadata as it moves through ArcGIS using ArcGIS Data Interoperability.
Overview
Previously, we discussed how to populate feature class and table metadata in ArcGIS Pro. In this article, we will focus on the other two use cases for metadata management:
- Populating metadata for a hosted feature layer in ArcGIS Online
- Updating metadata for multiple portal items in ArcGIS Online in bulk
We will also discuss how to deep copy data and metadata between environments using metadata.xml.
Populating Metadata for a Hosted Feature Layer
There is another level of metadata that we need – the metadata for the hosted feature layer. For this, we move onto Case #2 – populating metadata for a hosted feature layer using ETL tools.
Below is the details page of the hosted feature layer in ArcGIS Online:
The first thing that stands out is that the metadata for the hosted feature layer looks amazing! However, it is important to point out that we did not create it. Rather than manually adding the line of summary, description, tags, or terms of use on the item details page, we harvested it straight from the source – Vancouver’s portal.
“What is this magic?” you may ask. Well, the truth is, it is not magic at all. Here, let us show you how we did it.
Below is the landing page for Vancouver’s portal. The datasets are sorted based on popularity, though we are skewing the results for the sake of this demonstration. We then highlight the dataset we want to harvest as well as a link to an Excel spreadsheet near the bottom of the screen that contains all dataset metadata.
We then open the spreadsheet so we can see the row for the subject matter dataset – Issued building permits.
After that, it is just a matter of getting the cell values for metadata elements from the Excel spreadsheet to the hosted feature layer. The good news is that the RefreshBuildingPermits ETL tool was built for this sort of thing.
Without making this a tutorial on ETL tool design, the steps we’re using for our information product are to:
- Make an initial file geodatabase incarnation of the data using the ETL tool ImportBuildingPermits that has the shutdown script shown above.
- Create the layer with symbology and the desired pop-up behavior we want in ArcGIS Pro.
- Publish the feature class and related tables as a hosted feature layer to ArcGIS Online.
- Maintain the hosted feature layer using the RefreshBuildingPermits ETL tool.
By doing this, we are putting the hosted feature layer’s metadata update functions in a second ETL tool. If you look at the bottom right of the screen you will see a bookmark labeled Update Metadata. The FeatureWriter transformer handling actual permit data has a summary port which triggers a FeatureReader when any data is written to a hidden connector. The FeatureReader accesses the Excel file on the open data site and gets the right cell contents to an ArcGISOnlineConnector (also works with ArcGIS Enterprise) with the following settings:
Essentially, the process is as easy as filling in the form. As long as there are no data changes, there are no metadata changes either. How easy is that? You now know how to harvest and write metadata using ETL tools!
Updating Metadata for Multiple Portal Items in Bulk
Lastly, we must consider Case #3 – making bulk metadata changes to any number of portal items.
For the purposes of this demonstration, let’s say that we manage an ArcGIS Online or ArcGIS Enterprise organization with hundreds of items that need to have the terms of use updated. This change will need to be made for all hosted feature layers in a content folder.
Example:
This is what we have now:
This is what we want:
In order to make this bulk change, we can use another easy ETL tool:
An ArcGISOnlineConnector can be used to list the candidate portal items along with a Tester to filter for hosted feature layers, and another ArcGISOnlineConnector to update the target element.
If no value is supplied for an item, it is left unchanged. To zero out an item, you would supply a value of NULL.
Now, we mentioned above another ETL pattern of interest is deep copying of data and metadata between ArcGIS environments, for example between ArcGIS Enterprise and ArcGIS Online. There are core approaches for this use case in which metadata travels with the data, but there are also use cases where you wish to perform edits on the data and/or metadata during the ETL process. In this situation you might work with the metadata.xml file for each dataset. This can be done with an option in the Get Details and Set Details actions in the ArcGISOnlineConnector. This will get and set the value of metadata.xml.
Conclusion
With ArcGIS Data Interoperability it really is that simple to manage metadata within ArcGIS. Whether you are populating feature class and table metadata, populating metadata for hosted feature layers, or updating metadata for multiple portal items, Data Interoperability has the tools to enable you to manage and maintain your metadata. If you have any questions about the use cases above, or if you need additional assistance managing your own metadata, please reach out to your normal account representative or visit us on our Esri Community website.
To obtain a free trial of ArcGIS Data Interoperability, please visit this page.
Article Discussion: