Don't Let Your Content Go Stale!

Constantly keeping your layers updated can be a laborious task.  To keep some of our layers updated in the ArcGIS Living Atlas of the World we use python scripts.  These scripts are easy to use and available to you.  The scripts check to make sure your layers are using the most recent underlying source data each time they are executed.

Using the script, let’s walk through an example using Water Temperature from NOAA and the web upload method.

Step 1: Download the Script

Extract and place the script in: “c:\scripts\OverwriteFS”

Step 2: Create a copy of the script and name it “OverwriteFS”

When importing a script, Python does not like multiple periods in the filename.

Step 3: Download water temperature data from source URL

Step 4: Add Item to ArcGIS Online

Navigate to your content page and select “Add Item > From My Computer” (You must have publisher credentials to publish data).

Configure the field names that contain the coordinate information and add the appropriate tags:

The newly created content item for Water Temperature should look like this:

take note of the item ID in URL (highlighted in image above).

Step 5: Launch Python 3

If you have ArcPro installed and are relying on the associated python library the location is most likely here: “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe” although this location may be different on your computer.  More Info.

When the console launches type the following:

import arcgis
arcgis.GIS(“ArcGIS Online ORG URL”,”username”,”PASSWORD”,profile=”profile name”)

The highlighted portion is return indicating that the profile was created. (above)
1. ArcGIS Online Org
2. Username
3. Password
4. Profile Name

More Info: https://developers.arcgis.com/python/guide/working-with-different-authentication-schemes/

Step 6: Run script to keep data updated
Each one of these methods can be used to keep your data updated.

Method 1: Using ArcGIS Pro Python Command Line
Launch the Python Command Line from ArcGIS Pro “View” Menu.

First, register the directory where we saved the OverwriteFS.py script.
To do this, type the following:

import sys
sys.path
sys.path.append(r”C:\Scripts\OverwriteFS”)

Then import and execute the Overwrite FS.py script

import OverwriteFS
import arcgis
gis = arcgis.GIS(profile = “Keith VanGraafeiland“)
item = gis.content.get(“326829d3d88b4c7093c1abad69da5553“)
OverwriteFS.overwriteFeatureService(item,”https://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS?service%3DSOS%26request%3DGetObservation%26version%3D1.0.0%26observedProperty%3Dsea_water_temperature%26offering%3Durn%3Aioos%3Anetwork%3ANOAA.NOS.CO-OPS%3AAll%26responseFormat%3Dtext%2Fcsv%26unit%3DCelsius“)

*Items in bold are user/item specific.

Method 2 – Using Python
Calling and executing the script from python command line

Navigate to: “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3”
Run (double click) python.exe
First, register the directory where we saved the OverwriteFS.py script.
Type the following:

import sys
sys.path
sys.path.append(r”C:\Scripts\OverwriteFS”)

Then import and execute the Overwrite FS.py script

import arcgis
import OverwriteFS
gis = arcgis.GIS(profile = “Keith VanGraafeiland“)
item = gis.content.get(“326829d3d88b4c7093c1abad69da5553“)
OverwriteFS.overwriteFeatureService(item,”https://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS?service%3DSOS%26request%3DGetObservation%26version%3D1.0.0%26observedProperty%3Dsea_water_temperature%26offering%3Durn%3Aioos%3Anetwork%3ANOAA.NOS.CO-OPS%3AAll%26responseFormat%3Dtext%2Fcsv%26unit%3DCelsius“)

*Items in bold are user/item specific.

Method 3 – Command Prompt
Launch Command Prompt and call the Python Program to execute the script:

“C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe” “C:\Scripts\OverwriteFS\OverwriteFS.py” “Keith VanGraafeiland” “326829d3d88b4c7093c1abad69da5553” “SOS_WaterTemperature_Collection_Latest” “https://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS?service%3DSOS%26request%3DGetObservation%26version%3D1.0.0%26observedProperty%3Dsea_water_temperature%26offering%3Durn%3Aioos%3Anetwork%3ANOAA.NOS.CO-OPS%3AAll%26responseFormat%3Dtext%2Fcsv%26unit%3DCelsius“

*Items in bold are user/item specific.

The layer will be checked/updated each time the script is ran.

Step 7: Setting up script to run in Task Scheduler

Launch Task scheduler and “Create a Basic Task”.

Create a new Basic Task and name it appropriately.
Under security options: check “Run whether user is logged on or not” (so it will run without requiring the user to be logged in)
Select “Do not store password” (so it will run without requiring a stored user account password – which can change from time to time)

Inputs for 1 and 2 in image above:

1. “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe”
2. “C:\Scripts\OverwriteFS\OverwriteFS.py” “Keith VanGraafeiland” “326829d3d88b4c7093c1abad69da5553” “SOS_WaterTemperature_Collection_Latest” “https://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS?service%3DSOS%26request%3DGetObservation%26version%3D1.0.0%26observedProperty%3Dsea_water_temperature%26offering%3Durn%3Aioos%3Anetwork%3ANOAA.NOS.CO-OPS%3AAll%26responseFormat%3Dtext%2Fcsv%26unit%3DCelsius”

*Items in bold are user/item specific.

Schedule the task to run as often as you like using the triggers tab.
Click “OK” to save the properties.
In the Task Scheduler Library – Right click and run the task to test and make sure it completes successfully.  When completed the task should indicate “Operation Completed Successfully”.

It’s possible to setup your MAC to run Automator to do the same lift as window’s task manager as well.

Check the content item, now you can see the “Data Update Details” reflect the most recent update.

Since this layer (Water Temperature) is constantly being refreshed from a sensor network, the updates will occur every time it checks against the source data.   This script can also be used against data that is updated less frequently to ensure the hosted feature class is using the most recent version of the source data.  This script and methodology can prove to be very useful for keeping your data current and reliable.

Pros: Easy, Simple
Cons: Can be lengthy overwrite if large or complex data

This script is really easy to use.  If you or your organization relies on having updated layers this is a good method to automate that process.  There are many options for running the script to update the data, I prefer using task manager.  This also works with other formats.  I used a .CSV, but it works with other formats such as shapefiles, geodatabase, and geoJSON.  With the data being updated reliably and regularly – now you have the ability to detail out the associated content item and prepare the data for nomination to Living Atlas.

We have an update coming for the script.  It will include some new enhancements and additional functionality that includes:

• Persist Post-Publishing Service and Layer changes following an overwrite. Like Field Alias and Layer renaming, Optimization and Cache settings, and Initial Extent settings.
• Support for highly available Feature Views leveraging twin Feature Services (Layer Swizzle workflow).
• Initial support for Appending data to a Feature Service.

Links:

Query for Data: https://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS?service%3DSOS%26request%3DGetObservation%26version%3D1.0.0%26observedProperty%3Dsea_water_temperature%26offering%3Durn%3Aioos%3Anetwork%3ANOAA.NOS.CO-OPS%3AAll%26responseFormat%3Dtext%2Fcsv%26unit%3DCelsius

Overwrite Feature Service Script: http://esriurl.com/OverwriteFS

Other “Live Feeds” scripts are available here: http://www.esriurl.com/LiveFeed