Jul 29, 2019

Using Live Feed Layers in Analytical Workflows

By Gonzalo Espinoza

Have you tried to use near real-time GIS data in your analytical tools, such as tracking a hurricane, assessing areas affected by an earthquake, or mapping flooded areas during a storm? ArcGIS Living Atlas of the World contains a collection of near real-time layers known as Live Feeds. Live Feeds display the latest information available such as active hurricanes, recent earthquakes, or river streamflow forecast. Moreover, these layers can be used in analytical tools, connecting Live Feeds to real-time analysis of current conditions, such as interactively mapping flooded areas for first responders in Austin, Texas.

Utilizing near real-time data in analytical tools often requires the automation of tools and workflows that can run with minimal maintenance or supervision, which can sometimes be challenging. The good news is the development of these tools is easily done using Python. In this blog, I’ll show how to consume Live Feed data from Living Atlas through the ArcGIS REST API and the ArcGIS API for Python using two examples: acquiring river streamflow forecast data and plotting changes in arctic sea ice extent.

Retrieving data from Living Atlas through the ArcGIS REST API

ArcGIS Online items, such as the items in Living Atlas, contain a REST service endpoint which can be leveraged by the ArcGIS REST API. The ArcGIS REST API allows us to interact with the items’ service to query, subset, and retrieve Live Feed data. The Live Feed data returned from the interactive request can be seamlessly included in web apps, workflows and more.

Let’s explore how we can retrieve forecasted river streamflow data from the National Water Model.

Access the REST endpoint

Go to the Living Atlas webpage and search for “National Water Model.”

Select the “National Water Model (10 Day Forecast).”

On the item’s overview page, scroll to the bottom. The endpoint is in the URL box at the bottom-right corner. Click on “view” to open it in a new tab.

The item contains only one layer: “Flow Forecast (cfs)”. Click on it to access the REST endpoint. The webpage displays all the relevant metadata such as extent, time info, and field names. At the bottom, click on “query.”

The query webpage contains boxes to interactively construct a request. Alternatively, we can use the endpoint (i.e. url) https://livefeeds2.arcgis.com/arcgis/rest/services/NFIE/NationalWaterModel_Medium/MapServer/0/query to construct the query string and request the data.

Example: Query forecasted river streamflow data for a single reach of the National Hydrography Dataset (NHD)

We’ll plot the forecasted streamflow for Onion Creek in Austin, Texas. The station ID (i.e. COMID) of Onion Creek in the National Hydrography Dataset (NHD) is 5781733. We can query the forecast for this creek by adding the parameter:

where=egdb.dbo.LargeScale_v2.station_id=5781733

Additionally, we can request (1) to include all fields, (2) order the results by time, (3) return a maximum of 80 records, and (4) to return the data in JSON format.

outFields=*&
orderByFields=egdb.dbo.medium_term_current.timevalue&
resultRecordCount=80&
f=pjson

The query url is constructed as follows:

https://livefeeds2.arcgis.com/arcgis/rest/services/NFIE/NationalWaterModel_Medium/MapServer/0/query?
where=egdb.dbo.LargeScale_v2.station_id=5781733&outFields=*&
orderByFields=egdb.dbo.medium_term_current.timevalue&
resultRecordCount=80&f=pjson

The requested data is returned in json format.

Note: You can load the results in ArcGIS Pro using the JSON to Features tool and create Line Chart.

Forecasted river streamflow for a single reach

Example: Query forecast streamflow data for an area

We’ll subset the National Water Model (NWM) forecast data for South Austin. We provide the following parameters: (1) the subset area geometry, (2) the geometry type, (3) the spatial relationship, and (4) the time stamp 2019-07-22 09:00:00 (i.e. 1563786000000).

geometry={xmin:-10889680,ymin:3524030,xmax:-10873880,ymax:3537115}&
geometryType=esriGeometryEnvelope&
spatialRel=esriSpatialRelIntersects&
time=1563786000000

The query url is:

https://livefeeds2.arcgis.com/arcgis/rest/services/NFIE/NationalWaterModel_Medium/MapServer/0/query?
where=&
time=1563786000000&geometry={xmin:-10889680,ymin:3524030,xmax:-10873880,ymax:3537115}&
geometryType=esriGeometryEnvelope&
spatialRel=esriSpatialRelIntersects&
outFields=*&
resultRecordCount=50&
f=pjson

Note: We are requesting data from a Live Feed layer in which the Time Extent changes frequently. The time selected in this blog post won’t be available after the hosted layer is updated. You can select a different time that falls within the current Time Extent. The current Time Extent and Time Interval can be checked in the REST endpoint and also in the Return Updates operation.

The requested results for the forecast time of 2019-07-22 09:00:00 at South Austin are shown next.

Note: You can quickly transform time expressions from datetime variables to milliseconds from 1970-01-01 and vice versa as follows:

Note: The ARGIS REST API documentation includes a complete description on how to query a feature service.

Accessing Living Atlas data using the ArcGIS API for Python

Living Atlas items are also available through the ArcGIS API for Python. The layers can be queried, plotted, and used in analytical workflows. In addition, we can leverage the ArcGIS API for Python through ArcGIS Notebooks.

Let’s examine how Arctic Sea Ice Extent has changed over time.

Example: Plot the changes in arctic sea ice extent in September

Open an ArcGIS Notebook in ArcGIS Enterprise or in your desktop.

Load the required modules.

Search for the ‘Arctic Sea Ice Extent’ item.

Select the first layer and query the data as a DataFrame.

Visualize the structure of the data set by plotting the top rows.

Subset the DataFrame by selecting only the months of September for all years.

Plot all months of September in the data set.

The python script used in the example will query all the monthly data in the Arctic Sea Ice Extent feature service. As soon as new monthly values are added to the service, those can be queried and plotted with the same python script.

Summary

The ArcGIS Living Atlas of the World includes Live Feed layers that can be queried, subset, and retrieved systematically in analytical workflows using the ArcGIS REST API and the ArcGIS API for Python.

Related Learn ArcGIS resources

Update Real-Time Data with Python: Learn more about how to create your own Live Feed.

More Information?

Join GeoNet and ask a question to our community of experts.

Gonzalo Espinoza

Gonzalo Espinoza is a Principal GIS Engineer at ArcGIS Living Atlas of the World. Gonzalo specializes in hydrologic modeling, real time climate and hydro-informatic tools such as flood mapping and forecasting using National Water Model data. He masters the full geospatial tech stack required to develop scalable and operational services relying on large volumes of EO data in the fields of water and environment. At Living Atlas, he also supports the development of workflows for automation of data updates known as Live Feeds, technology used at Esri’s Disaster Response Program. Before ESRI Gonzalo worked in the development of Water Accounting at UNESCO IHE. He holds a PhD in Civil Engineering from the University of Texas at Austin.

ArcGIS Blog