Dev Summit 2023: Custom data feeds

Organizations require the ability to leverage data from a wide range of data sources, including those not natively supported with ArcGIS.

At this year’s Developer Summit plenary, Shreyas Shinde demonstrates how to query data from two such sources—MongoDB and Yelp—by creating custom data feeds.

To kick off his demonstration, Shreyas shows the data that analysts in his organization are interested in leveraging: the locations of Supplemental Nutrition Assistance Program (SNAP) authorized retailers in Connecticut stored in a MongoDB database and their business metrics such as ratings, reviews, and price levels on Yelp.

The first step in configuring a custom data feed is to create a custom provider. Shreyas makes a custom provider for each of his data sources with the help of a new command line tool in ArcGIS Enterprise SDK called cdf. He runs the tool twice: first using the cdf createapp <app name> command to create a custom data app and then cdf createprovider <custom provider name> to create custom providers.

Using the cdf tool to create a custom provider generates boilerplate code, including a model.js file which encapsulates the logic required to access data in the data source. If you have ever worked with Koop—an open-source geospatial data server—this should look familiar. Shreyas updates the JavaScript file for his MongoDB and Yelp providers with pertinent code.

The next step in configuring a custom data feed is to export code and dependencies as a custom data package file (.cdpk) and deploy it to ArcGIS Server. Once the code is in place, Shreyas uses the cdf tool to generate custom data package files for each of his providers by running the cdf export <custom provider name> command. Next, he uploads the files and registers them with ArcGIS Server using ArcGIS Server Administrator Directory.

The last step in configuring a custom data feed is to create a new feature service that references the custom provider registered with ArcGIS Server. To create the feature services, Shreyas submits JSON representations that include information about his data providers.

Shreyas’s data feeds are now ready as read-only feature services, and analysts in his organization can leverage the data using the services’ ArcGIS Server Services Directory URL.