Craft your own dynamic entity data source for ArcGIS Maps SDKs for Native Apps

If you’ve developed an app using ArcGIS Maps SDK for Native Apps, then you’ve probably worked with features and graphics. Have you met their geoelement cousin, dynamic entity? As part of the geoelement family, dynamic entities have attributes and geometry. They’re used to represent observations from a data stream to enable some really cool real-time workflows like:

• Tracking moving vehicles, such as delivery vehicles, commercial airline traffic, public transit, or emergency responders.
• Monitoring information from stationary sensors, like weather stations, traffic sensors, or stream gauges.
• Sending and receiving notifications about point-in-time events, such as crime or accidents.

In this article, I’ll give an overview of dynamic entity data sources and step through the process of creating one. I’ll use a .NET MAUI example created using ArcGIS Maps SDK for .NET and C# to highlight these steps. While the code and some of the implementation details differ across Native Maps SDKs, the basic concepts are the same. If you’re new to dynamic entities or want to simply work with them out-of-the-box, you might prefer my previous blog. See Eight essential real-time techniques with ArcGIS Maps SDKs for Native Apps for a more general discussion. However, if you want to craft your own custom dynamic entity data source, read on!

• Why create a custom data source?
• Example: Bike rental stations
• Implementation steps
• Summary

It’s about to get real (time)

When working with dynamic entities, a great deal of work is managed by the dynamic entities API. A DynamicEntityDataSource provides data (observations) as a data stream, delivering a constant series of updates. Once connected to such a data source, dynamic entities automatically update as new observations come from the stream. This information includes updated locations and attributes. Attributes describe the properties of each entity. For moving things, these may be speed, heading, or altitude. For our bike stations, attributes describe the available bike inventory. A dynamic entity layer from the data source can manage the display of dynamic entities on the map.

The initial release of the dynamic entities API at version 200.1 only supported ArcGIS stream services as a data source. With the 200.2 release, however, Native Maps SDKs also support custom dynamic entity data sources. This gives you the ability to wrap just about any data feed as a DynamicEntityDataSource for use in your app.

Why create a custom data source?

Perhaps your app can use an existing ArcGIS stream service as a dynamic entity data source. If you have access to ArcGIS GeoEvent Server or ArcGIS Velocity you can easily create your own. In those cases, you may not benefit much from creating a custom dynamic entity data source. GeoEvent Server and ArcGIS Velocity both allow you to configure a data source using a variety of data feed formats. You can then expose the data as an ArcGIS stream service. Supported feed types include AWS, Azure, HTTP polling, and many more. If, however, you have a data feed that is not in a standard format or you cannot easily expose your feed with an ArcGIS stream service, a custom data source is a great option.

From zero to dynamic in five steps

OK … creating a custom dynamic entity data source might not be “simple”. Of course, the complexity involved depends on the nature of the data feed and your requirements. But, good news! Once you create your data source, it fits into the dynamic entities API to leverage additional functionality. For example, you can display it using a dynamic entities layer, get notifications for data updates, and manage the local data cache. For dynamic entities that move, a dynamic entity layer shows tracks of previous locations and has options for track display.

Example: Bike rental stations

I created a .NET MAUI app to illustrate a custom DynamicEntityDataSource. The app shows bike rental stations for a few major cities. These stations are, well, stationary, so you won’t see anything “dynamic” about their locations on the map. Attributes like bike availability, however, are updated frequently, so these values are dynamic. The color of the stations indicates the number of bikes available. Station attributes show the last inventory update time and the change in available bikes (+/-) since the previous update. The page shows a summary of available bikes for the entire city that updates as observations come in. The user can select favorite stations to quickly see inventory updates for those stations.

When observations come in, corresponding stations flash red to indicate a bike left or blue to indicate a bike returned.

The bike information comes from the CityBikes API. This is a REST API that sends information in a JSON response. The app uses a timer to periodically request information from the service. It then deserializes the response into classes that represent the stations (point locations and relevant attributes).

To create a custom dynamic entity data source, follow these basic steps.

1. Create a class for your custom data source that derives from the ArcGIS Maps SDK DynamicEntityDataSource base class
2. Connect to a data feed
3. Define and return metadata about your data source
4. Implement logic to read data from the feed and add observations Full disclosure: this step will likely be composed of several substeps (and cups of coffee)
5. Clean up when the data source is disconnected

You can then consume your custom dynamic entity data source in your ArcGIS Maps SDK app.

1. Create an instance of your data source
2. Create a new DynamicEntityLayer to display the data source
3. Handle data source events to get update notifications

Create a class for your data source

Start by creating a new class to represent your dynamic entity data source. The class must inherit from the DynamicEntityDataSource base class and override a handful of methods. I describe the details of these overrides in a bit. My class (CityBikesDataSource) consumes a data feed that shows available bikes for rental stations in some major cities.

My class requests updates for a specified city at a specified interval. It has variables to store the CityBikes URL for the selected city and a timer (IDispatcherTimer) to periodically get updates. I also store the city name and a dictionary of the last observations received. This allows me to evaluate the change in bike inventory between updates. These are all details specific to my implementation and could vary greatly on your data feed and use case.

  
internal class CityBikesDataSource : DynamicEntityDataSource
{
    // Timer to request updates at a given interval.
    private readonly IDispatcherTimer _getBikeUpdatesTimer = Application.Current.Dispatcher.CreateTimer();
    // REST endpoint for a city in the CityBikes API (http://api.citybik.es/).
    private readonly string _cityBikesUrl;
    // Previous observations for bike stations (to evaluate change).
    private readonly Dictionary<string, Dictionary<string, object>> _previousObservations = new();
    // Name of the city.
    private readonly string _cityName;

In the constructor for my class, I accept values for the city name and URL (strings) and an update interval in seconds (int). I store those values in the appropriate variables and then define a function to run when the timer interval expires. The PullBikeUpdates() function contains the logic to send a request to the REST endpoint, deserialize the results, and add observations to the data source.

    public CityBikesDataSource(string cityName, string cityBikesUrl,
        int updateIntervalSeconds)
    {
        // Store the name of the city.
        _cityName = cityName;
        // Store the timer interval (how often to request updates from the URL).
        _getBikeUpdatesTimer.Interval = TimeSpan.FromSeconds(updateIntervalSeconds);
        // URL for a specific city's bike rental stations.
        _cityBikesUrl = cityBikesUrl;
        // Set the function that will run at each timer interval.
        _getBikeUpdatesTimer.Tick += (s, e) => _ = PullBikeUpdates();        
    }

Connect to a data feed

Your custom DynamicEntityDataSource must override the OnConnectAsync method. This is where you put any logic required to connect to the data feed. For my CityBikesDataSource class, I simply start the timer so updates will start coming in at the specified interval.

    protected override Task OnConnectAsync(CancellationToken cancellationToken)
    {
        // Start the timer to pull updates periodically.
        _getBikesTimer.Start();
        

        return Task.CompletedTask;
    }

It might be tempting to make a call to PullBikeUpdates() here to get an initial set of observations. Otherwise, the user has to wait until the next timer interval before anything appears on the map, right? Unfortunately, you cannot add observations until the data source is connected (when this override completes). To handle this dilemma, I added a public method to populate the initial set of dynamic entities. I’ve spared you the details here. Refer to the bike-rental-stations-maui project on GitHub if you’re interested.

    public async Task GetInitialBikeStations()
    {
        // Exit if the data source is not connected.
        if (this.ConnectionStatus != ConnectionStatus.Connected) { return; }

        // <Logic here to add dynamic entity observations>
    }

The consuming code calls this method after the connection is established (I’ll show you that code later).

Define metadata about your data source

You need to provide metadata about some important aspects of your data source. Specifically, the schema (fields) to include for observations, a field name that uniquely identifies dynamic entities, and the spatial reference used for their geometry. Notice that geometry is not defined as a field in the schema. Instead, you create points from the latitude and longitude values. You then add an observation and provide the location and attributes. I’ll describe that process later in the article. As of this writing (version 200.2), only point geometry is supported for dynamic entities.

Define data source metadata with a DynamicEntityDataSourceInfo object. Return it from an override of the OnLoadAsync() method.

    protected override Task<DynamicEntityDataSourceInfo> OnLoadAsync()
    {
        // When the data source is loaded, create metadata that defines:
        // - A schema (fields) for the observations (bike stations)
        // - Which field uniquely identifies entities (StationID)
        // - The spatial reference for the station locations (WGS84)
        var fields = new List<Field>
        {
            new Field(FieldType.Text, "StationID", "", 50),
            new Field(FieldType.Text, "StationName", "", 125),
            new Field(FieldType.Text, "Address", "", 125),
            new Field(FieldType.Text, "TimeStamp", "", 50),
            new Field(FieldType.Float32, "Longitude", "", 0),
            new Field(FieldType.Float32, "Latitude", "", 0),
            new Field(FieldType.Int32, "BikesAvailable", "", 0),
            new Field(FieldType.Int32, "EBikesAvailable", "", 0),
            new Field(FieldType.Int32, "EmptySlots", "", 0),
            new Field(FieldType.Text, "ObservationID", "", 50),
            new Field(FieldType.Int32, "InventoryChange", "", 0),
            new Field(FieldType.Text, "ImageUrl", "", 255),
            new Field(FieldType.Text, "CityName", "", 50)
        };
        var info = new DynamicEntityDataSourceInfo("StationID", fields)
        {
            SpatialReference = SpatialReferences.Wgs84
        };

        return Task.FromResult(info);
    }

The values for observations might come directly from the data feed (for example StationName, Address, and BikesAvailable attributes). They may also come from values you provide or calculate yourself (for example, the InventoryChange and ImageUrl).

Read data from the feed

The details for how updates from the data feed are read and processed can vary according to your implementation. I won’t go into detail about specifics for my CityBikesDataSource class. If you’re interested, check out the code on GitHub.

To start with, I check the connection status before requesting updates to avoid a potential exception. You cannot add observations to your data source if it’s not in a connected state.

   if (this.ConnectionStatus != ConnectionStatus.Connected) { return; }

When you read from the feed, you’ll create a dictionary of attributes (Dictionary<string, object>) and a location (MapPoint) for each observation. You then store the observation by calling AddObservation() and passing the attributes and location.

    // --Code here to get a set of updates from the feed—-

    // Process each update.
    foreach (var update in bikeUpdates)
    {
         var attributes = new Dictionary<string, object>
         {
              { "StationID", update.StationInfo.StationID },
              { "StationName", update.StationName },
              { "Address", update.StationInfo.Address },
              { "TimeStamp", update.TimeStamp },
              { "Longitude", update.Longitude },
              { "Latitude", update.Latitude },
              { "BikesAvailable", update.BikesAvailable },
              { "EBikesAvailable", update.StationInfo.EBikesAvailable },
              { "EmptySlots", update.EmptySlots },
              { "ObservationID", update.ObservationID },
              { "InventoryChange", 0 },
              { "ImageUrl", "" },
              { "CityName", _cityName }                    
         };
         // Create a MapPoint for the location from the lat/long values.
         var location = new MapPoint(update.Longitude, update.Latitude, SpatialReferences.Wgs84);

         // Add the observation to the data source.
         AddObservation(location, attributes);
    }

For the bike updates, I discovered I was adding an observation for each bike station on every request. I added observations even when there was no change for a station. This was pretty inefficient since I was adding many more “updates” than I needed to. I also realized that this made my notification events essentially meaningless. They fired whether a change had occurred in the available bikes for that station or not.

The solution was to use a dictionary to store the previous set of values. I could then compare each station’s previous inventory with the updated values. If the values for available bikes or e-bikes change, I record the inventory change and add the observation. If the values are the same for available bikes, I skip the update. For the consumer of the data source, this means notifications only arrive when a station’s values change.

    // Check if the update has different values for BikesAvailable or EBikesAvailable.
    if ((int)attributes["BikesAvailable"] != (int)lastObservation["BikesAvailable"] ||
        (int)attributes["EBikesAvailable"] != (int)lastObservation["EBikesAvailable"])
    {
         // Calculate the change in inventory.
         var stationInventoryChange = (int)attributes["BikesAvailable"] – 
                 (int)lastObservation["BikesAvailable"];
         attributes["InventoryChange"] = stationInventoryChange;
         totalInventoryChange += stationInventoryChange;

         // Add the update to the data source.
         AddObservation(location, attributes);
    }

Clean up when the data source is disconnected

You must override the OnDisconnectAsync method on the data source to handle when the data source is disconnected. The code you add here can vary according to your implementation. For me, I simply need to stop the timer and clear the dictionary of previous observation values.

    protected override Task OnDisconnectAsync()
    {
        _getBikeUpdatesTimer.Stop();
        _previousObservations.Clear();

        return Task.CompletedTask;
    }

At this point, you can consume your custom DynamicEntityDataSource in your app as you would any other data source.

Create an instance of your data source

Create a new instance of your dynamic entity data source class and pass in all required arguments. For my CityBikesDataSource, this is the city name, a URL for the bike stations, and the seconds between updates.

    // Create an instance of the custom dynamic entity data source.
    _cityBikesDataSource = new CityBikesDataSource(cityName, cityBikesUrl, UpdateIntervalSeconds);

Get an initial set of dynamic entities

Remember that I couldn’t add logic inside the OnConnect() override to get an initial set of dynamic entities? You might recall that this was because the data set isn’t officially connected until after that method completes. I added a separate public method to get the initial set of data (GetInitialBikeStations). I can now call that from the code that consumes the data source. The trick is to make sure the data source is connected before making the call. I can do that by handling the ConnectionStatusChanged event and then making the call once connected.

    // When the connection is established, request an initial set of data.
    _cityBikesDataSource.ConnectionStatusChanged += (s, e) =>
    {
        if (e == ConnectionStatus.Connected)
        {
            _ = _cityBikesDataSource.GetInitialBikeStations();
        }
    };

Display the dynamic entities from your data source

ArcGIS Maps SDK for Native Apps provides a layer designed to display dynamic entities, called DynamicEntityLayer. This layer is created from a DynamicEntityDataSource and manages the display of all dynamic entities and observations. The layer updates as observations are added, purged, or updated in the data source. You can define the display of dynamic entities in the layer using any of the available renderer types (simple, class breaks, unique value, or dictionary). You can also display labels for dynamic entities to display information for the current observation.

Adding a DynamicEntityLayer to a map or scene automatically loads and connects the associated DynamicEntityDataSource.

For bike stations, I created a class breaks renderer to show relative bike availability. Darker green means more bikes and gray means no bikes available.

Handle data source events

When the data source adds observations, the value of the entity ID field indicates the entity to which the observation applies. This field is defined in the OnLoadAsync() override. The data source raises the DynamicEntityReceived event when the first observation for a dynamic entity is added. This indicates that a new dynamic entity has appeared in the data. All subsequent observations will raise the DynamicEntityObservationReceived event, indicating that an existing dynamic entity has been updated.

I use the DynamicEntityReceived event to calculate the initial bike availability for the entire city.

    // Listen for dynamic entities being created, calculate the initial bike inventory.
    _cityBikesDataSource.DynamicEntityReceived += (s, e) =>
    {
        var bikeStation = e.DynamicEntity;
        var emptySlots = (int)bikeStation.Attributes["EmptySlots"];
        var availableBikes = (int)bikeStation.Attributes["BikesAvailable"] +
            (int)bikeStation.Attributes["EBikesAvailable"];

        TotalBikes += emptySlots + availableBikes;
        BikesAvailable += availableBikes;

        BikesOut = TotalBikes - BikesAvailable;
        PercentBikesAvailable = (double)BikesAvailable / TotalBikes;
    };

I handle the DynamicEntityObservationReceived event to respond to updates to stations. I adjust the total bike inventory for the city and call a function to flash each station that has an update. The station flashes red when a bike leaves and blue when a bike returns.

    // Listen for new observations; flash the station and update inventory if there's an update.
    _cityBikesDataSource.DynamicEntityObservationReceived += async (s, e) =>
    {
        var bikesAdded = (int)e.Observation.Attributes["InventoryChange"];
        if (bikesAdded == 0) { return; }

        UpdateBikeInventory(bikesAdded); // note: this might be negative if more bikes were taken than returned.
        await Task.Run(() => FlashDynamicEntityObservationAsync(e.Observation.Geometry as MapPoint, bikesAdded > 0));
    };

Changes for a particular entity are available by handling its DynamicEntityChanged event. This lets you respond to observations received or purged (removed) for the entity and be notified when the entity itself is purged (removed from the data source). I don’t use this event in my app, but a good use case would be to flash the favorite card in the “favorites” collection view when a favorite station is updated.

Display consistent updates

When testing, I noticed observations usually came in as a large group of updates. With an update interval of a few seconds, nothing happens for a few minutes and then a flurry of updates appears at once. That cycle repeated as I watched and waited for updates. For the cities I tested, updates were batched on the server and released every 4-5 minutes.

I wanted to provide a more consistent and smooth display for the updates. I set the interval to four minutes, store those updates, and then use another timer to slowly (consistently) display them in the app. This technique improves the display and the overall user experience at the expense of having the most recent updates. Using this technique, updates are four minutes behind. For me, I thought that was a good trade-off. Check out the CityBikesDataSource code to see what that looks like.

Summary

I hope this article helped you understand the process of creating a dynamic entity data source for your ArcGIS Maps SDK for .NET app. By extending the DynamicEntityDataSource class, there are only a handful of overrides required to create your own data source. Hopefully, you’ve seen some of the advantages of using a custom data source as part of the existing API. Perhaps most importantly, many of the data management and display details will be handled for you.

With the 200.2 release of ArcGIS Maps SDK for Native Apps, you can wrap just about any data feed as a dynamic entity data source for use in your app. While there are other ways to bring such data into your app, using dynamic entities allows you to offload a lot of work to the API. Once your custom data source is complete, you can plug it into the existing dynamic entities API to get a lot of additional functionality. You can display the data source in a dynamic entities layer, get notifications for data updates, and manage the local data cache. For dynamic entities that move, a dynamic entity layer shows tracks of previous locations and provides options for track display.

Try it yourself!

• GitHub repo with the code described in this article
• Custom dynamic entity data source Native Maps SDKs sample: Java | .NET | Qt | Swift
• Add dynamic entity layer Native Maps SDKs samples: Java | .NET | Qt | Swift
• Work with dynamic entities (Native Maps SDKs Guide topic): Java | Kotlin | .NET | Qt | Swift
• ArcGIS Enterprise Stream services documentation
• Get started with ArcGIS Velocity: Tutorials, Articles, and Videos
• ArcGIS Enterprise GeoEvent Server documentation

Each ArcGIS Maps SDK for Native Apps (.NET, Qt, Java, Kotlin, and Swift) provides a rich set of guide documentation, API references, tutorials, and samples to help you build great apps. If you’re new to working with these SDKs, visit the ArcGIS Developers site to get started.