Eight essential real-time techniques with ArcGIS Maps SDKs for Native Apps

It wasn’t long ago that maps were strictly static documents whose information was quickly out-of-date. In the last few decades, GIS and the internet have revolutionized the map, turning it from a static document into a dynamic online resource. To provide data that is relevant and up-to-date, mobile data collection and crowdsourcing now ensure data updates occur quickly and efficiently. Despite such advances, traditional GIS data updates are still not frequent or consistent enough for some use cases.

In this article, I walk through creating an application that highlights some common workflows for processing data in real-time using ArcGIS Maps SDKs for Native Apps. I use .NET (C#) code to show some examples, but the concepts apply to each of the Native SDKs that support consuming data from a stream.

1. Connect to a data stream
2. Filter data coming from a stream
3. Display stream data
4. Listen for stream data updates
5. Find unique values for a set of stream objects
6. Keep the display tidy with scale-dependent rendering
7. Select and identify objects from the stream
8. Listen for updates to a specific stream object

For an overview of creating real-time apps using your preferred Native SDK, see the Real-time topic in the developer guide: Java | .NET | Qt | Swift.

Keeping it real

Sometimes, data-driven decisions require information for current (or very recent) conditions in a quickly changing environment. Real-time apps rely on data that is updated frequently to show its current (or nearly current) state. This data commonly comes from GPS and other sensors that provide location and attribute updates. You can connect to a data stream to bring data updates directly into your ArcGIS Maps SDK for Native Apps application and process them in real time. Once established, a connection to a data stream continues to deliver updates until the connection is closed.
The following are examples of real-time apps in various application areas:

• Utilities—Monitor power consumption across the grid to respond quickly to demand changes or outages.
• Public safety—Identify responders in the area best suited to respond to an incoming emergency.
• Environmental—Monitor stream gauges and current weather conditions to assess the risk of floods.
• Transportation—Track vehicles across public transportation routes so riders can anticipate delays.

Real-time apps may receive and interpret observations from various data streams, including stationary sensors like weather stations, moving assets like vehicles, or point-in-time events like crime and accidents.

API overview

• DynamicEntityDataSource connects to a stream data source and provides notifications for data updates and connection status changes. The connection is maintained (automatically reconnected as needed, for example) to make sure data continues to flow.
• DynamicEntity represents one of the real-world objects described by the data source.
• DynamicEntityObservation represents a snapshot in time of the state (attributes and location) of a dynamic entity.
• DynamicEntityLayer displays dynamic entities and observations and updates them as data is received.

A flight-tracking example

I include an app that illustrates some of the essential concepts, and perhaps a few pitfalls, you may encounter. The app uses a stream service created with ArcGIS GeoEvent Server with global flight updates from a FlightAware data feed. You can filter the stream to only show dynamic entities in the current map extent and list all flights going to a selected airport. Selecting a specific flight shows attributes for the entity (airline, speed, altitude, origin, destination, and so on). The selected flight is also displayed as a graphic that is tracked in an inset scene view. Click a dynamic entity in the map to see a callout with its flight number and departure and arrival airports.

Download the code and try it yourself: flight-dynamic-entities

Connect to a data stream

The constructor for an ArcGISStreamService requires the service URI. If the service requires authentication, you’ll need to handle providing the appropriate credentials or challenging the user for them. See the ArcGIS Maps SDK for .NET OAuth sample for an example.

Properties on the ArcGISStreamService control connection behavior, define how long previous observations are stored, and listen to notifications. Note that the service won’t start streaming data until it’s loaded and connected. You can do this explicitly by calling LoadAsync() and ConnectAsync(). It can also happen implicitly when you add a layer that uses the stream service as its data source. Creating a DynamicEntityLayer from a stream data source is discussed later in this article.

  
  // Create the stream service from the service URL.
  var streamService = new ArcGISStreamService(new Uri(serviceUrl));

  // Set option to purge updates older than 12 hours.
  streamService.PurgeOptions.MaximumDuration = new TimeSpan(12, 0, 0);

  // Set the max reconnection interval time and max reconnect attempts.
  streamService.ReconnectionInterval = new TimeSpan(0, 0, 15); 
  streamService.MaximumReconnectionAttempts = 6;

  // Handle changes to the connection status.
  streamService.ConnectionStatusChanged += ConnectionStatusChanged;

Filter data from the stream

Apply a ArcGISStreamServiceFilter to the data source to filter the data received from the stream. You can filter based on a specific area (envelope) or with an attribute expression (or both). The filter is applied on the server to limit the data sent to the client. Applying a filter to the stream can reduce the amount of data stored locally.

     
  // Filter to show flights higher than 10,000 ft within the map extent
  // (define this before calling LoadAsync() on the service).
  var streamFilter = new ArcGISStreamServiceFilter
  {
      Geometry = currentMapExtent,
      WhereClause = "alt >= 10000"
  };
  streamService.Filter = streamFilter;

  await streamService.LoadAsync();

Display data coming from the stream

Once the dynamic entity data source connection is loaded and connected, data will begin to flow from the stream. Create a DynamicEntityLayer from the data source and add it to your map or scene to see dynamic entities appear. If you create a DynamicEntityLayer with a data connection that is not loaded or connected, the connection will load and connect automatically when the layer is added to the map or scene. Like other layers you may have worked with (such as FeatureLayer), you can control rendering (symbols), labeling, and various other display properties. The layer’s TrackDisplayProperties allows you to control the display of previous observations. You can set a maximum number of observations to show, connect them with a track line, and render the observations and/or track lines.

  // Create a dynamic entity layer.
  _dynamicEntityLayer = new DynamicEntityLayer(streamService)
  {
      // Create and apply a simple renderer.
      Renderer = new SimpleRenderer(
      new SimpleMarkerSymbol(SimpleMarkerSymbolStyle.Circle, 
                              System.Drawing.Color.Red, 
                              6))
  };

  // Turn tracks, entities, and labels off initially
  // (The app will manage the display according to the current map scale).
  _dynamicEntityLayer.TrackDisplayProperties.ShowPreviousObservations = false;
  _dynamicEntityLayer.TrackDisplayProperties.ShowTrackLine = false;
  _dynamicEntityLayer.LabelsEnabled = false;

  // Add the layer to the map.
  MainMapView.Map.OperationalLayers.Add(_dynamicEntityLayer);

Depending on the nature of your data, you may want to toggle additional display settings according to the current map scale. For example, labels, previous observations, and track lines might clutter the display at a small scale. Often, that level of detail is only meaningful and clear at a larger (more zoomed-in) scale.

Listen for when dynamic entities are added or removed

As information comes from the stream, the local data store adds or removes dynamic entities as needed. Initial observations mean a new flight, while flights with no more associated observations are removed (purged). While the DynamicEntityLayer manages the display of the dynamic entities, you can handle the DynamicEntityReceived and DynamicEntityPurged events for notifications about dynamic entities added or removed from the local data store.

As a simple example, you could increment or decrement a count to keep track of the current number of entities. I use a variable called DynamicEntityCount to keep track of the number of flights currently in the data store and display that value in a label.

  private void StreamService_DynamicEntityReceived(object? s, DynamicEntityEventArgs e)
  {
      DynamicEntityCount++;
  }

  private void StreamService_DynamicEntityPurged(object? s, DynamicEntityEventArgs e)
  {
      DynamicEntityCount--;
  }

Get a list of unique values for the current dynamic entities

Getting unique values for an attribute is a common workflow when using geographic data. With dynamic entities, however, it can be a bit trickier for a couple of reasons. First, you can’t query the dynamic entity data source directly. Second, the values coming from the stream are not always predictable and may change rapidly. For example, working with flight data from around the globe, I can’t know for sure which destination airports will appear in the feed for a given time, especially if I allow the user to filter the data with a map extent. I could build out a master list of all possible airports and provide that in my app, but that adds too much clutter to the list. A much better experience for the user is to only show the relevant values for current data from the feed.

What you can do instead is listen to update notifications from the stream. Use them to maintain a list of unique values from the observations as they come in. First, you need to handle one of the following events that are raised when data is received:

• DynamicEntityReceived notifies when a new dynamic entity is received from the stream.
• DynamicEntityObservationReceived notifies of a new observation, either for a new or existing dynamic entity, is received.

Get unique destination airports

In my app, the user can change the selected airport anytime, perhaps after most of the dynamic entities are received. I therefore handle the observation received event. To work as expected in a multi-threaded environment, my code uses a ConcurrentDictionary (rather than a plain Dictionary) with the airport code as the unique key. When an observation is received, the dictionary is used to see if the airport code has already been added. If it’s a new airport code, it’s added to an ObservableCollection<string>. The observable collection is bound to a ComboBox to display the unique airport codes currently in the layer.

  private void StreamService_DynamicEntityObservationReceived
                (object? sender, DynamicEntityObservationEventArgs e)
  {
      // Get the attributes dictionary for this observation.
      var attr = e.Observation.Attributes;
      // Make sure there's a value for the destination airport.
      if (attr["dest"] != null)
    {
        // Get the destination airport code as an upper-case string.
        var thisVal = attr["dest"].ToString().ToUpper();

        // Call 'TryAdd' on the ConcurrentDictionary.
        if (_uniqueAirportCodesDictionary.TryAdd(thisVal, thisVal))
          {
              // If new code, add it to the observable collection.
              Application.Current?.Dispatcher?.Invoke(() =>
              {
                  CurrentAirports.Add(thisVal);
              });
          }
      }
  }

This code uses the hard-coded value for the destination airport field (“dest”). You could easily modify this code to look for unique values in a field selected by the user. For example, you could also create a unique list of airlines, departure airports, or aircraft types for the current flights. You could do something more sophisticated here if you wanted, like keep a count for each destination airport. You might also handle the DynamicEntityPurged event to decrement the counts (and remove airports) as appropriate.

Find dynamic entities that match an attribute expression

When an airport is selected from the list, a UniqueValueRenderer is applied to the layer. The renderer shows flights traveling to that airport in red and all others in blue. The selected airport is also geocoded and the inset weather map zooms to its location.

Similar to the code above to find unique values for a field, I use the DynamicEntityObservationReceived event handler to get a list of flights with a selected destination airport. The logic is essentially the same: when a new observation is received from the stream, I check its attribute values and create a list of unique values. This time, it’s a list of flight numbers that have the selected airport code for their destination.

  private void StreamService_DynamicEntityObservationReceived
            (object? sender, DynamicEntityObservationEventArgs e)
 {
    // Get the attributes dictionary for this observation.
    var attr = e.Observation.Attributes;
    // Make sure there's a value for the destination airport.
    if (attr["dest"] != null)
    {
        // Get the destination airport code as an upper-case string.
        var thisVal = attr["dest"].ToString().ToUpper();       

        // ...< code here to get unique airport destinations >...

        // Does the destination ("dest") match the selected airport?
        if (_findAirport is not null && thisVal == _findAirport)
        {
            // The "ident" field holds the flight ID ("DAL1234", eg).
            var displayId = attr["ident"]?.ToString();

            if (!string.IsNullOrEmpty(displayId))
            {
                // Get the dynamic entity for this observation.
                var dynamicEntity = e.Observation.GetDynamicEntity();
                // Try to add it to the ConcurrentDictionary.
                if (_dynamicEntityIdDictionary.TryAdd(displayId, dynamicEntity.EntityId))
                {
                    // If a new ID, add it to the collection.
                    Application.Current?.Dispatcher?.Invoke(() =>
                    {
                        DynamicEntityCollection.Add(dynamicEntity);
                    });
                }
            }
        }
    }
 }

Notice that you can get the dynamic entity from an observation. Likewise, you can get the collection of observations that belong to a dynamic entity.

To accurately track the current set of flights to the selected airport, I also handle the DynamicEntityPurged event to check for entities that are no longer displayed (flights that have landed, for example). If an entity from the list is purged, I remove it from the collection.

  private void StreamService_DynamicEntityPurged
            (object? sender, DynamicEntityEventArgs e)
 {
    DynamicEntityCount--;

    // Get the dynamic entity being purged.
    var dynamicEntity = e.DynamicEntity;

    // Get the flight ID.
    var displayId = dynamicEntity.Attributes["ident"]?.ToString();

    if (!string.IsNullOrEmpty(displayId))
    {
        // Try to remove it from the ConcurrentDictionary.
        if (_dynamicEntityIdDictionary.TryRemove(displayId, out long entityId))
        {
            // If removed, also remove from the collection.
            Application.Current?.Dispatcher?.Invoke(() =>
            {
                DynamicEntityCollection.Remove(dynamicEntity);
            });
        }
    }
 }

Show more details when zoomed in

When displaying a lot of dynamic entities on the map, it might not make sense to display additional details. Previous observations, track lines, and labels might be informative with a handful of dynamic entities on the display. However, these can clutter the display when there are several hundred or thousands of entities. A good technique is to show additional information only at large scales (when zoomed in). At smaller scales (when zoomed out), hide that information to help declutter the display.

The flight-tracking map shows lots of dynamic entities at small scales. I decided to show and hide previous observations, track lines, and labels according to the current map scale. I did this by handling the ViewpointChanged event on the MapView.

  private void MainMapView_ViewpointChanged(object sender, System.EventArgs e)
  {
    // Get the current map scale.
    var currentScale = MainMapView.MapScale;
    // Show tracks at scales larger than 1/400000
    // (otherwise hide them).
    var showTracks = (currentScale <= 400000);

    // Layer might not be loaded yet.
    if (_dynamicEntityLayer != null)
    {
      // Toggle tracks, observations, and labels.
      _dynamicEntityLayer.TrackDisplayProperties.ShowTrackLine = showTracks;
      _dynamicEntityLayer.TrackDisplayProperties.ShowPreviousObservations = showTracks;
      _dynamicEntityLayer.LabelsEnabled = showTracks;
    }
  }

Select or identify dynamic entities

Like any other geoelement, you can identify dynamic entities with a tap (or click) on the view. You can also select (and unselect) dynamic entities (and dynamic entity observations) in a DynamicEntityLayer.

When a flight is selected in the ListBox, I use the following code to select it on the map. The DynamicEntityLayer also provides methods for selecting or unselecting a set of entities (rather than just one) and for getting the currently selected entities.

  // Get the item selected in the ListBox.
  SelectedFlight = e.AddedItems[0] as DynamicEntity;
  if (SelectedFlight == null) { return; }

  // Clear any currently selected flights.
  _dynamicEntityLayer.ClearSelection();

  // Select the flight in the layer.
  _dynamicEntityLayer.SelectDynamicEntity(SelectedFlight);

The following code identifies a dynamic entity observation when the MapView is tapped. From the observation, you can get the dynamic entity it applies to. This technique allows the user to click any observation and have the callout jump to the dynamic entity itself (essentially, the most current observation). As the dynamic entity location is updated, the callout will move with it (no additional code or logic is required).

  private async void MainMapView_Tapped
            (object sender, Esri.ArcGISRuntime.UI.Controls.GeoViewInputEventArgs e)
  {
    // First dismiss any existing callout (previous identify, eg).
    MainMapView.DismissCallout();

    if(_dynamicEntityLayer is null) { return; }

    // Identify the dynamic entity layer using the tap/click location.
    var idResult = await MainMapView.IdentifyLayerAsync
                   (_dynamicEntityLayer, 
                    e.Position, 
                    4, 
                    false, 
                    1);

    // Get the first DynamicEntityObservation from the results.
    var obs = idResult.GeoElements.FirstOrDefault() as DynamicEntityObservation;

    // Get the DynamicEntity for this observation.
    // (if the user clicked a previous observation,
    // this will "jump" the callout to the last one).
    var de = obs?.GetDynamicEntity();

    if (de != null)
    {
        // Get the flight ID, departure and arrival airports.
        var flight = de.Attributes["ident"]?.ToString();
        var from = de.Attributes["orig"]?.ToString();
        var to = de.Attributes["dest"]?.ToString();
        var fromTo = $"From '{from}' To '{to}'";

        // Show the flight info in a callout at the entity location.
        var calloutDef = new CalloutDefinition(flight, fromTo);
        MainMapView.ShowCalloutForGeoElement(de, e.Position, calloutDef);
    }
  }

Handle updates for one dynamic entity

Listen for updates to a specific dynamic entity by handling its DynamicEntityChanged event. In the event handler, you can respond to observations received or purged for the entity. You can also be notified when the entity itself is purged. In my app, I subscribe to this event for the selected flight. When I get an update (received observation), I use it to update a graphic in an inset SceneView. The graphic is used to track the selected flight’s location, altitude, and heading. There are other ways you could do this, such as creating another stream data source and filtering it for the one entity you’re interested in, then creating another dynamic entity layer to display it. I felt like manually updating the graphic was the most straightforward approach for my needs.

  private void SelectedFlight_DynamicEntityChanged
            (object? sender, DynamicEntityChangedEventArgs e)
  {
    // Get updates from a new observation.
    var obs = e.ReceivedObservation;
    if(obs != null)
    {
      // Get the updated altitude for the graphic point z-value.
      double planeAltitude = 0.0;
      double.TryParse(obs.Attributes["alt"]?.ToString(),  out planeAltitude);

      var updatedLocation = obs.Geometry as MapPoint;
      var pointZ = new MapPoint
                       (updatedLocation.X, 
                        updatedLocation.Y, 
                        planeAltitude, 
                        updatedLocation.SpatialReference);

      Dispatcher.Invoke(() =>
      {
          // Update the geometry and the heading attribute.
          _planeGraphic.Geometry = pointZ;
          _planeGraphic.Attributes["heading"] = obs.Attributes["heading"];
      });
    }
  }

Summary

ArcGIS Maps SDKs for Native Apps abstract much of the work required to consume data from a feed, process it in real time, and display it dynamically in your app. Use the API to connect to a supported stream data source and it can automatically display and update dynamic entities created from the stream. While no other work is required to read and display the data, several events exposed by the data source provide fine-grained control of the data updates and connection status.

The nature of stream data can vary widely: delivering a few or several thousands of entities, describing moving or stationary objects, receiving updates extremely frequently or occasionally, and so on. Hopefully, this article has given you an idea of the types of apps you can create and some of the common techniques you can use to consume a variety of data.

Try it yourself!

• flight-dynamic-entities [GitHub repo]: the project described in this article
• Add dynamic entity layer [Native SDKs Sample]: Java | .NET | Qt | Swift
• Work with dynamic entities [Native SDKs Guide Doc]: Java | .NET | Qt | Swift
• Stream services [ArcGIS Enterprise Doc]
• Get started with ArcGIS Velocity [Tutorials, Articles, and Videos]
• GeoEvent Server [ArcGIS Enterprise Doc]