High performance web apps with big datasets as feature layers

Dealing with big datasets is a frequent topic of discussion among web mapping app developers. The prevalence of huge, crowdsourced datasets ensures that big data will continue to be a common point of discussion when you get more than two web mapping geeks in the same room. The traditional approaches of generating a raster tile cache or using a dynamic map service work, but if you want any interactivity with the underlying data, it requires a server round trip. This post will discuss some of the key concepts behind building a web app that provides an extensive interactive experience while maintaining responsiveness and performance.

The main factor causing big data to be a tricky topic is that browsers cannot elegantly handle upwards of a few thousand features. Send more than a few thousand points, or a few hundred polygons to a browser and you can watch it grind to a halt, especially if you’re running a legacy version of Internet Explorer. This is a problem because slow performance is a death knell for web apps. If someone using your web app has to wait ten seconds, twenty seconds, or longer for five megs of JSON data to be retrieved and displayed, there’s a good chance they’ll just give up and move on. What good is a web app that no one uses?

At some point, you cross a threshold where it’s not practical to send all of your data across the wire and let the client sort it out. That threshold is an ambiguous one (and was explored in a previous post), but you know when you hit it. If you’re a JavaScript developer, how should you deal with big data? If you’ve read my recent posts, this answer will not surprise you: Feature Layers.

Building on the concepts discussed in the previous feature layer posts (generalizing features and vector tiling) and client side graphics performance post, I’ve put together an application that demonstrates how to use Feature Layers to set up custom scale dependencies and follow best practices to ensure performance does not suffer when querying, retrieving and displaying large datasets in an ArcGIS API for JavaScript application: High Performance Maps with Feature Layers.

The goal was to display appropriate US boundaries (states, counties or census block groups) for the current map scale. For example, trying to display census block groups for the whole US would overwhelm the browser with features. This app makes sure that only the states appear at the small scales, counties at the medium scales, and census block groups at the large scales. This ensures that a manageable number of features are always being transferred and displayed.

The app is simple, but the concepts can be applied to just about any web mapping app that has to deal with big data. The total size of the data used by the app is around a couple of hundred megabytes. This is accomplished through custom scale dependencies. This is as simple as creating a few feature layers, listening for their onLoad event and setting their minScale and maxScale properties. Here’s a simple code sample showing how this is done:

var fl = new esri.layers.FeatureLayer(url, options);
dojo.connect(fl, ‘onLoad’, function() {
	fl.minScale = minScale;
	fl.maxScale = maxScale;
});

The app includes a couple of other bells and whistles that demonstrate some of tools that are available when you have your features client side. The first is a rich Popup that displays feature attributes, as well as an option to zoom to the feature when you touch a graphic. The second is the ability to filter features based on population using a slider. Neither of these would perform nearly as well if features weren’t available client side.

The final point to drive home is that the app’s performance is kept snappy because only the required data is being sent to the client. For geometries, that means using maxAllowableOffset to generalize geometries on the server before they’re sent to the client (see my previous post on maxAllowableOffset and generalizing features on the fly for more info). For attributes, only the fields required by the Popup are sent across the wire.

Contributed by Derek Swingley of the ArcGIS API for JavaScript development team