ArcUser Online

• ArcUser Main
• Current Issue
• Previous Issues
• Subscribe
• Advertise
• Submit an Article

Spring 2011 Edition

Augmented Reality & GIS

Making GIS content available in interesting and useful ways

By Keith Mann, Esri Writer

This article as a PDF.

If you own a smartphone, such as an iPhone or Android, you probably have an app that uses augmented reality (AR). This technology superimposes digital information on whatever you're looking at through your phone's camera.

These types of AR apps usually combine an assortment of miniaturized technology devices—optical sensors, accelerometers, GPS, gyroscopes, solid state compasses—along with context-sensitive information, that is delivered as a web service. The service content may be displayed as location-based labels (or billboards) that hover over or in front of objects you see through the camera display. The labels usually present additional information, including photos, but often provide reviews and contact information for things like restaurants, Wi-Fi hot spots, or houses for sale. You might use AR apps to discover and learn more about objects around you.

Some AR apps allow you to use your phone camera to perform sophisticated measurements of distances and angles or even track cosmological events such as the azimuth of the sun. Still other AR apps amazingly recognize patterns in photos, diagrams, and bar codes that then link you to additional content or services, such as product reviews, price comparisons, or videos.

Familiar Applications of AR

While the concepts of AR have been around for decades, the technology has been applied to so many different types of media that it is difficult to point to any one application as the embodiment of augmented reality. For example, American football fans are used to seeing the virtual first down line that appears on the field during televised games—they expect to see it. Movie directors have portrayed augmented reality in movies such as Minority Report (2002) or Avatar (2009) to show how immersive technology might be used. Some of the most visually exciting examples of AR entertainment are augmented projections or projection mapping in which computer imagery is projected onto physical objects, such as a building facade, to create an augmented but realistic-looking new object that morphs continuously before the viewer's eyes.

The military uses a type of AR in the head-up display (HUD) that appears on the cockpit window (or view port) of a jet fighter. These displays make it easier for pilots to access critical information such as air speed or angle of attack without having to break their focus from the reality in front of them. Similarly, automobile manufacturers are also experimenting with HUD windshield displays to improve driver safety.

Seeing GIS Data Using AR

Richie Carmichael of the Esri Applications Prototype Lab has been experimenting with using AR to visualize GIS content. He thinks of AR as simply adding to reality in a way that enhances or improves upon what you see displayed in a web or desktop application.

"GIS data is already a source of spatially enabled information that can be fed directly to your mobile device," explained Carmichael. "Your phone knows where it is, the direction the camera is facing, and the inclination of the device in your hand. This information, combined with GIS data, not only adds to the end user's experience but also makes GIS accessible in a new and interesting way."

As AR becomes more prevalent in phone, tablet, and computer applications, more developers will begin integrating GIS services and content to serve very real and practical purposes. Mansour Raad, a senior software architect at Esri, envisions AR apps that will channel GIS content to professional end users using mobile devices to help them make decisions on-site using the most up-to-date and reliable information available.

"Imagine that you could point the camera on your phone at the ground and see the location and orientation of water pipes and electric cables buried under the ground because your AR app is cross-referencing the GIS system with your location and giving you a kind of x-ray vision so that you can visualize the infrastructure that is underneath you," explained Raad.

"Not only would you be able to see hidden objects, you'd have access to their attributes, you'd be able to pull up engineering diagrams, and even tap into real-time sensor networks to view water pressure or amperage." Raad suggested that you might use your phone's camera to take a photo of the augmented display and e-mail it to your project team or sync it with the GIS-enabled work order system so that it can be accessed by the project manager and the work crew. "What makes the combination of AR and GIS really exciting is not just the visualization of GIS content, which is very cool, but the integration of that content with the rest of the enterprise. That's what GIS is all about."

Getting Started with AR Development

AR app development is surprisingly straight-forward. AR service providers, such as juniao or Layar, provide both an AR browser and the development resources needed to simplify deployment for iPhone and Android.

To use the junaio app, you start by registering a "channel" with junaio. The channel provides a callback to your server, where the processing takes place. For example, if you use the app to find geographic information near your current position, junaio brokers the calls to your server, which processes the request, then passes that information back to your device. For a GIS developer, this means accessing the REST endpoints of ArcGIS Server services as the link to dynamic GIS content and tools or preprocessed geographic information such as 3D terrains, videos, or map layer stacks.

"Augmented reality can be difficult to explain to people," said Thomas Emge, also from the Esri Prototype Lab. "It's often easier to show someone an AR application first, let them experience it, and then start the conversation about what you want to do with it. At first, there's the cool factor that's fun to do and easily impresses most people. Next, you have to start thinking about design and usability."

Emge suggests that developers begin by considering the end user's experience. "There are different ways to apply AR to your app," said Emge. "In one case, you are overlaying additional information in your camera's viewfinder. Here the device's location and orientation play an important role. In a different case, the information seen by the camera is sent to a server and processed, then sent back to you. Here, pattern recognition technology is being employed, and the response can be as simple as playing a video or as complex as showing the results of spatial analysis."

Raad sees great potential for using the pattern recognition capability of AR. "The image can be anything—a photo, a map, a barcode, or even the stars in the sky. Using the camera, the AR app recognizes the pattern and does something that augments or mixes additional content with what you're seeing."

For example, Raad has created a sample app that uses pattern recognition to let users point an iPhone camera at a photo of a map, which triggers the loading of a video that shows a rotating globe displaying the same information. Raad used ArcGIS to compose the 2D map data and drape it on a globe. Then he animated the globe and created the video, which is hosted on a server. Next, he registered a channel with junaio that links his video with the junaio AR application. Anyone with an iPhone can download the junaio app, search for the channel, point the iPhone camera at a photo of the map, and view the rotating globe.

Carmichael also used pattern recognition. He linked a photo on a piece of paper to a related image of a map layer stack in his desktop application. However, in this case, the layer stack appears to dynamically hover above the photo—even if the photo is moved around.

Any developers using an AR toolkit and one of the ArcGIS APIs for Silverlight, Flex, or JavaScript can start building their own AR-GIS apps. The original ARToolkit was developed by Hirokazu Kato of Nara Institute of Science and Technology in 1999 and was released by the University of Washington Human Interface Technology Lab (HITLab). Since 1999, numerous ARToolkit spinoffs have been created. In 2010, René Schulte ported the ARToolkit to Microsoft's Silverlight platform, which was released as the SLARToolkit.

Emge cautions developers to recognize the limitations of mobile devices. "You may find that the limited accuracy of the GPS won't allow you to place labels and billboards as precisely as you'd like or that AR calculated measurements are only approximations," he said. "This may frustrate end users with GIS and CAD experience because they expect a high degree of accuracy and precision." Both Emge and Carmichael recommend that AR apps be used to query trusted GIS services for precise measurements and reliable attribute information instead of relying on a phone's capabilities.

Regardless of the popularity of AR or the current limitations of some mobile devices, AR and GIS would seem to be a perfect match, not only for consumers but for professionals who rely on geographic information to make real-time decisions.

Learn More

Raad maintains a very active personal blog at thunderheadxpler.blogspot.com. Carmichael and Emge are regular contributors to the Applications Prototype Lab blog. Both blogs contain posts on AR topics.

Sources for AR information
junaio: www.junaio.com
Layar: www.layar.com
HITLab: www.hitl.washington.edu/artoolkit
Wikipedia: en.wikipedia.org/wiki/Augmented_reality