Out in the Field, New Jersey Utility Uses Mixed Reality to View Simulated GIS Features
In March 2014, articles about virtual and augmented reality being the next big thing flooded the Internet. Len Bundra, the IT/GIS director for the Toms River Municipal Utilities Authority (TRMUA) in Toms River, New Jersey, foresaw a day when this technology would allow field-workers to “see through the ground,” as he put it, and view the utilities beneath their feet without having to rely on vehicle-mounted rugged laptops, which display static 2D GIS features. He wrote a post on LinkedIn outlining his idea to use virtual or augmented reality goggles to project features from ArcGIS onto the real world.
While Bundra’s post was viewed several hundred times, it sat on the social media site for almost three years without any responses. But then, in October 2016, he received a message from Alec Pestov, a developer from Toronto, Canada, who shared Bundra’s vision. Pestov, the founder and CEO of Esri startup partner Meemim, Inc., proposed that, together, they build something entirely new—a mixed reality app that would allow utilities (and eventually other organizations) to collaborate with shared content and data in novel ways.
Mixed reality, often described as a combination of virtual reality (VR) and augmented reality (AR), enables users to see virtual objects that are not visible from a real-world perspective. This is akin to a VR experience, though the real world doesn’t get occluded. With mixed reality, users can view and retrieve information about both the visible, real-world objects surrounding them and the virtual objects being projected by headsets. In Bundra’s opinion, mixed reality was the perfect environment for allowing his fieldworkers to see concealed utilities while remaining situationally aware of their actual surroundings.
Bundra and Pestov immediately set out to turn this idea into reality.
Getting Support for Simulation
Before the two innovators could get started on the project, Bundra had to get backing from TRMUA’s executive director and Board of Commissioners. Pestov proposed that Meemim fund the up-front programming costs so long as staff at TRMUA made themselves available for technical consultations and field testing during the app’s development phase. TRMUA would be able to defer additional expenditures until Meemim had completed a demonstrable pilot project.
Fortunately, TRMUA’s board had a history of supporting the early adoption of technology that would allow the utility’s employees to be more efficient and, in turn, save its ratepayers money. So after Meemim created a test case and presented the technology to the board, TRMUA received approval to go forward with the project.
To get things going, Pestov’s team chose the Microsoft HoloLens as the smart glasses TRMUA would use. The head-mounted holographic computer was the only viable device that met all TRMUA’s functional requirements. Plus, it would be supported by the largest network of hardware and software technicians and developers at Microsoft. Esri’s longstanding partnership with Microsoft factored in, too, since TRMUA has been an enterprise client of both companies for many years.
Meemim then had to develop a way to digest TRMUA’s ArcGIS features—surveyed coordinates, data, and projections. Although Bundra had 25 years of experience editing, creating, and exporting GIS features to and from conventional formats, exporting holographic projections to mixed-reality hardware was something completely new. Since several of Meemim’s programmers were already experienced in working with Unity, a development platform used to create 3D worlds for many online games, the startup chose that as the simulation engine. Thus, GIS programmers met Unity programmers, and it worked.
Turning the Virtual into Reality
Meemim began developing the technology in December 2016. Within three months, the company had created its app, called vGIS, which could process ArcGIS feature services, upload the output to Meemim’s Microsoft Azure cloud service, and communicate directly with HoloLens to create the holographic projection of underground utility features from the perspective of the headset wearer’s location and orientation. Both Microsoft and Esri independently confirmed that Meemim and TRMUA were innovating, since they were not aware of any other project in the world that was doing what the two organizations were trying to do.
By May 2017, it was time to test the technology out in the field. Bundra’s main concern was whether the Microsoft HoloLens could accurately project and display TRMUA’s underground GIS features—which were field surveyed and postprocessed to subdecimeter accuracy—so the wearer would see them aligned with the real world.
“We are dealing with five or six utility lines underneath a street,” said Bundra. “We have to locate our utility lines so that when we dig up a street, we are confidently aware of all other utilities around us. Given a more dynamic situation—a car hits a telephone pole, the pole collapses onto the street—seven separate agencies are on scene, each viewing [its] own GIS utilities on separate devices. What I am trying to do is provide all field personnel with one common interface to see all underground utilities beneath their feet—[so] one shared vision via a shared interface.”
On Tuesday, May 2, Bundra set out to take one small step for the future of GIS, going where no other GIS specialist had ever gone. On a quiet street in Toms River, he stood on the sidewalk, adjusted his HoloLens headset, and launched Meemim’s vGIS app.
Using Meemim’s patented geocalibration process, Bundra anchored himself to a nearby visible GIS feature (a sewer manhole) and then provided the HoloLens with a precise headset orientation via a simple two-step process. It took less than a minute for the HoloLens to perform the calibration, and afterward, Bundra could see all the water, gas, electric, and sanitary and storm water sewer utility features begin to project themselves around him.
Then, using a voice control system that Meemim built into its vGIS interface, Bundra was able to give the HoloLens standard mapping commands, such as “zoom in,” “pan left,” “select,” and “close.” This allowed Bundra—and would enable other fieldworkers—to see and retrieve information about the utilities around him, hands-free. Meemim had color-coded the utility features by utility layer, according to common conventions; electric utilities were red, water utilities were blue, sewer and drain lines were green, and so on. The startup ensured that the holograms displayed pipe sizes according to their actual diameters, so that a pipe with a two-inch diameter would appear smaller than a pipe with an eight-inch diameter. Additionally, each utility line was represented by two lines—a solid one that showed the utility line at its depth and a dashed one that showed where the utility line would be marked out (spray painted) on the road or sidewalk.
Whenever Bundra moved his eyes, a small white cursor would follow. If he placed his gaze on a GIS feature, he could say, “select,” and an attribute information window would appear. If he said, “close,” the window would disappear.
Bundra walked along a water main. He made a turn at the first water service line and ended his mixed-reality tour at a water meter beneath his feet. Everything coincided both visually and virtually to within inches of each other.
For the first field test, Bundra could not have expected better results. What he saw only reinforced his initial vision: that mixed reality and GIS technology would soon converge to provide tangible benefits for all utility field operations.
Game-Changing Impact
TRMUA now uses the app in the field. And anyone who sees it quickly realizes that this technology is most powerful when organizations share their data so that everyone can see what is beneath their feet.
Given that TRMUA serves 100,000 citizens across multiple jurisdictions and coordinates its operational activities with seven public and private entities, the app is now bringing the utility closer together with neighboring political entities. So far, the app has helped initiate long-overdue conversations regarding data sharing and GIS mapping standards.
Ideally, mixed reality can bring the same game-changing impact to the GIS community that it has initiated in the media, entertainment, and gaming worlds.
For more information about mixed reality, email Bundra at lbundra@tomsrivermua.org or Pestov at alec.pestov@meemim.com.
