NJ Utility on Forefront with New Mixed Reality Application

A midsize utility in New Jersey has made the leap to adopt a cutting-edge mixed reality (MR) holographic application while blazing a path for other industries and businesses to follow.

Toms River Municipal Utilities Authority (TRMUA) may just be the first utility in the world to use mixed reality headsets to guide fieldworkers in locating underground utilities. This technological leap brings utilities and other companies with underground assets closer to realizing an age-old wish: to see through dirt. The mixed reality solution is helping field technicians in Toms River close service tickets more quickly and avoid costly repairs.

“This is not science fiction anymore,” says Len Bundra, the IT/GIS director for the Toms River agency, and the person who saw the potential in a mixed reality application.

At its most basic, the Toms River system takes three well-established technologies and combines them into one brand-new application. All three are commercially available—Esri’s geographical information system (GIS), which stores location and attribute information on TRMUA’s underground assets; Microsoft’s Azure cloud-computing service; and Microsoft’s HoloLens. The three were integrated and customized by Meemim Inc., a Toronto-based company founded by Alec Pestov, who previously managed several startups.

Early results are sparking widespread interest, driven in part by the growing excitement and investment in mixed reality applications. As more agencies and organizations see the potential savings and increase in efficiencies and job safety, the use of hands-free mixed reality headsets may set the new standard for avoiding utility breaks while digging, and preventing errors when marking line locations on asphalt, concrete, or lawns.

Just a couple months into the pilot program, it’s hard to quantify savings. But in addition to minimizing lost time, anything that lowers the chances of a utility main break is likely saving tens of thousands of dollars.

The first steps toward this technological leap came in 2005, when Bundra oversaw the installation of TRMUA’s GIS, which included precise (subdecimeter) data on the locations and sizes of more than 400 miles of wastewater pipes. With that rich database and other layers embedded in GIS, Bundra began to wonder how to create even more efficiencies in the field. He eventually found in Meemim a partner that wanted to explore the idea.

The partnership has turned out to be very useful, so much so that the dozens of utility representatives who have seen the TRMUA application in action have been impressed by its capabilities and ease of use and want to know more.

Mixed Reality Technology across Industries

The use of mixed reality or augmented reality (AR) devices in work environments continues to grow.

Goggles, smart glasses, headsets, and helmets that project information, instructions, or images on the wearer’s field of vision are being used by manufacturing and retail companies that build equipment, repair machinery, or want to show customers what kitchens or homes will look like when completed. And the list of the technologies’ applications keeps getting longer.

In December 2016, the Wall Street Journal reported that companies were performing quality checks 20 percent faster and speeding up production by 25 percent with versions of the smart glasses. It also noted that Forrester Research, Inc., predicted that the number of US workers using smart glasses will grow from the current 400,000 to nearly 14.5 million in 2025. The research company also expects spending on smart glasses among large companies to increase from $6 million to $3.6 billion in the same period.

Those numbers do not surprise Cindy Elliott, head of commercial industry marketing for Esri. “The combination of GIS and augmented reality can address a lot of interesting business cases across industries,” Elliott says. “The technology is certainly powerful in outdoor settings like Toms River, but GIS and AR can also create efficiencies in warehousing, real estate, retail planning, and manufacturing.” (See the sidebar for one possible scenario.)

In essence, the use of mixed reality technology and GIS affords companies a new level of operational intelligence, which drives efficiency, increases safety, and can even inspire new service offerings. (Learn more about operational intelligence.)

As word has spread about the Toms River implementation, even utilities much larger than TRMUA—which has 435 miles of underground utility lines, 18 pump stations, and 47,000 ratepayers—are paying attention, and new partnerships are forming around the business potential of mixed reality.

While many utilities traditionally have kept a tight lid on data related to gas lines, electrical wires, and fiber-optic connections, Bundra sees that changing, thanks to the HoloLens project. “This technology is like a caveman fire for people to huddle around ” Bundra says. “When you put on the HoloLens, the more utility lines you see underneath you, the more utilitarian the product becomes. Now we’re suddenly huddling around the fire.”

A Huge Potential for Savings

A simple scenario that was repeated several times during the first months of the Toms River project helps illustrate the increased efficiencies of GIS when coupled with the holographic headset.

The holographic headset includes audio, tiny cameras, movement sensors, and a system for confirming the wearer’s exact location relative to the underground pipes. While wearing the HoloLens transparent visor, utility workers see a mixed reality—including the real world in front of them as well as a holograph of the lines of wastewater pipes underground. The pipes are color coded and projected to scale. With slight movements or audio commands, the wearers can summon screens showing relevant information.

This video offers a brief tour of the MR/AR experience. The relationships among the pipes are made clearer and the ability to gauge depth is much richer with the headset on, but the video conveys the basic functionality.

While wearing the headset, workers operate hands free, precluding trips back to the truck to get their bearings using two-dimensional maps of utility lines, or a tablet computer. And even more importantly, the technology allows fieldworkers to connect with remote colleagues—often an engineer in the office—who can see exactly what the fieldworkers see. This is accomplished through a simple Skype connection. So if the employee in the field—whether marking lines with paint or digging to fix or replace a line—is not sure how to proceed, an engineer or manager back at the office can highlight or circle certain areas where extra caution should be taken, and those marks will appear in real time on the fieldworker’s visor.

For instance, if the manager notices a grouping of utility lines, then the worker may be directed to stop using heavy equipment and start digging by hand until the issue is resolved. In the past, fieldworkers attempted to resolve these incidents over the phone or by summoning the manager from the office to the field—a slow and pricey process.

Having accurate GIS data and engineering help available at nearly all times can avert costly line breaks. A break in any utility line means that work comes to a screeching halt until repairs are made. Many of those problems occur because the aboveground markings are off. “So now a simple two-hour markup just turned into a $23,000 dig up and repair,” Bundra says.

And if the issue happens after business hours, the new application means that fewer employees need to drive to the site and risk exposure to open holes and heavy machinery at night.

But even the avoidance of small issues can add up to big benefits. For instance, a TRMUA worker on a routine assignment discovered a manhole cover that had been paved over. Accordingly, the next version of Meemim software will allow the wearer of the headset to mark the location electronically and then record an audio message to note the data discrepancies and any needed repairs.

A Company’s Bridge to the Future

TRMUA’s data is precise. Wastewater pipes are plotted within a decimeter of their position. Indeed, the utility’s GIS helped emergency managers know exactly where wastewater lines were located after Superstorm Sandy blew away or covered up many of the landmarks in 2012.

For companies looking to bring mixed reality technology to their operations, indoors or outside, a richly detailed GIS database makes most applications possible. It can store information on the location of fixed assets, track mobile assets like trucks in real time, and monitor the conditions of anything equipped with a sensor. “Without a GIS, there’s nothing,” Bundra says.

That database of information allowed Bundra and Toms River to pioneer a different approach to operational intelligence, experimenting with MR technology that many people have heard of but few have actually used.

For his part, Bundra relishes the pioneering spirit of the project, as well as its early returns.

“I’m looking at this like it’s 1979 and I’m holding the first Walkman.”