The intersection of ArcGIS, geospatial data, and video games is becoming more blurred as more video games are based on real-life examples. Re-creating an accurate, realistic virtual world in a video game that provides unique game experiences and interactions is a growing field. Even for entirely fictional landscapes, real-world geospatial data incorporated into a game can make the setting realistic.
For example, the action-adventure video game Assassin’s Creed Syndicate is set in 1869 London. Game developers at Ubisoft worked tirelessly to create an accurate, one-to-one scale representation of London. Many players have reported that when they visited London for the first time in real life, they had already learned how to navigate the city from their gaming experiences. Although incorporating real-world geospatial data into video games is promising, many challenges still exist.
Project Background
At the Rochester Institute of Technology (RIT), the authors of this article worked as a team that explored using ArcGIS Desktop, ArcGIS Online, ArcGIS CityEngine, and ArcGIS Maps SDK for Unity to create a serious game about natural disasters. The purpose of the game was to educate people and influence policies that can impact lives during disasters. Previously, RIT created Project Lily Pad and Project EOC, both focused on events related to Hurricane Harvey in 2017, as part of a National Science Foundation (NSF) Research Experiences for Undergraduate Students (REU) project. [Read more about these games in “Serious Game Play,” which appeared in the August 2020 issue of ArcWatch.]
With new advancements in Esri technology that are specifically focused on game development, the team decided to use Esri technology to make a serious game and simulation called Project Lake Ontario. It is related to the flooding of Lake Ontario, which severely impacted several coastal residential neighborhoods in 2017 and 2019. A goal of Project Lake Ontario was to educate the coastal residents about Plan 2014, a water level regulation policy.
Specifically, the team wanted to use Esri technology to create a realistic simulation of how water levels would look under various scenarios. From a GIS education perspective, the team also wanted to use Project Lake Ontario to teach game development and design to students as well as Esri technology and broader GIS concepts. This would be accomplished during a 10-week undergraduate student summer research and development experience.
Video Games That Use GIS and Geospatial Data
The team started building Project Lake Ontario using ArcGIS Pro. Team members downloaded lidar data and imagery from the Monroe County, New York, website for the project area to establish accurate ground terrain for the game. One of the first challenges was that the lidar detected the trees and water in addition to the terrain and buildings.
After examining the lidar data by using ArcGIS Pro, the team found it had only the ground classified so that information was extracted for the height map of the land. To get the buildings, the trees needed to be removed. To solve this problem, the lidar points were filtered by selecting only single return data points. This was done to isolate the buildings because trees tend to have multiple returned values.
With the mostly treeless lidar, it was much easier to begin to extrude and construct buildings in ArcGIS Pro. Although some of the houses that appeared in the lidar imagery did not match the imagery of the surrounding area due to differences in data collection times, the lidar data was chosen for determining which houses to display and how they would appear.
After constructing land and buildings in ArcGIS Pro as accurately as possible, the next challenge was importing that data into Unity, an industry-leading game development environment that would be used to create Project Lake Ontario. The team decided to evaluate two Esri products: the latest beta release of ArcGIS Maps SDK for Unity and then ArcGIS CityEngine.
Using ArcGIS Maps SDK for Unity
The documentation for the latest beta release of ArcGIS Maps SDK for Unity seemed to show it as an out-of-the-box tool that could easily incorporate geospatial data directly into Unity. Objects and layers were first created in ArcGIS Pro, and the layers were then exported to ArcGIS Online as scene layers. The ArcGIS Online scene layer URLs were fed into ArcGIS Maps SDK for Unity for direct incorporation into Unity. The full process involved a lot of smaller steps such as selecting settings and manually exporting/importing each time data was brought into Unity. After overcoming some initial challenges and gaining more experience with Esri tools, the landscape for the project area was imported into Unity using ArcGIS Maps SDK for Unity.
With this initial success came the next set of challenges that had to be dealt with to meet the broader goals of Project Lake Ontario. Creating the game required two things. First, features in the geospatial layers needed to become GameObjects, which are the base class for all entities in Unity scenes and the core building blocks of video games.
GameObjects need to respond to player interactions. In video games that have buildings, a player can’t (usually) walk through a building. Colliders prevent this from happening. [Colliders define the shape of a GameObject for the purposes of physical collisions or interactions in the game.] For custom game interactions, GameObjects can have C# scripts attached to them.
Second, the geospatial layers must exist in Editor mode so they can be seen when building the game versus appearing in Play mode when the game is being run to test it. The team quickly realized that the ArcGIS Online layers incorporated into the game via ArcGIS Maps SDK for Unity appeared only in Play mode and were visible only in the absence of colliders.
Without geographic features from ArcGIS Online available in Editor mode, attaching scripts and colliders was much more difficult. It required that the entire workflow from its start in ArcGIS Pro to publishing layers to ArcGIS Online be redone anytime the underlying data was modified (for example, to move a building).
The team noticed another serious issue. Specifically, ArcGIS SDK for Unity had its own camera rendering in the back end, which caused conflicts when multiple cameras were implemented in the game scene. Additionally, the ArcGIS camera would exhibit strange movement if another camera was used or moved.
After some initial success and challenges with building Project Lake Ontario using the latest beta release of ArcGIS Maps SDK for Unity, the team decided to explore a different Esri technology: CityEngine.
Using ArcGIS CityEngine
With ArcGIS CityEngine, the team used the following general workflow. Layers created in ArcGIS Pro were imported into ArcGIS CityEngine. Those layers were exported from ArcGIS CityEngine as Wavefront OBJ files and opened in Unity. This workflow allowed models to be textured in ArcGIS CityEngine and modified easily without continually republishing to ArcGIS Online. In ArcGIS CityEngine, multipatch feature layers could be divided into individual models for items such as houses. These models could then be imported into Unity and modified in the Unity Editor mode.
Once the ArcGIS Pro to ArcGIS CityEngine to Unity workflow proved successful, more advanced functionality was added. For example, interactively changing water levels in the game allowed players to add sandbags to simulate flood control. Real-world objects, such as gas tanks, cars, and trees, were added to enhance the game’s realism.
Reflections and Future Outlook
Creating Project Lake Ontario proved to be a challenging yet rewarding experience. Despite lacking experience with Esri technology, the team created a viable video game that incorporated advanced game design functionality using real-world geospatial data and Esri technology in the short span of 10 weeks.
In its current form, the team found that ArcGIS Maps SDK for Unity is a great tool for exploring geospatial layers and rapid prototyping in a game engine tool like Unity. Given that ArcGIS Maps SDK for Unity was still in beta, the team expects many of the challenges faced in developing Project Lake Ontario will soon be overcome.
According to Rex Hansen, the ArcGIS Maps SDK product manager, “ArcGIS Maps SDK is designed to provide developers with direct integration of live and local ArcGIS data within a game engine like Unity. As the Maps SDK moves toward its first production release in 2022, we’ll continue to improve integration with game engines to support use in Editor mode; provide a geoplacement framework for more intuitive placement of cameras and game objects; enable ArcGIS data for use in ray casting or as a collision mesh; and, of course, continue to add support for additional ArcGIS data sources.” [A ray casting algorithm uses lines (rays) in virtual space to represent the path of light from an object to an observer.]
Project Lake Ontario also represented an excellent opportunity to build unique game experiences that can be used to educate and inform the public on pressing challenges facing society. For example, the goals of the Project Lake Ontario game are related to understanding water level regulation policy.
In addition, there are great opportunities to train and educate the next generation of game development students in the use of real-world geospatial datasets and Esri technology. Ideally, there will be greater integration of game development and the geospatial field as shown in the creation of Project Lake Ontario.
For more information, contact
Brian Tomaszewski, PhD
Center for Geographic Information Science & Technology
School of Interactive Games and Media
Rochester Institute of Technology
bmtski@rit.edu
Resources
- ArcGIS Maps SDK for Unity
- More information about Project Lake Ontario including source code and playable executable file
- Overview video about combining Esri technology and video games
About the authors
Brian Tomaszewski
Carson Lampila
Noah Richards
Liam Andres
Chase Call
