ArcGIS Online allows users to store, manage, and share spatial data as files, web layers, or connections to other data sources. This data, in all its forms, becomes the foundation for visualization, analysis, and editing workflows.
To maximize data’s potential in ArcGIS Online, it is essential to integrate data effectively and keep the information current. Here are seven ways to get data into ArcGIS Online.
1. Directly from the Source
For data that doesn’t get updated very frequently and is in a ready-to-use state for mapping, analysis, or reporting, it is easy to add this data to ArcGIS Online directly from its source. Users can add data from a local drive or network or by uploading the data from various cloud storage solutions such as Microsoft OneDrive, Google Drive, and Dropbox.
ArcGIS Online supports a long list of file and data formats. Users can add zipped shapefiles, zipped file geodatabases, and CSV files containing location information directly to ArcGIS Online by going to the My content tab and either selecting New item or dragging the files into the window. From there, users can create web layers out of their files, transforming the data into a format that is ready to use in web maps, web apps, and analysis tools.
Adding data directly to ArcGIS Online is a great option for when a user’s goal is to ingest a static, one-time copy of the data. If the source data changes regularly, is frequently edited, or is growing, then it might be challenging to keep the data current. Users would need to either overwrite relevant items with the latest copy of the data or write a Python script in a notebook to automate the task.
2. ArcGIS Data Pipelines
If the source data is edited outside ArcGIS Online or needs to be cleaned, formatted, and transformed prior to using it in a web map or analysis workflow, it’s best to use ArcGIS Data Pipelines. This is a native data integration capability in ArcGIS Online that’s available out of the box to all ArcGIS Online organizations.
Data Pipelines offers a low-code, drag-and-drop visual authoring experience that allows users to design data pipelines that ingest, prepare, and engineer data so it is ready to use for mapping, analysis, and reporting. Data Pipelines can establish secure connections to a variety of cloud-based data stores, including Snowflake, Google BigQuery, Amazon S3, and Microsoft Azure Blob Storage. It also supports other data sources such as public URLs and existing feature layers.
Data Pipelines goes beyond merely ingesting data; it can be used to clean, combine, and format datasets. The transformed data is then written out to a feature layer, making it ready to use across ArcGIS. Data Pipelines also includes built-in scheduling functionality that enables users to run their data pipelines on a recurring basis. Thus, as the source datasets change and evolve, Data Pipelines can be used to keep the related layers in ArcGIS Online up-to-date.
3. ArcGIS Velocity
What if the data that needs to be integrated with ArcGIS Online needs to be kept current while also supporting real-time data, such as feeds from Internet of Things (IoT) platforms, message brokers, or third-party APIs? If having real-time situational awareness is the goal, then using ArcGIS Velocity is a good choice.
This real-time and big data analytics extension for ArcGIS Online allows organizations to monitor and gain precise location-based insight on live assets such as cars, trucks, airplanes, boats, and environmental sensors. Velocity does this by ingesting data from IoT platforms and other sensors via the cloud.
In addition, Velocity includes powerful analytics tools, such as dynamic geofencing and visual analysis models. It also supports alerts that can communicate actionable information when time is of the essence.
4. Sharing from ArcGIS Pro
For organizations that begin their work in Esri’s powerful, desktop-based GIS software ArcGIS Pro, rest assured that it is designed to seamlessly integrate with the web-based components of ArcGIS, including ArcGIS Online.
Many users start their workflows in ArcGIS Pro, where data creation, editing, and analysis take center stage. ArcGIS Pro offers the flexibility to connect to many different types of data, including files stored on a computer or network drive and data that resides in enterprise geodatabases, cloud data warehouses, and other locations. Users can connect to various data sources, leverage their data in maps and scenes, and perform advanced analytics and visualizations in 2D and 3D.
Once the data is curated, it can be shared to ArcGIS Online as web layers, web maps, web scenes, and more, depending on the use case. This fosters collaboration both internally and externally, whether in the office or around the world. Once the data is published to an ArcGIS Online organization, it becomes available to edit, query, visualize, analyze, and collaborate with.
If the source data changes and updates need to be made to the data in ArcGIS Online, users can either manually overwrite the web layers or automate the process using ArcGIS Python libraries in a notebook or script.
5. The ArcGIS Data Interoperability Extension for ArcGIS Pro
The ArcGIS Data Interoperability extension for ArcGIS Pro broadens the data integration and transformation capabilities of ArcGIS Pro. Like Data Pipelines and Velocity, Data Interoperability offers an intuitive, no-code visual diagramming interface that can be used to engineer data.
The Data Interoperability extension is an extremely powerful extract, transform, and load (ETL) solution. It provides robust capabilities for connecting to a vast range of supported inputs and file types. It excels at handling complex transformations and allows data to be written back to its source—even beyond ArcGIS.
If changes to source data need to be reflected in ArcGIS Online, organizations do need to have ArcGIS Server licensed with the ArcGIS Data Interoperability extension. This allows users to schedule the ETL jobs to run on a repeated basis via a notebook.
6. ArcGIS Python Libraries
When data integration needs are complex or must be automated and no out-of-the-box or commercial off-the-shelf solutions fit the bill, developers and GIS practitioners can use the ArcGIS Python libraries—Python packages that include ArcGIS API for Python and ArcPy.
These are powerful, modern Pythonic libraries that provide a consistent experience across ArcGIS for scripting and automating, and they can be used to integrate data.
With ArcGIS Python libraries, users can author Python scripts to connect to an extensive array of data sources and file formats, perform advanced data manipulation and analysis, and write the results to web layers in ArcGIS Online. The Python API has functions and methods for adding and publishing files, packages, and Spatially Enabled DataFrames as web layers in ArcGIS Online. ArcPy can share, overwrite, and replace web layers using a combination of functions and geoprocessing tools.
ArcGIS Online includes ArcGIS Notebooks, providing an integrated platform for writing, running, and scheduling automation scripts using the Python API and ArcPy. Python scripts that leverage these libraries can also be written and run externally to ArcGIS Online.
7. Distributed Collaboration
Organizations that use ArcGIS Online and ArcGIS Enterprise together should consider employing distributed collaboration to share data from one system with another. This approach requires no scripting expertise and makes it easy to integrate data from ArcGIS Enterprise with ArcGIS Online. Data can be synced in both directions, and once content is shared, updates are sent automatically, keeping the information in sync in both systems.
With distributed collaboration, organizations can seamlessly share maps, apps, web layers, and other content between the two systems. In the case of web feature layers, collaborations can be set up to support the two-way sharing of feature layer edits. This allows a feature layer published to ArcGIS Enterprise to be edited in ArcGIS Online—with the edits then written back to the source, whether that’s a hosted feature layer or a feature layer that is referencing data in an enterprise geodatabase.
Unlock the Full Potential of Data
Once data is integrated in ArcGIS Online, it can be used to perform spatial analyses, create visually stunning maps, and solve problems. The right approach to integrating data depends on each project’s unique requirements and each organization’s goals.
To learn more and get a list of additional resources related to data integration, see the accompanying Esri Community blog post, “Seven Ways to Integrate Data with ArcGIS Online: Additional Resources.” To ask questions or share additional ideas, visit Esri Community.
