Is data preparation the most time-consuming part of your work? Are you tired of maintaining and re-running Python scripts to clean or update your feature layers and tables? ArcGIS Data Pipelines is a data integration application in ArcGIS Online that provides a drag-and-drop interface for you to connect to your data, build and transform it, then use that tailored data to create or update feature layers in ArcGIS Online. This blog outlines the new features and enhancements available in the June 2024 update of Data Pipelines, and shares how they can help you in streamlining your data integration workflows.
What’s New?
The June update of Data Pipelines includes support for additional input file formats, enhanced tools, new scheduling options, the ability to disconnect in the editor, and more. Check out the video below for a quick tour of the new features or keep reading for more information.
New file formats
There are two new file formats supported as input to data pipeline workflows:
- GeoParquet–You can now use GeoParquet datasets from public URLs, Amazon S3 buckets, or Microsoft Azure storage containers.
- File geodatabase–Feature classes and tables from file geodatabases are now supported. You can use file geodatabases from ArcGIS Online content, Amazon S3 buckets, or Microsoft Azure storage containers. Datasets other than feature classes and tables are not supported (that means you can’t use any raster datasets, relationship classes, etc.).
Check out the dataset configuration documentation to learn about all of the supported sources and file formats.
Tool enhancements
The following data engineering tools have been enhanced:
- Filter by attribute now supports expressions of “in the last” or “not in the last”. For example, you could use this to look for the most recent records that occurred in the last 7 days, or find historical records that did not occur in the last 5 years. You can use this expression for date (contains a date and time) and date only type fields.
- Calculate field now supports Arcade geometry functions. This means you can create, manipulate, build, and enhance geometry information in your datasets. For example, you could use Arcade to create new geometries, buffer or clip existing geometries, or calculate geometry attributes such as angles, lengths or widths.
Want to learn more about the other tools that are included with Data Pipelines? Check out the data processing documentation.
Scheduling enhancements
There are two new scheduling options that provide more control over your scheduled data pipeline tasks:
- Schedule more frequent updates to your data using the new Repeat type option of Minute. You can now schedule updates to run at 15, 30, or 45 minute intervals.
- Set an automatic timeout for task runs using the new Maximum run duration parameter. For example, if you use a data source that can be slow to respond, set a maximum run duration to cancel the run when the source doesn’t respond after the allotted duration.
To learn how to automate your data preparation workflows, see the help topic Schedule a data pipeline task.
Disconnect experience
The new option to disconnect the editor enables you more control over your connection. When you’re finished working in the editor, choose to Disconnect all to stop incurring credit costs. Disconnect all will disconnect all open editors (i.e. browser tabs with connected data pipeline editors). After disconnecting, you’ll have the option to save any edits, leave without saving, or reconnect and continue your work.
Since data pipelines consumes credits based on the amount of time you are connected, this new experience offers you more control over credit management.
For more details on how and when credits are consumed while using Data Pipelines, refer to the Compute resources documentation. To learn more about credit consumption rates, refer to the Understand credits documentation.
More information
More details on new enhancements and improvements with the February update can be found in the What’s new documentation. If you’re new to Data Pipelines or just want to learn more, here are some additional resources to get started:
- Watch this webinar recording to learn more about what Data Pipelines can do, plus some demonstrations of how to use the application.
- Check out some other ArcGIS Blogs about Data Pipelines.
- Follow this tutorial to get started building your own data integration workflow.
- If you have questions, check out the FAQ documentation or post in the Data Pipelines Community where the Data Pipelines team will respond.
Read the What’s New in ArcGIS Online (June 2024) blog for news on other exciting updates.
We want to hear from you! There is more to come for ArcGIS Data Pipelines in the future, and we value your opinion on what new features and enhancements we can add to help you with your data preparation workflows. Share your ideas or ask us a question in the Data Pipelines Community.
Article Discussion: