We’re excited to introduce the new Run Data Pipeline action in ArcGIS Connectors for Power Automate, which can significantly enhance your data integration workflows.
ArcGIS Data Pipelines is a data integration application in ArcGIS Online. It allows you to connect to datasets from various sources, perform data engineering workflows, and then write your cleaned and prepared data out to a hosted feature layer in ArcGIS Online. Additionally, you can automate data pipelines to keep your feature layers up to date using task scheduling, running via the ArcGIS API for Python, and now with Microsoft Power Automate.
Keep reading to learn how you can leverage the new Run Data Pipeline action in ArcGIS Connectors for Power Automate to further strengthen your data integration workflows.
What can I do with Data Pipelines and Power Automate?
Power Automate is a workflow automation tool that connects to over 370 data sources, offers a wide variety of triggers that can be used to kick off a workflow, and has alerting and notification capabilities. Power Automate is widely used for orchestrating large workflows, including data preparation tasks, so naturally many of you have requested the ability to run your data pipelines from Power Automate. With ArcGIS Connectors for Power Automate, which enable spatial workflow automation and integrate geographic data with Microsoft Power Automate, it is now possible. We have introduced the new Run Data Pipeline action in ArcGIS Connectors for Power Automate, which will help you automate your spatial data prep workflows in Power Automate.
Watch the quick video below for a demonstration showing how ArcGIS Data Pipelines can be used with Power Automate or keep reading for more information.
Automate running a data pipeline when the source data changes
You can schedule data pipeline tasks in ArcGIS Online to run based on a defined time interval, which is ideal for datasets that update regularly. But what if you want your data pipeline to run based on a trigger or event instead of on a set schedule? For example, you want to run a data pipeline only when the source data changes in Microsoft Azure Blob storage. Or when an email is received indicating that new data is available. In these cases, you can configure a Power Automate flow to run your data pipeline when that specific trigger or event occurs. With this new capability, you no longer need to actively monitor the event and then manually run the data pipeline to update your data; Power Automate will do it for you.
The following triggers may be of interest to Data Pipelines workflows:
- When an Amazon S3 object is updated
- When a blob is added or modified in Microsoft Azure Storage
- When a record is created, updated, or deleted in an ArcGIS Online feature layer
- When a new email arrives with a specific subject
Receive notifications when a data pipeline run completes
Up-to-date information can be critical for making timely, data-driven decisions. When new data is available to your organization, it’s often the time for you and your colleagues to kick off your analysis workflows or generate new reports. Do you want to be notified when a data pipeline run is complete? Now, you can use Power Automate to send an email to your colleagues or post an update to Teams to let them know the latest data is available in ArcGIS Online.
The Run Data Pipeline action in ArcGIS Connectors for Power Automate returns information about the run that can be used in downstream actions for alerting and notifications. For example, send an email with a link to the run details page to view run details and links to the updated data. For more information on specific properties returned by the Run Data Pipeline action, see the About ArcGIS Connectors for Power Automate section of this blog.
To get started with sending notifications, the following actions may be of interest:
Integrate data pipelines into your larger workflows
Power Automate orchestrates complex, cross-functional workflows across many organizations, driving efficiency. With the new Run Data Pipeline action in ArcGIS Connector for Power Automate, you can now incorporate your data pipelines into existing flows to keep your processes in one place.
To learn more about Power Automate connectors and the supported triggers and actions, you can refer to the Microsoft’s Connector reference overview. If you want to learn how to get started, read the Get started with ArcGIS Connectors for Power Automate documentation.
About the Run Data Pipeline action in ArcGIS Connectors for Power Automate
The Run Data Pipeline action in ArcGIS Connectors for Power Automate can be run after any trigger you’ve configured in your Power Automate flow. The following parameters are available for you to configure:
- Data Pipelines In–Specify the ArcGIS Online content location of the data pipeline you want to run. The options are My content (this is the default) and My favorites. Note that in Power Automate you can only run data pipeline items that you own.
- Data Pipeline–Specify the data pipeline item to run.
- Maximum Run Duration (Minutes)–Use this parameter to determine how long the data pipeline can run before it is automatically canceled. The default is 30 minutes.
The Run Data Pipeline action in ArcGIS Connectors for Power Automate returns the following properties for each run:
appUrl–A link to the data pipeline run results page where you can view the outputs and any messages that were returned.runId–The ID for the run.itemId–The item ID for the data pipeline that was run.status–The status of the data pipeline run. Status returns one of the following values:completed–All outputs were successfully written.completedWithErrors–Some outputs were successful, and some outputs failed.failed–No outputs were successfully written.timedOut–The run took longer than the maximum run duration and was cancelled.cancelled–The run was cancelled.
You can leverage this information in downstream actions for sending notifications, or as conditionals for running next steps in the flow. For example, if the data pipeline fails you can leverage the failed status to block running the next steps of your flow.
Additional information
If you’re new to Data Pipelines or just want to learn more, here are some additional resources to get started:
- Read the What’s New in ArcGIS Data Pipelines (November 2024) blog for the latest news on other exciting updates.
- Watch this webinar recording to learn more about what Data Pipelines can do, plus some demonstrations of how to use the application.
- Check out some other ArcGIS Blogs about Data Pipelines.
- Follow this tutorial to et started building your own data integration workflow.
- If you have questions, check out the FAQ documentation or post in the Data Pipelines Community where the Data Pipelines team will respond.
We want to hear from you! There is more to come for ArcGIS Data Pipelines in the future, and we value your opinion on what new features and enhancements we can add to help you with your data preparation workflows. Share your ideas or ask us a question in the Data Pipelines Community.
Article Discussion: