This is the final article of the series rounding up the latest features related to data interoperability, enterprise integrations, ArcGIS extensibility, and open source analytics. These features highlight key advantages of the ArcGIS open platform architecture. With the distraction of global pandemics and all the other feature goodies, we wanted to make sure you scientists, data scientists and code-curious analysts didn’t miss these new features and how-to’s from across the ArcGIS platform. So we’ve looked back through the past several months of blogs and videos and picked out the most important related announcements.
Part 4 focuses on how data scientists can save time by integrating open source analytics tools into their ArcGIS enterprise workflows. If you want all the recent updates by product instead, see this Q1 ArcGIS Release roundup.
Save Time on Advanced Analytics Projects
ArcGIS is blowing out its offering for the enterprise data scientist this year with more ArcGIS Notebooks options and deeper integrations with deep learning frameworks and the R community. These new releases are making it easier than ever to integrate open science practices within your organization. This round up focuses on those tools that leverage interoperability with external, open source libraries and tools.
Run Python Notebooks from Desktop or the Cloud- New Products!
ArcGIS Notebooks allow you to create and run your own Python-powered notebook using open source and ArcGIS Python Libraries. You can also hit the ground running by importing existing Jupyter notebooks (.ipynb) to better share your analysis with your organization. And you can redeploy downloaded ArcGIS Notebooks in other python environments as well after tweaking how you access ArcGIS services. Bringing Notebooks inside ArcGIS allows seamless access to your organization’s location assets as well as ArcGIS analytical tools using the ArcGIS Python Libraries.
Hosted ArcGIS Notebooks have previously been available with the ArcGIS Enterprise Notebook Server with options to customize your runtime libraries. But now you can work natively with ArcGIS Notebooks in the ArcGIS Pro 2.5 desktop application, and as hosted items in the ArcGIS Online SaaS platform. With hosted notebooks in Online, you don’t have to worry about managing package versions and infrastructure to share your results with colleagues and maintain your analysis for use across the organization. These options increase flexibility in how you can bring the power of Notebooks fully into the enterprise.
Convenient access to open source Python integration has long been a feature of ArcGIS Pro, but now with built-in Python notebooks, you can more readily explore all that open source integration has to offer. Create pandas dataframes, data visualizations with libraries like matplotlib, or read in large data sets using dask. The interactive notebook interface gives instant feedback, if your code worked or didn’t, without needing to debug the entire script, making it ideal for iterative analysis and data discovery. See the live demo below.
Learn more about Notebooks in ArcGIS Pro. Learn more about the Notebooks Public Beta offering in ArcGIS Online and the options for standard or advanced prepackaged libraries (which are the same as for Enterprise 10.7.1). Or watch the Python API technical deep dive from virtual 2020 Esri Developer Summit.
Further Enterprise Notebook Automation
Don’t forget about hosted Notebooks in ArcGIS Enterprise! First released as a server role for Enterprise, Notebooks in Enterprise 10.8 now allows you to automate analyses with the new “Execute Notebook” operation. This operation can be triggered dynamically by an Enterprise webhook or as a scheduled operation within Pro itself. The Execute Notebook API allows notebooks to run using external task schedulers. Learn more about automated notebook execution.
Deep Learning Integration Improvements
Integration with open source deep learning frameworks has been part of ArcGIS Desktop and Enterprise for several releases. Recent upgrades allow you to bring open deep learning frameworks inside ArcGIS Pro, leverage multiple GPUs in parallel and more easily train and apply new deep learning models on full motion video.
Previously, you could use ArcGIS geoprocessing tools to create training data and export those packages to other frameworks to train deep learning models. Now, you no longer need to leave ArcGIS Pro to train your model in the deep learning framework. You can install the TensorFlow, Keras, PyTorch, Fast.ai, and Torchvision Python packages for ArcGIS Pro deep learning frameworks to work with ArcGIS Pro. See Install Deep Learning frameworks for ArcGIS for more details.
Integrating deep learning frameworks with ArcGIS Pro, coupled with the new tools and features described below, results in a seamless end-to-end deep learning workflow experience. Now with the ArcGIS Image Analyst extension in ArcGIS Pro 2.5, two new tools have been added to the Deep Learning toolset that can train a deep learning model and classify objects:
- Train Deep Learning Model—Trains a deep learning model using the output from the Export Training Data for Deep Learning tool.
- Classify Objects Using Deep Learning—Runs a pre-trained deep learning model on an input raster and an optional feature class to produce a feature class or table in which each input object has a class label assigned.
Additional tools and capabilities provided for deep learning can also be accessed through the arcgis.learn module of the ArcGIS API for Python. Learn more about new ArcGIS features.
Given that imagery analysis is a popular use case for deep learning, starting at ArcGIS Image Server 10.8, users can use single node parallel processing across multiple GPUs to speed up deep learning workflows. Learn more about Image Server’s deep learning updates.
Applying Deep Learning to Full Motion Video
At the recent Esri Developer Summit, Shannon Kalisky demonstrated how to train a catfish detector model using a training set derived from ArcGIS Pro’s tools for full motion video. Using ArcGIS Notebooks, she walks through how to:
- Create training data from a video
- Export that training data
- Run the Train Deep Learning Model tool (or use SingleShotDetector from the ArcGIS API for Python’s arcgis.learn module)
- Conduct subsequent surveys: Processing drone video using the SingleShotDetector’s predict_video method
The trained model effectively detects objects in an entire video and creates a .csv file containing the location and size of the detected objects in each frame of the video.
R-eproducing the Science of Where: R-Integrations with ArcGIS Pro
The importance of sharing and the ability to repeat workflows are resonating in every discipline, ranging from natural sciences to data science. To continue scientific progress and strengthen “digital resilience” (Wright, 2016) we need the ability to share and repeat analytical work freely. Through Python and R analytical packages, much of the open innovation can be reproduced. With new recent capabilities, the contributions of the R community can be better supported, extended and operationalized within organizations using ArcGIS.
The R-ArcGIS Bridge (arcgis-binding) has been around for several years to help users with two-way data exchange between ArcGIS and integrated development environments for R users like R Studio and Microsoft R. In addition, it has allowed ArcGIS Pro users to run domain specific R functions inside ArcGIS geoprocessing tools.
Now, with the recent improvements, users can:
- Enable R and ArcGIS integrations in Python Notebooks through the Conda package r-arcgis-essentials
- Access remote data services hosted in ArcGIS Enterprise or Online
- Call ArcPy code and run geoprocessing tools with R-enabled notebooks using the package reticulate
These features bring more of the power of ArcGIS data and functions to the R Community and helps integrate the best of the Python tools. While these R integrations work for local and third party R notebook clients and servers, they are not enabled directly in the existing ArcGIS Notebooks implementations for Pro or Online. However, users can install and run separate R-enabled containers in the ArcGIS Enterprise Notebook Server.
For a recent overview and demo of R integrations with ArcGIS desktop applications, see the video below.
GIS or TIS? Better Space Time Cube Support
ArcGIS continues to improve capabilities for analyzing scientific data over space and time. The Geographic Information System (GIS) is increasingly becoming a Temporal Information System (TIS) at the same time. Scientists can now make Space Time Cubes from TIF, netCDF and CRF data with ArcGIS Pro 2.5. Learn more about how to mine patterns in raster data over time.
ArcGIS Enterprise Image Server 10.8 also now better supports multidimensional CRF data files and allow users to make a space-time cube with netCDF data, in addition to supporting the HDF5 (4d) format and the OPeNDAP protocol to help scientific organizations easily access their data. Learn more about What’s New in Image Server.
In Sum
This wraps up our Open Platform roundup series. You can peruse earlier articles that highlight other recent features.
- Part 1: Data Flow – Bringing Data In and Sending it Out
- Part 2: Streamline Workflows with Enterprise Integrations
- Part 3: Customize to Meet Your Unique Needs
- Part 4: Save Time on Advanced Analytics Projects
Commenting is not enabled for this article.