In our recent webinar “Introducing Generative AI and Machine Learning in ArcGIS Survey123”, we fielded several insightful questions. We’ve compiled and addressed these subjects in this blog. If you’re tuning in for the first time or simply revisiting the content, you can watch the webinar recording below.

Answering Your Questions

Administration/Access:

How do I enable the beta features for the Survey123 Assistant and Auto Translate for organization?

Your ArcGIS Online administrator has permission to enable your organization’s AI Assistant and Auto Translate beta features. Once your administrator has toggled the feature on, then the rest of the organization will be able to see it available in their surveys. To do this the administrator needs to login to survey123.arcgis.com, click on the Organization tab | Setting | Extensions, and then toggle on/off the Assistant and Auto translate features.

Do users have to have their own separate account (Microsoft or ChatGPT) to use the assistant?

No, the Survey123 assistant is available to all survey creators if it has been enabled by the ArcGIS organization administrator.

Pricing and Licensing:

What is the pricing model for AI features within Survey123? Will services consume credits?

The pricing model for the use of AI features is still being developed.

Is additional licensing required to use AI?

No, while in beta the Survey123 AI features available to all survey creators if it has been enabled by the ArcGIS organization administrator.

Privacy and Security:

What are the privacy and security guidelines for AI features? Does this use comply with the same privacy and security standards as the rest of ArcGIS Online?

Yes, the Survey123 assistant and image extraction features are both hosted in the ArcGIS Online ecosystem so all privacy and security requirements there will apply to these features. Your data and input are always secure and private to you.

AI within Survey123 Components:

What Smart Assistant and Machine Learning features are available for use between the Survey123 Web Designer, Connect, and Field App?

AI in the Survey123 Mobile App: 

Will the AI features work in an offline survey?

Smart redaction, smart annotation, and smart attributes function in an offline environment because they use a TensorFlow Lite model that is downloaded with the survey file. A connected environment is required to use the Survey123 assistant, auto translate, and extracting information from images using generative AI.

Does the object detection model work on photos added from the mobile devices photo gallery?

Yes, smart redaction, smart annotation, and smart attributes can be used with images selected in the app from the file system or with images taken with the camera.

AI and ArcGIS Enterprise:

Does the Survey123 Assistant work within ArcGIS Enterprise?

Survey123 Assistant will not be available with the ArcGIS Enterprise Survey123 local website installer as it is an ArcGIS Online service. However, there are plans to enable ArcGIS Enterprise organizations to configure access to AI features, similar to other  ArcGIS Online services (such as the Survey123 Report service). 

AI Image Calculations:

What kinds of features can the image extraction assistant recognize from an image, and is there a way to train the information extraction from image tool?

The image extraction assistant can identify any features that match the prompt you provide the assistant. It may take a few tries of prompt engineering to identify the prompt that works best for the specific features you are trying to identify. There is not a way to train the image extraction assistant to better understand your workflows, for more specialized workflows we would suggest training and using your own model.

With object detection and image classification, does the model save these observations in the feature layer as a list?

When extracting attributes from an image a “ImageDescription” EXIF tag will be written to the image. By default, the JSON object written to the “ImageDescription” EXIF tag won’t be saved as a field in the ArcGIS feature service. To do so, you can use string(pulldata(“@exif”, ${photo}, “ImageDescription”)) in the calculate column of another question to write this information to the feature service. For more information, please see Prepare smart assistants documentation.

Does the smart attributes feature provide an accuracy rating or percentage based on the information it was able to extract from an image?

When extracting attributes from an image a “ImageDescription” EXIF tag will be written to the image. That “ImageDescription” Exif tag will contain a JSON object containing an array of “classes”. Each class will be an object that will contain a “score” attribute providing the confidence score. For more information, please see Prepare smart assistants

Pre-Trained Models:

Can a Survey123 user train their own model?

Our end users can create their own models for object detection and image classification using ArcGIS tools. However, a model’s accuracy depends on the number of images used to create it. More on model creation here.

Are there any pre-trained models available for customer use?

The Common Object Detection model is available in the ArcGIS Living Atlas of the world for use in Survey123. This is the only model in the Living Atlas optimized for use in Survey123. Please note that this model is not intended for production workflows and is only provided to demonstrate functionality.

How is Esri going about improving the Large Language Model used by the Survey123 web designer?

The Survey123 web designer assistant leverages a commercially available Large Language model, which is not owned by Esri. While Esri has no plans to modify this foundational model, the development team is continuously incorporating user feedback to improve the output of this model for the purposes of Survey123.

AI and Advanced Workflows:

Will Survey123 allow the survey designer to choose what Azure Open AI model to use? Will users be able to their own Azure Open AI service instead of using Esri’s?

At this time users will not be able to choose their LLM of preference, but it is something we can look to support in the future.

Does Esri have any QA/QC processes that conduct quality control of the Survey123 web designer assistant?

Yes, the Survey123 team conducts regular QA/QC processes against the assistant. This includes tests to monitor the quality of outputs as well as strict controls to guarantee compliance with ArcGIS privacy, security, and legal guidelines. Esri is not responsible for the responses from LLM’s. For more on current AI information, please see our section on AI Assistants in our UC Q&A page.

Final Words

Thank you to those of you that attended and to those who are watching later. If you would like to learn more about these enhancements made to ArcGIS Survey123, read our February 2024 release blog summarizing these features. Stay connected with our Survey123 Community and Survey123 Newsletter to always get the latest and greatest from ArcGIS Survey123!