We recently enhanced the Search widget in that you will see across all web apps like Map Viewer, Map Viewer Classic, Instant Apps, Experience Builder and Web AppBuilder. The recent change to the Search widget can return faster results more efficiently and maintain stability for the layers being used to search for features. The layer-based search performance benefits increase as the number of features increase, the searched term character count length increases, and the field being searched also has more data along with repeated searches.
Updates Made to Search Widget
Using Full-Text field indexes for layer-based feature searches with ArcGIS Online hosted feature layers.
First off you are probably asking – What is a Full-text field index?
This type of index is created when you author a feature search under the application settings of a web map or in the instant apps layer-based search configuration. When the index is created essentially it will take the field specified in the search configuration and then scan the data in the field and tokenize the words within each row. Which pretty much means behind the scenes we are splitting up the data in the field into smaller units, such as individual words or terms that will improve the overall search performance.
Example:
Search is configured on layer with all the US Post Offices and using the Name field. An example result on the name field is Highlandtown Station Baltimore Post Office. The index will split up this example into these searchable terms Highlandtown Station Baltimore Post Office. This will make all the the terms now searchable with the full-text index where you could search Baltimore Office, Highland Baltimore, Stat Balt Office and be able to retrieve this feature from the search results.
How does the Search widget use these indexes?
The Search widget will now leverage Full-Text field indexes on hosted feature layers layer-based source searches. The index automatically gets generated when feature search is authored on a field for a hosted feature layer in web maps authored after the November 2022 release. You can update existing maps or create new ones to generate these indexes, so if this index type exists on the field being searched the widget will use it automatically. The search type being used in the search widget is called prefix. This will query the field with the characters searched and return results for words that begin with the specified text. These indexes greatly improve performance and stability especially with large search terms which we have seen with our testing take sub second vs. the double wildcard taking over ten seconds (more on the wildcard changes below). You can see an example of this prefix query in the above example Stat Balt Office where the prefix search is used to find Station, Baltimore and then a complete term Office. However, searched for More you would not get results for Baltimore.
Remove the leading wildcard for layer-based feature searches.
When searching for features using the Contains layer-based search and for suggested results on Equals the search widget in previous releases would send the searched for term using the LIKE operator and a % wildcard on each side of the term. This would then go through the field set in the layer-based feature search and try and match that character pattern with all the data in the field. This becomes a very heavy query when the number of characters in the search term increases the amount of data that has to be scanned in each row and the number of rows that exist. Eventually this could cause issues for your content as traffic increases, the number of features that exist increases and more searches are performed on your data.
Example:
Search is configured on the layer Post Office and using the field Name on a layer with NO Full-Text Index. When searching for Highlandtown Station Baltimore Post Office the entered search text would have to now start with Hig to start returning results that begin with Hig and Highlandtown Station Baltimore Post Office falls within that searched character pattern and could be a returned result. This is because the leading wildcard is not used, and the query becomes “starts with”. The change will not support being able to search for other character patterns that could happen elsewhere in the string like searching for example searching Baltimore will not return Highlandtown Station Baltimore Post Office but would return any results that start with Baltimore.
What’s Next
First off – I hope this blog helps you understand the recent changes we made to searching for features. We want to bring you faster results and stability to your content. Up next we have plans for bringing the Search authoring experience into Map Viewer. This will make building and trying out your searching you author easier and faster. We also have plans to expose the indexes that exist on you’re layers to make it easier to understand what indexes exist and give you additional controls there. Please leave feedback here in the blog post or reference this blog if you have any questions in the Esri Community.
Russell, can you share any information on plans to enable this for Feature Layers that are not “hosted” but are “registered” ArcGIS Server services? Is there a realistic chance of that happening? That would help us plan ahead. Great blog post, thank you.
looking into this and will post a new reply when I have some info for you.
How will Full-Text Field Indexes work for feature layers we do not own? If an index is created when the search is added, assume you need to be the owner of the feature layer.
What is the feature layer is either a Living Atlas layer, or another agencies rest service. What is the best way to add a search to those layers?
Indexes are built by owners or admins bc this info is added to the service definition. The non-index case where the index does not exist on the service will use the search without the leading wildcard.
Thank for for the fast response.
For the non-index cases does the lack of leading wildcard mean you are unable to search “Baltimore Post Office“.
When the full name is “Highlandtown Station Baltimore Post Office”, previously we have used contain for this, recently have seen some issues with the search results.
Yup you would have to add the % manually in front of the “Baltimore Post” Office to get the “Highlandtown Station Baltimore Post Office” result.
Awesome, great to know thank you.
I wish there was an option to automatically add the “%” in the search configuration options. The general public would not think to add it if they are just trying to find a Post Office.
Adding the indexing is only an option, if I own the data.
Is this something that is implemented automatically when we create a hosted feature service? Also, you mentioned Instant Apps and the application settings of a web map – is there documentation on how to do this in the settings of a web map? What about Experience Builder?
The index gets generated either when search is configured in the web map settings or instant app layer based search. No changes were made to the feature search authoring experience in map settings. Once you setup your layer the index gets built behind the scenes. Experience Builder will support it in the next release. https://doc.arcgis.com/en/arcgis-online/create-maps/configure-feature-search.htm
The Search widget doesn’t work in our maps since the November release. Or it works as ‘starts with’ while the search is configured as ‘contains’. I could repair this in a few maps by deleting/re-adding the search on layer settings, but this trick doesn’t work with most maps. This is really annoying and confusing for our users, is it meant to be so? This is not documented, and our local distributor didn’t seem to be aware of the problem neither. Are we really the only ones experiencing this?
Yes this change switched from using the double wildcard to a single wildcard at the end of the searched characters. Adding a % at the beginning will restore the same search but this is something we want to move away from because this can cause stability issues for your content depending on the number of active searches taking place, num of characters being used and the num of characters in the field rows the searched characters are being compared against. You can add a full text index either through making a change through the search settings config panel or if… Read more »
Hi Russell, the first option doesn’t work in most cases, so I’d like to hear about the second one, thank you.
When you go to the addToDefinition on the rest endpoint of your Hosted Feature Layer (not the top level but the individual sub layers) and no other Full-Text index exists you will remove the text already in the text box and enter this below with the fields you want to be used. Name must be unique. { “name”: “fulltext_indx”, “indexType”: “FullText”, “fields”: “FeatureCategory” (coma separated fields you want to have an index on, only one fulltext index per layer) } If a fulltext index already exists that needs to be deleted from the definition first before adding new field indexes.… Read more »
It just doesn’t work with a layer with views, I can’t use addToDefinition.
Error message:
Adding the index works for another layer without views, I can see the index is added when printing the properties, but the search still doesn’t work in the map, even after deleting the map’s search conditions – saving – re-adding the search conditions.
So what do I do now? This really sucks, the search widget was working perfectly before, this is not an enhancement but a regression for us.
For the above issue can you log a support ticket for us to investigate and track? Please include a sample of the feature value you are trying to search for.
The change had to be made b/c the old search method could impact your entire orgs performance and stability.
Hi Russell, I have a ticket logged with my local distributor since 25 November (#03206890). They say they’re in touch with Esri US. Do you have access to it?
Yup I can see it.
For the view case – the index exists on the parent layer not the view. So you wouldn’t need to build the index on the view layer. If you do this in a map you can see in the network the addToDef gets applied to the parent and not the view. The view leverages the FT index and you will see in the search widget the query going through the view using the FT index.
I’m not using the views. But the feature layer I’d like to update (and can’t) has related views.
And anyway, the search still doesn’t work with a layer with a full text index successfully added.
Will need to see the map, layer and the search you are trying to perform.
Your comment earlier mentioned you are using it with views and trying to add it to a view.
I said it doesn’t work with a layer with views. How could I give you access to the maps and layers?
Add them to the support case or create a post on Esri ArcGIS Online community which we can use to further look at the issue. Share the content to a group and invite russell_jsapi into it.
I’ve invited you to a group with the maps and layers, but not sure how I can add them to the case. And should I keep on writing here, or via the case, or email?
Create a post under the ArcGIS Online community and I’ll catch it and jump in. I just need to know the map and layer being searched. Thanks!
OK, done. Thanks!
9/25/2023 – Russell I have enabled search on my “Web Map”. This then automatically as noted in documentation enables that search in the “Field Apps”. I see documentation that says it also instantly enables that search in the new apps. I have an “Instant – Side Bar App” and the search bar shows my configurable words from the “Web Map” in that App where I put “Hint Text – Search My Data” but nothing returns. Is there still some disconnect? I wanted to avoid configuring the search in the “Instant – Side Bar App”
Solved it: In the “Instant – Side Bar App” I had to enable “Search Configuration – All Sources”. Otherwise it was just searching my top layer.