Dataverse search: the good, the bad and the ugly

The enhanced search experience for model driven apps (initially called relevance search) on Dataverse and Dynamics 365 CE was initially announced in November 2020 and you could start using it a couple of months later. Recently Microsoft announced some changes (and improvements) for relevance search in Goodbye Relevance search, hello Dataverse search! 

When you plan to use Dataverse search, there are a number of things that you need to keep in mind which are not so obvious from the Microsoft documentation so let's dive right in and look at the Good, the Bad and the Ugly about Dataverse search.

Dataverse search - The Good

The search box has moved in the UI and is now very easy for end users to find back. When you start typing text in the Dataverse search box, you will see a list of suggested results as well as previous queries.  The way that the suggested results are shown can even by slightly changed as outlined in this blog post - Quick tip - Data in the inline suggestion view of relevance search in Dynamics 365 CRM. Dataverse search is powered by Azure Cognitive Search so you get support for fuzzy search and advanced capabilities like word stemming and work breaking. For more documentation on the end-user experience see Search for tables and rows by using Dataverse search

With the new solution explorer update it also is a lot easier to configure which entities/tables you want to be indexed for search. The settings are solution aware so if you have multiple solutions installed in your Dataverse environment be aware of solution layering. To get an overview of all indexed tables - select the Default solution in the maker portal and click Manage Search Index - for more details see Configure Dataverse search for your environment

Relevance search is also available through a REST API meaning that you can also use it in Canvas Apps & Power Automate (See Use Relevance Search API in Power Automate & Canvas App - Part 1). To begin using Dataverse search, your application simply needs to use POST requests  to the /api/search/v1.0/query endpoint - for the complete documentation see  Search across table data using Dataverse search in Microsoft docs.

Dataverse search - The Bad

Although there is no specific license required for Dataverse search - and it works both for environments without Dynamics 365 apps enabled and with Dynamics 365 apps - there might be a cost associated to it. 

When you enable Dataverse search, additional storage is required for creating the indexes in the Dataverse database storage. You will probably not notice this in your development environments since they might not contain sufficient data. But in production environments the increase can be quite significant - I have seen Dynamics 365 Customer Service environments where as much as 16% of the database usage is taken by the RelevanceSearch table.

In my opinion it would be better to add a more prominent warning of possible storage increase to the main documentation pages about Dataverse search - there is however a mention on the Free up storage space documentation - remove unrequired tables and columns from Dataverse search. 

With the Wave 2 2021 release (after October 2021) Microsoft enabled Dataverse search by default for all production environments so probably want to find out what the impact is on your storage  consumption.

I also did not expect the storage to be added in the database which at roughly 40€ GB/month for additional database storage does not come cheap.

Dataverse search - The Ugly

Now suppose that you are suddenly faced with an increase in storage in one of your production environments after you activated Dataverse search. Unfortunately just disabling the Dataverse search again is probably not an option because your end users probably liked the functionality. So your only options are to start reviewing which tables and which fields to add to the index. The entities which are enabled for search are clearly visible from the administration screen but unfortunately finding out the fields which are added to the relevance search index is a little more involved.

So if we take the example of the "accounts" table and look at the "Quick find" - out of the box there will be 8 fields marked as added to the index - account name, account number, primary contact (lookup), address1 city, main phone and e-mail. If you want to modify this e.g. remove account number - you will create a new solution and add the "accounts" table and "quick find" to the solution (via add existing components) and remove the column/field from the Quick find view and from the "Find by" settings. After you made these changes - you will notice that the number of indexed fields decreased.

Unfortunately in bigger organizations, you are not allowed to make changes in production environments without following a strict process. 

So an iterative process of  removing tables from Dataverse search results and modifying the fields selected for each table in an iterative manner to assess the impact is not always easy to find the balance between storage increase and providing functionality is not always easy. It would be a lot easier to see how much each table and preferably also each field contributes to the RelevanceSearch table storage size so that you now which changes are required.

The amount of storage required depends on the number of rows in the table, the number of indexed columns/fields and the amount of data stored in a specific column - a text field/column with many characters will consume more space then a text field with only a few characters. So if you are indexing e-mail activities - adding the description field might cause a significant increase. The indexed fields include Find columns (isSearchable), View Columns (isRetrievable) and Filter criteria (isFilterable).

References:

• Simplified Dataverse Search Configuration by @D365Goddess
• Quick tip - Data in the inline suggestion view of relevance search in Dynamics 365 CRM by @Inogic
• Goodbye Relevance search, hello Dataverse search!
• Enhancements in Relevance Search in Dynamis 365 CRM by @Inogic
• Configure Dataverse search for your environment