We don't talk about storage

When designing a solution using Dataverse or Dynamics 365 CE - Dataverse storage is rarely one of the hot topics and probably most of the times overlooked. In this post I will dive a little deeper into Dataverse storage architecture and why it is important to discuss this during a Dynamics 365 CE or Dataverse implementation 

Fundamentals of Dataverse storage architecture

Since April 2019, both Dataverse and Dynamics 365 CE (Online) use a tiered storage model. This means that different data types in Dataverse are stored in the most optimal storage type. Azure files and blobs are used for attachments, relational data (tables) is stored in Azure SQL, audit logs are stored in Azure Cosmos DB, search is powered by Azure Cognitive Services, etc .... 

From an end user perspective, this is completely transparent and administrators don't need to manage all of these underlying Azure components since Microsoft takes care of all of this.

Microsoft has however different pricing schema's for these underlying storage types and the price difference is quite significant. You can follow up on the storage capacity in the Power Platform Admin Center. Based on the number of licenses inside your tenant, you will get a specific capacity entitlement for database, file and logs. (See New Microsoft Dataverse storage capacity > verifying your new storage model for more details)

Understanding the cost impact

Database storage in Dataverse comes at a premium price tag. Usually you don't notice this unless you will need to buy additional storage (see  What Dataverse capacity is included with the Power Apps and Power Automate plans? for how much additional storage you get with additional licenses).

(1) Pricing based on the list price shown in my personal tenant - August 2023. Prices may vary depending on your licensing agreement. If you buy additional  eligible licenses you will also get additional storage allocated, it is also possible to use Power Platform Pay-as-you-go plans instead of buying license but the capacity pricing using PAYG is even higher.

You need to buy additional storage capacity if you are over the allocated storage capacity since storage capacity is being enforced - if you exceed storage capacity you will not be able to create a new environment (requires a minimum of 1 GB capacity) but also copy, restore and recovery operations will be blocked.

Archiving and data retention policies

Besides the impact on your budget of retaining all data in Dataverse (or Dynamics 365) forever - you however also need to consider the potential security and legal risk. Best practices dictate that data should only be kept as long as it is useful or as long as you are legally allowed to retain it - GDPR mandates to define specific data retention period's for personal data.

Defining a data retention policy helps businesses reduce legal risks, security threats and also reduce costs. Data retention policies contain the data retention period and the required actions to take when this period lapses. You should have a data retention period defined for each of entities/tables in use for your Dataverse environment.

In July 2023 finally released Dataverse long term data retention overview in public preview.  This feature allows you to create a view on Dataverse tables/entities for data that you need to retain for a longer period. The view is used as a selection criteria to define  which data is moved to long term archive storage in a Microsoft managed data lake. With this feature, you might be able to use out of the box functionality  instead of having to built your own custom solution.. This functionality is still in preview and should not be used on production - also keep in mind that no pricing details are available yet (they will be announced around the GA timeframe). In a upcoming post I will delve a little deeper into the archive/long term data retention functionality but you can already take a look at Early dive into Dataverse Long Term Retention

Related posts and references:

• Storage capacity management for Dynamics 365/Dataverse - how to track storage evolution with Power BI
• Inside Dataverse Data Storage -Power CAT Live YouTube video
• Cost optimization: older mails in Dataverse now stored in Dataverse file storage
• Storage capacity management for Dynamics 365/Dataverse - how to track storage evolution with Power BI

Reducing size of the SubscriptionTrackingDeletedObject in Dynamics 365 CE or Dataverse

When you delete a large amount of data from Dynamics 365 CE or Dataverse, you will notice that the total storage consumed does not decrease. This is because the SubscriptionTrackingDeletedObject table stores deleted records from the Dynamics 365 database. This table is used to track deleted records for replication and synchronization purposes and records are being kept by default for quite a large amount of time.

When recently encountering this problem, Microsoft support suggested us to reduce the number of days the records are kept in the table from the default 90 to 15 days - to do this  take the following steps:

• Install the latest version of the OrganizationSettingsEditor solution which you can download from https://github.com/seanmcne/OrgDbOrgSettings/releases   (The OrgDBOrgSettingsTool dates back to Dynamics CRM 2011 but is being maintained by Sean McNellis and is still quite relevant)

• Change the ExpireChangeTrackingInDays and ExpireSubscriptionsInDays from the default setting of 90 days to 15 days. In sandbox environments, you might be able to change this value to 0 but for production environments Microsoft recommended to keep it on a minimum of 15 days. The OrgDBSettings utility utilizes the solution configuration page to provide access to the editor (in classic mode) - see OrgDBOrgSettings - where to find it after installing? for details.

Afterwards - just keep patient until the lapsed number of days have been passed and then you should see a size in reduction for the SubscriptionTrackingDeletedObject

Cost optimization: older mails in Dataverse now stored in Dataverse File Storage

Back in April 2019, Microsoft introduced a tiered storage architecture. This tiered storage architecture allows Microsoft to use the most appropriate storage type for different types of data used in Dynamics 365 and Dataverse. Last month, Microsoft started moving  the storage contents of the e-mail body (email.description) from more expensive Dataverse Database storage to cheaper Dataverse File Storage - for the official documentation check out Dataverse - Email activity tables -Email storage.

Based on the list prices I see in my personal tenant (see screenshot below) - this cuts the unit cost for additional capacity required for e-mails down from 37,4 €/GB/month to 1,87€/GB/month (Quite interesting how they still use the older name Common Data Service :-) )

Your pricing might defer based on the type of licensing agreement that you have.  If you had to buy additional storage capacity in the past, you might want to revise your storage usage and the required storage capacity add-ons.

You can check out the storage capacity in the Power Platform Admin Center - I added an example from a specific instance were Dataverse storage for e-mails (see ActivityPointerBase table) from 63 GB to 30GB.

A the same time, we saw a new storage file type appearing called Email which grew to 3,8 GB. This is a lot less than the storage saved in the database since the data in file storage (Azure Blob Storage behind the scenes) is actually compressed.

All in all, a great job from Microsoft which might save you a lot of money.  Don't hesitate to add a comments sharing your experience with this.... 

Storage capacity management for Dynamics 365/Dataverse - how to track storage evolution with Power BI

After the introduction of the new storage management model in Dataverse in April 2019, both Dataverse and Dynamics CRM Online are using a tiered storage model (separation between database capacity, file capacity and log capacity). 

Based on the number of licenses acquired, every customer has a specific storage capacity entitlement - you can find an overview of storage consumption within the Power Platform Admin Center - section 1 shows an overview of all storage types consumed across all environments (support and trial instances are excluded), section 2 shows the allocated capacity based on number of licenses or purchased additional capacity and section 3 shows the biggest environments. 

To see where Dataverse storage is being used across your environments, you need to click on the second tab. From this page you are also able to download a CSV file with storage allocations for all environments to which you have access. The information on this page is updated every 24 hours and unfortunately no historical data is available so it is quite hard to track storage evolution.

I currently track the evolution of storage by downloading this CSV file and storing the exported file on SharePoint - then it is quite easy to use Power BI to combine the CSV files and create your own Power BI report and dashboard based on the storage details. I added a screenshot of one of the report pages below, for those of you interested in this Power BI report send me a DM on @jopxtwits 

References:

• Storage capacity enforcement for CDS instances (June 2020)
• Microsoft Dataverse data archival (2021 Wave2) - public preview initially expected in March 2022 but postponed to a later date.
• Dynamics CRM Data Cleaning
• Administer Power Platform - Free up storage space [Microsoft docs]
• Administer Power Platform - New Microsoft Dataverse storage capacity [Microsoft docs]
• Dynamics 365 Storage Utilization | Dataverse storage | Power Platform Admin Center