When designing a solution using Dataverse or Dynamics 365 CE - Dataverse storage is rarely one of the hot topics and probably most of the times overlooked. In this post I will dive a little deeper into Dataverse storage architecture and why it is important to discuss this during a Dynamics 365 CE or Dataverse implementation
Fundamentals of Dataverse storage architecture
Since April 2019, both Dataverse and Dynamics 365 CE (Online) use a tiered storage model. This means that different data types in Dataverse are stored in the most optimal storage type. Azure files and blobs are used for attachments, relational data (tables) is stored in Azure SQL, audit logs are stored in Azure Cosmos DB, search is powered by Azure Cognitive Services, etc ....
From an end user perspective, this is completely transparent and administrators don't need to manage all of these underlying Azure components since Microsoft takes care of all of this.
Microsoft has however different pricing schema's for these underlying storage types and the price difference is quite significant. You can follow up on the storage capacity in the Power Platform Admin Center. Based on the number of licenses inside your tenant, you will get a specific capacity entitlement for database, file and logs. (See New Microsoft Dataverse storage capacity > verifying your new storage model for more details)
Understanding the cost impact
Database storage in Dataverse comes at a premium price tag. Usually you don't notice this unless you will need to buy additional storage (see What Dataverse capacity is included with the Power Apps and Power Automate plans? for how much additional storage you get with additional licenses).
(1) Pricing based on the list price shown in my personal tenant - August 2023. Prices may vary depending on your licensing agreement. If you buy additional eligible licenses you will also get additional storage allocated, it is also possible to use Power Platform Pay-as-you-go plans instead of buying license but the capacity pricing using PAYG is even higher.
You need to buy additional storage capacity if you are over the allocated storage capacity since storage capacity is being enforced - if you exceed storage capacity you will not be able to create a new environment (requires a minimum of 1 GB capacity) but also copy, restore and recovery operations will be blocked.
Archiving and data retention policies
Besides the impact on your budget of retaining all data in Dataverse (or Dynamics 365) forever - you however also need to consider the potential security and legal risk. Best practices dictate that data should only be kept as long as it is useful or as long as you are legally allowed to retain it - GDPR mandates to define specific data retention period's for personal data.
Defining a data retention policy helps businesses reduce legal risks, security threats and also reduce costs. Data retention policies contain the data retention period and the required actions to take when this period lapses. You should have a data retention period defined for each of entities/tables in use for your Dataverse environment.
In July 2023 finally released Dataverse long term data retention overview in public preview. This feature allows you to create a view on Dataverse tables/entities for data that you need to retain for a longer period. The view is used as a selection criteria to define which data is moved to long term archive storage in a Microsoft managed data lake. With this feature, you might be able to use out of the box functionality instead of having to built your own custom solution.. This functionality is still in preview and should not be used on production - also keep in mind that no pricing details are available yet (they will be announced around the GA timeframe). In a upcoming post I will delve a little deeper into the archive/long term data retention functionality but you can already take a look at Early dive into Dataverse Long Term Retention
Related posts and references:
- Storage capacity management for Dynamics 365/Dataverse - how to track storage evolution with Power BI
- Inside Dataverse Data Storage -Power CAT Live YouTube video
- Cost optimization: older mails in Dataverse now stored in Dataverse file storage
- Storage capacity management for Dynamics 365/Dataverse - how to track storage evolution with Power BI
