Here is a little trick that I found on how you can improve your blogposts with Microsoft Copilot. Use this two prompts:
There is a new (annoying) functionality which show meeting invite notifications in your Microsoft Teams activity feed. To disable this go to your Teams settings and open the 'Notifications and activity' menu on the left sside. At the bottom you will find the calendar notifications which you can change to 'Off'. See Manage notification settings on Microsoft support for more details
In Python, a naive datetime object is one that does not contain any information about time zones or daylight saving time. This means it is unaware of the context in which it exists, such as whether it represents local time, UTC, or any other time zone. By default, the datetime object in Python is naive. You can make them timezone aware using the pytz library.
If you are working with pandas dataframes or series, you can also use the tz_localize method of the Pandas DateTimeIndex object.
References:
If you encounter issues when setting up Azure Synapse Link for Dataverse - the first place to check is the Azure Synapse Link Troubleshooting Guide - https://aka.ms/synapselinkTSG .
It is important to keep your Dynamics 365 development tools up to date, in the past I did this by using a PowerShell script provided by Microsoft.
But now the Plugin Registration Tool (prt) and Configuration Migration Tool (cmt) are part of the Power Platform CLI - see Dataverse development tools for more info - so you can launch the Plugin Registration Tool easily from command line using "pac tool prt"
When you install the Power Platform CLI with .NET tool (which requires .NET 6.0 to be installed) you need to use the same mechanism to keep the CLI up to date.
Mid May 2023, a warning popped up in Microsoft Message Center regarding "Non-Compliant Power Automate Flows" - soon after the message however disappeared so this got probably missed by the majority of Dataverse and Dynamics 365 admins.
This however raised some concerns in the broader Microsoft community ( see Will Power Automate enforcement licensing kill your flows? and Upcoming licensing enforcement in Power Automate explained ).
To be honest I did not pay a lot of attention since the message apparently vanished in thin air and after consultation with Microsoft support they said that this message was sent prematurely. But then, end of October another warning popped up.
There is a PowerShell script to identify the flows which at risk to be turned off across your tenant - see I have many environments - how can I get the flows that need my attention across tenants in the Power Automate Licensing FAQ - which uses the Get-AdminFlowAtRiskOfSuspension cmdlet
The Get-AdminFlowAtRiskOfSuspension cmdlet is part of a separate PowerShell module which you can install using Install-Module -Name Microsoft.PowerApps.Administration.PowerShell. It will run a scan of your environments and outline
Check out Associate flows with apps - Power Automate | Microsoft Learn on how you need to link up a flow with an app (see below screenshot on where to do this in the Power Automate flow detail screen). If you make this change on a flow which is a part of a solution, then the associations will be part of the solution file and can be transported cross environments.
Related articles/blog posts:
This is the first in a series of blog posts about more advanced generative AI and Large Language Model (LLM) concepts which I use as notes to myself (check out Why I blog and you might want to consider it as well)
Retrieval-augmented generation (RAG) is an AI framework that enhances the quality of responses generated by large language models (LLMs). LLMs are trained on a massive amount of data and understand statistical relationships between words but lack true comprehension of their meanings. So when faced with specific questions in a dynamic context so that is where RAG comes in.
RAG integrates information retrieval into LLM answers by using these steps:
Technical topics (Configuration, customization and extensibility)
Since end of 2022, there is also a SQL Server Integration Services Project template available for Visual Studio 2022 which you can install from the Visual Studio Marketplace. You can install it from the direct download link here or you can search for it in the Visual Studio 2022 extension manager and install it from there.
If you missed it - classic Azure Application Insights will be deprecated on February 29th 2024. If you missed the different notification e-mails, you can quite easily see the warning if you navigate to an Azure Application Insights resource in Azure Portal.
Migration is actually quite easy - you just click on the link provided and this will open up the menu depicted below which allows you to associate your Azure Application Insights resource to a Log Analytics Workspace. The good news is that there are no pricing changes when moving to the workspace-based model.
As indicated in the migration window, this is a one way operation so plan for it in advance - the points below might impact on how you will do the migration:
Lijst van ondersteunende video's bij het boek Procreate 5 Handleiding - tekenen op de ipad (Stefan de Groot, 2020)
Lifting and shifting SSIS packages to Azure Data Factory (ADF) can provide several benefits. By moving your on-premises SSIS workloads to Azure, you can reduce operational costs and the burden of managing infrastructure that you have when you run SSIS on-premises or on Azure virtual machines.
You can also increase high availability with the ability to specify multiple nodes per cluster, as well as the high availability features of Azure and of Azure SQL Database. You can also increase scalability with the ability to specify multiple cores per node (scale up) and multiple nodes per cluster (scale out) - see Lift and shift SQL Server Integration Services workloads to the cloud
To lift and shift SSIS packages to ADF, you can use the SSIS Integration Runtime (IR) in ADF. The Azure SSIS-IR is a cluster of virtual machines for executing SSIS packages. You can define the number of cores and compute capacity during the initial configuration (Lift and shift SSIS packages using Azure Data Factory on SQLHack)
Even though there is Microsoft article which explains how to Configure the Azure-SSIS integration runtime for high performance, there is not a lot of guidance of how to run it at the lowest possible cost but still being able to complete the jobs. So would you recommend a higher sizing running on a single node or running a lower sizing on multiple nodes? Based on experience, it seems perfectly possible to run most jobs on a single node and up until now we have been running all of them on a D4_v3, 4 cores, 16GB Standard. If you decide to run it on a lower configuration, it would recommend monitoring failures, capacity usage and throughput. (See Monitor integration runtime in Azure Data Factory for more details)
Reference:
2023 Release Wave 2
Technical topics (Configuration, customization and extensibility)
Copilots, AI and machine learning
Topics for Dynamics 365 Business Applications Platform consultants, project managers and power users
Azure Synapse Link for Dataverse allows you to easily export data from a Dataverse (or Dynamics 365) instance to Azure Data Lake Storage Gen2 (ADLS) and/or Azure Synapse. Azure Synapse Link for Dataverse provides a continuous replication of standard and custom entities/tables to Azure Synapse and Azure Data Lake.
I highly recommend you to view the awesome YouTube playlist Azure Synapse Link and Dataverse - better together from Scott Sewell (@Scottsewell) as an introduction.
1. Check the region of your Dataverse/Dynamics 365 instance
The configuration of Azure Synapse Link for Dataverse is done through the Power Platform maker portal but before you can get started you should first setup Azure Data Lake Storage Gen2 and Azure Synapse in your Azure subscription.
It is however best that you first check in the configuration screen in which region your instance is located since the storage account and Synapse Workspace must be created in the same region as the Power Apps environment for which you want to enable Azure Synapse Link. From the PPAC user interface it is currently not possible to create a Dataverse/Dynamics 365 instance in a specific region but this is possible with the PowerShell - see Creating a Dataverse instance in a specific Azure region using Power Apps Admin PowerShell module.
If you need to move a Dataverse or Dynamics 365 instance to a different Azure region, you can open a Microsoft support tickets. Based on recent experience this specific type of Microsoft support request is handled fairly quickly (within 1-2 business days).
Azure Data Lake Storage is a set of capabilities, built on Azure Blob Storage. When you create a storage account and check the "enable hierarchical namespace" checkbox on the advanced tab, you create an Azure Data Lake Storage Gen2.
2. Make sure all prerequisites are in place before enabling Azure Synapse Link
Definitely make sure that all security configuration outlined on Create an Azure Synapse Link for Dataverse with your Azure Synapse Workspace (Microsoft docs) are correctly setup. The exception messages which are shown in the Azure Synapse Link configuration pages aren't always very helpful.
3. Azure Synapse Link for Dataverse is a Lake Database
In the documentation from Microsoft (Understand lake database concepts) a lake database is defined as:
A lake database provides a relational metadata layer over one or more files in a data lake. You can create a lake database that includes definitions for tables, including column names and data types as well as relationships between primary and foreign key columns. The tables reference files in the data lake, enabling you to apply relational semantics to working with the data and querying it using SQL. However, the storage of the data files is decoupled from the database schema; enabling more flexibility than a relational database system typically offers.
The data is stored ADLS Gen2 in accordance with the Common Data Model (CDM) -the folders used conform to well-defined and standardized metadata structures (mapped 1:1 with Dataverse tables/entities). At the root you will see a metadata file (called model.json) which contains semantic information about all of the entity/table records, attributes and relationships between the tables/entities.
The way the files are being written depends on the Azure Synapse Link for Dataverse configuration - both the partitioning mode and in place vs append only mode can be configured - see Advanced Configuration Options in Azure Synapse Link
4. Synapse Link for Dataverse uses passthrough authentication using ACLs in Azure Data Lake - no support for SQL authentication
Since all the the data for the tables in Azure Synapse Link for Dataverse are CSV files which are stored in Azure Data Lake Storage, this also means that security needs to be set at the level of the files in Azure Data Lake Storage Gen2. There is no support for SQL authentication in the Lake DB which is created by Azure Synapse Link for Dataverse.
References:
The Azure Synapse Link for Dataverse documentation contains a section about Access near real-time data and read-only snapshot data but it does not really explain why you want to use one or the other.
When you open an Azure Synapse SQL Serverless LakeDB in SQL Server Management Studio you see a clear distinction between the two versions of the table data - whereas in Azure Synapse Studio there is no obvious distinction besides the name you will see the "account" table the "account_partitioned" view:
In most scenarios, it best to do queries against these partitioned views since you will avoid read conflicts and you are sure that a full transaction has been written on the CSV files in Azure Data Lake storage.
A typical exception that you might receive when doing queries directly against the "tables" is "https://`[datalakestoragegen2name].dfs.core.windows.net$$/[lakedbname]/[tablename/Snapshot/2023-05_1684234580/2023-05.csv" does not exist or you don't have file access rights)" but this also depends on your specific context. If you have a lot of create, updates or deletes on Dataverse tables this might happen more regularly. Even though, the partitioned views are update on an hourly basis - it might be that the Synapse Link engine is just refreshing the views at the same point that you perform a query, which will give you a similar exception but the changes that this occurs are more rare.
You can check the last sync timestamp and sync status in the Power Platform maker portal (see screenshot below)
For the moment, you will also have to manually check the monitoring page (which can be quite tedious if you have a lot of environments) but there is an item in the Microsoft release planner "Receive notifications about the state of Azure Synapse Link for Dataverse" which is apparently in public preview but I haven't seen it in for environments (not in the https://make.powerapps.com and also not in https://make.preview.powerapps.com/) I have access to.
It is also not easy to see if something went wrong with the refresh of the partitioned views - up until now the easiest way to find out is running a SQL query - select name,create_date from sys.views order by create_date desc against the LakeDB.
Procreate Beginners video series
The Beginners Series is a four-part guide to Procreate, the award-winning digital art app for iPad. Ideal for people new to Procreate, and with plenty of extra tips for advanced artists.
SQL Server Management Studio 19.x and higher now allows you to login to SQL using Azure Active Directory application ids and secrets - nice improvement and a reason for me to upgrade.
A couple of months ago Microsoft activated Automatic deletion of inactive Microsoft Dataverse environments. To avoid that an environment gets deleted, sign in to the Power Platform Admin Center (PPAC) and select the environment. On the environment page, select Trigger environment activity.
When designing a solution using Dataverse or Dynamics 365 CE - Dataverse storage is rarely one of the hot topics and probably most of the times overlooked. In this post I will dive a little deeper into Dataverse storage architecture and why it is important to discuss this during a Dynamics 365 CE or Dataverse implementation
Fundamentals of Dataverse storage architecture
Since April 2019, both Dataverse and Dynamics 365 CE (Online) use a tiered storage model. This means that different data types in Dataverse are stored in the most optimal storage type. Azure files and blobs are used for attachments, relational data (tables) is stored in Azure SQL, audit logs are stored in Azure Cosmos DB, search is powered by Azure Cognitive Services, etc ....
From an end user perspective, this is completely transparent and administrators don't need to manage all of these underlying Azure components since Microsoft takes care of all of this.
Microsoft has however different pricing schema's for these underlying storage types and the price difference is quite significant. You can follow up on the storage capacity in the Power Platform Admin Center. Based on the number of licenses inside your tenant, you will get a specific capacity entitlement for database, file and logs. (See New Microsoft Dataverse storage capacity > verifying your new storage model for more details)
Understanding the cost impact
Database storage in Dataverse comes at a premium price tag. Usually you don't notice this unless you will need to buy additional storage (see What Dataverse capacity is included with the Power Apps and Power Automate plans? for how much additional storage you get with additional licenses).
(1) Pricing based on the list price shown in my personal tenant - August 2023. Prices may vary depending on your licensing agreement. If you buy additional eligible licenses you will also get additional storage allocated, it is also possible to use Power Platform Pay-as-you-go plans instead of buying license but the capacity pricing using PAYG is even higher.
You need to buy additional storage capacity if you are over the allocated storage capacity since storage capacity is being enforced - if you exceed storage capacity you will not be able to create a new environment (requires a minimum of 1 GB capacity) but also copy, restore and recovery operations will be blocked.
Archiving and data retention policies
Besides the impact on your budget of retaining all data in Dataverse (or Dynamics 365) forever - you however also need to consider the potential security and legal risk. Best practices dictate that data should only be kept as long as it is useful or as long as you are legally allowed to retain it - GDPR mandates to define specific data retention period's for personal data.
Defining a data retention policy helps businesses reduce legal risks, security threats and also reduce costs. Data retention policies contain the data retention period and the required actions to take when this period lapses. You should have a data retention period defined for each of entities/tables in use for your Dataverse environment.
In July 2023 finally released Dataverse long term data retention overview in public preview. This feature allows you to create a view on Dataverse tables/entities for data that you need to retain for a longer period. The view is used as a selection criteria to define which data is moved to long term archive storage in a Microsoft managed data lake. With this feature, you might be able to use out of the box functionality instead of having to built your own custom solution.. This functionality is still in preview and should not be used on production - also keep in mind that no pricing details are available yet (they will be announced around the GA timeframe). In a upcoming post I will delve a little deeper into the archive/long term data retention functionality but you can already take a look at Early dive into Dataverse Long Term Retention
Related posts and references:
