The answer to every licensing question is 42

The answer to life, the universe and everything else is 42 (check out The Hitchhiker's Guide to the Galaxy if you want to know where this comes from), so when a customer asks a question about Dynamics 365 and Power Platform licensing, I would also like to just be able to answer 42.

Having worked with Microsoft products for over 20 years now,  the single constant seems to be that Microsoft licensing is in a continuous state of flux. This is especially true for cloud platforms so  make sure that you involve the necessary experts  to help you in making the correct decisions and keep you up to date. 

In this blog post I am trying to shed some light on some interesting licensing updates for Microsoft Dynamics 365 and the Power Platform. I will also provide  some links to excellent posts/videos from other community members - this is by no means an exhaustive list.

Power Platform and Dynamics 365 request limits and allocations

I recently wrote a blog post about the Update to Power Platform and Dynamics 365 request limits and allocations . Microsoft has finally released a preview of the Power Platform requests reporting in May 2022. 

As noted in the official docs - What are the timelines for Power Platform Request limits? - enforcement of these limits will be activated at least 6 months after reports have been made GA.  In my opinion, things like requests limits and allocations are quite hard for customers to understand and also difficult to design for.

Pay as you go licensing for Power Platform

In March 2022, pay-as-you-go licensing for Power Platform reached general availability. With apay-as-you-go plan, customers can use an Azure subscription to pay for Power Apps based on active usage of the apps instead of purchasing and allocating licenses in advance. If you compare the unit pricing you will see that for similar workloads - the pay-as-you-go licensing price per unit is higher. So this will make an interesting balancing act for customers.

Scenarios combining Microsoft Teams and Dynamics 365

Don't get me wrong; I am really excited about the integration scenarios between Microsoft Teams and Dynamics 365 but announcements like Bring Dynamics 365 into the flow of work with Teams - at no additional cost generate a lot of excitement with customers but once you try to implement it you will notice that users do need a Dynamics 365 team member license. Be cautious and take big announcements with a grain of salt and always validate them yourself. When a user has a valid Dynamics 365 license, you are now able to view and edit Dynamics 365 records inside Microsoft Teams. 

Integration scenarios with Microsoft Teams are a great feature but the obscurity of Microsoft licensing around this, makes the discussion for consultants harder than needed. 

References:

• Power Platform licensing revisited
• Power Platform licensing, November 2021 updates
• PowerApps Pay As You Go licensing model - compare and configure (Youtube) from @ShaneYoung
•  Is Dynamics 365 data now "free" in Microsoft Teams by @Jukkan

Using Python and Pandas Datareader to retrieve financial data - part 2: Fama & French data library

This post is part of a series on using Pandas datareader to retrieve financial data:

• Using Python and Pandas Datareader to retrieve financial data - part 1: Federal Reserve Data (FRED)
• Using Python and Pandas Datareader to retrieve financial data - part 2: Fama & French library
• Using Python and Pandas Datareader to retrieve financial data - part 3: Yahoo Finance

In this post we will look at the datasets made available by Eugene Fama and Kenneth French. Eugene Fama and Kenneth French did a lot of research on which factors drive security returns. In 1993, they published the Three Factor Model (see article "Common risk factors in returns of stocks and bonds", Journal of Financial Economics 33, 1993), which showed that their factors (size of the firm, book-to-market values and excess return) capture a statistically significant fraction of the variation of stock returns. In 2014, Fama and French adapted their model to include five factors.  Fama won the Nobel Prize for Economics in 2013 for his research. Fama also published a number of papers on the Efficient Market Hypothesis and random walk theory.

Fama and French still publish the returns of various investment factors analyzed by them on their homepage on a regular basis.  You can download this data using the pandas_datareader library - you can take a look at the official documentation,  Fama-French Data (Ken French's Data library) to get started or take a look at the Jupyter notebook that I shared on Github https://github.com/jorisp/tradingnotebooks/blob/master/FAMA.ipynb

References:

• Fama/French forum ideas and observations from Fama and French
• How to use the Fama French model
• Fama-French 5-factor model: five major concerns
• In pursuit of the perfect portfolio: Eugene F. Fama (YouTube - MIT Laboratory for Financial Engineering )

Large AsyncOperationBase increase in Dataverse/Dynamics 365 CE: the canary in the coalmine

In normal circumstances, you probably rarely look at the System Job views in your  Dynamics 365 environment. But if you see a sudden and unexpected (large) increase in the storage used by the AsyncOperationBase table or AsyncOperation in File storage is like a canary in the coalmine - something is wrong in your Dynamics365/Dataverse environment.

System jobs (or Async Operations) are tasks that Dynamics performs in the background that don't have to be completed straight away. When such an asynchronous task is being setup, Dynamics creates a row in the AsyncOperationBase table and puts in all the details about when to run, what function/plugin/cloud flow to call, parameters to pass etc... 

All of this information used to be stored in columns of the AsyncOperationBase table but a row could  get really large (KB or MBs) depending on the actual task being scheduled.  Part of this payload has now been moved out of the database and stored into files on Azure Blob Storage and the database row now has a reference to the blob file.  You can follow up on the numbers in the capacity details for an instance in the Power Platform Admin Center (PPAC) - where you will now also see AsyncOperation in the file section after this architectural change from Microsoft. (Also see Storage capacity management for Dynamics 365/Dataverse - how to track storage evolution with Power BI and New Microsoft Dataverse storage capacity (Microsoft docs))

Dynamics 365 has a recurring system job to delete successfully completed system jobs  - see screenshot below - when an asyncoperation record is cleared from the database, the linked file storage is also cleared up.

One of the reasons you might see a large increase in storage consumed is because of a large amount of failed or cancelled system jobs.  The system jobs status (internal name statecode, possible values = ready, suspended, locked and completed)  and status reason ( internal name statuscode, possible values for completed jobs = succeeded, cancelled and failed)  can be used to identify possible issues.

I really like the SQL 4 CDS XrmToolBox add-in to troubleshoot issues with system jobs - it allows you to write simple SQL queries to identify system jobs causing issues. 

I also use  SQL4CDS for further analysis when troubleshooting async plugins using SQL queries e.g. select top 100 * from plugintracelog where typename like 'JOPX.CRM.Plugins.Notification%' but there are also other tools in XrmToolBox which you can use

References:

• Cleaning up records from the AsyncOperationBase/WorkflowLogBase tables
• asyncoperation EnityType (Microsoft Docs)
• Microsoft CRM System Job Views explained
• Asynchronous operation states (Microsoft Docs)
• SQL 4 CDS 6.1 released by Mark Carrington
• Dynamics CRM Data cleaning
• Storage capacity management for Dynamics 365/Dataverse - how to track storage evolution with Power BI

Using Python and Pandas Datareader to retrieve financial data - part 1: Federal Reserve Data (FRED)

The pandas-datareader Python library covers a number of APIs with global fundamental macro- and industry data sources including the following (for a full list see  Pandas Datareader - data sources ):

• St. Louis FED (FRED): Federal Reserve data on the U.S. economy and financial markets
• Fama/French data library : market data on portfolios capturing returns on key risk factors like size, value, and momentum by industry
• Yahoo Finance : retrieve daily stock prices, historical corporate actions (dividends and stock splits) from Yahoo Finance 
• World Bank: global database with economic/social indicators and demographics.

This post is part of a series on using Pandas datareader to retrieve financial data:

• Using Python and Pandas Datareader to retrieve financial data - part 1: Federal Reserve Data (FRED)
• Using Python and Pandas Datareader to retrieve financial data - part2: Fama & French library
• Using Python and Pandas Datareader to retrieve financial data - part 3: Yahoo Finance

In this post I will focus on retrieving data from FRED using pandas-datareader.  Federal Reserve Economic Data (FRED) - https://fred.stlouisfed.org/  is a database maintained by the Federal Reserve Bank of St. Louis. It has more than 800.000 data time series covering categories such as Economic growth & employment, monetary & fiscal policy, demographics, industries, commodity prices at different frequencies (daily, monthly, annual).  One of the interesting time series you can find here are 3-month Treasury Bill Secondary Market rate (TB3MS) or 1-year US Treasury bills which are used a proxy for the risk free rate in financial modeling. 

There is however some missing data on the TB1YR - so I will be using the TB3MS (3 Month) in my next example. You will notice that all time series are identified by a short abbreviation that you can find by searching on the FRED website.

Full source code is available at https://github.com/jorisp/tradingnotebooks/blob/master/FRED.ipynb 

References:

• FRED API: get US economic data which drives the real estate market
• Pandas-datareader python library

Documenting Power Automate Flows with Flow to Visio XrmToolBox Add-on

Last year I needed to document some rather complex Power Automate cloud flows and that's when I stumbled upon this great tool from Carl Cookson (aka @LinkeD365): the Flow to Visio XrmToolBox add-on. This XrmToolBox addon - XrmToolBox is the Swiss army knife for CRM  and Power Platform) consultants, check out https://www.xrmtoolbox.com/ for more info -  allows you to generate a Visio diagram for Power Automate Flows and Azure Logic Apps. Being able to generate Visio documents for Power Automate Flows is a great way to automatically generate a starting point for documentation. 

When you connect to a Dataverse instance, it will show you all of the different cloud flows. You can select one or more flows and simply click "Create Visio".

What I especially like about this tool that since the documentation is generated in Visio you can still clean up the output or remove parts of the flow in the Visio document which don't require explanation.

Note: You also might want to take a look at PowerDocu which contains 2 console apps you can use to document Power Automate Flows as well as Power Apps Canvas Apps

References:

• Flow to Visio XrmToolBox Add-on blogpost with basic documentation
• FlowToVisio - github repo with the source code

Dynamics 365 and Power Platform monthly reading list April 2022

 

2022 Release Wave 1

• Duplicate detection for leads (Preview) by @D365Goddess
• Updates to the Power Platform Admin Center
• D365 Customer Service: Swarming using Teams (Preview)
• The new AADUser Virtual Table (Youtube)
• React (virtual) and Fluent UI code components using platform libraries

Technical topics (Configuration, customization and extensibility)

• Enhanced user experience with new modern control on subject field
• Customizing the side panel in the deal manager workspace
• Disable hyperlinks for lookup columns in grid views of model driven apps by @linnzawwin
• Dataverse's planet scale infrastructure by Power CAT Live (Youtube) by @topness
• Power Platform and Dynamics 365 Technical Update Briefing - March 2022 from @PFEDynamics
• Dataverse: improve CRUD performance using partition key
• Setup Postman to connect to Dynamics 365 CRM using oAuth 2.0 | Azure App Registration by @priyesh_wngman7
• Integration patterns in Dynamics 365 Finance & Operations
• ALM Power Platform Part 1 - Overview and Process
• Solution deployments: automated vs manual
• The curious case of the missing application user view (Dynamics 365/Dataverse)
• Install Power Apps for Windows (preview) - run model-driven or canvas apps on your Windows device.

Topics for Dynamics 365 Business Applications Platform consultants, project managers and power users

• What KPIs should service-based businesses monitor to ensure lasting success
• Evaluating software with Microsoft's Catalyst Framework
• The limits of legacy campaign management tools
• 4 ways to accelerate revenue with Dynamics 365 Sales and AI
• Customer case: Siemen's bold choice to pursue CRM transformation leads to unique collaborative results
• Power Platform - rise of the business technologists

Reading and writing files in Azure Blob Storage with Python

Azure Blob storage is Microsoft's object storage solution for the cloud and allows you to store massive amounts of unstructured data, such as text or binary data at low cost for every scale. If you are not familiar with it, I can recommend taking a look at the Store data in Azure learning path on Microsoft Learn

Using Python in combination with Azure Blob Storage is quite easy using the azure-storage-blob client library for Python . You can set up a container with private access meaning that you will need to provide credentials to access the containers and the blobs contained within. The easiest way to do this is using a shared access signature (SAS) token. You can generate a SAS token from the Azure Portal.

To interact with the different parts of Azure  Blob Storage you will typically use the BlobServiceClient to work with the Azure storage account itself, the ContainerClient to work with a specific container and the BlobClient to work with a specific blob.  Below is the sample code which uses these different clients in a Jupyter notebook (based on Quickstart: Manage blobs with Python v12 SDK)   - you can find the full Jupyter notebook at tradingnotebooks/AzureBlobStorage.ipynb at master · jorisp/tradingnotebooks (github.com) 

 References:

• Quickstart: Manage blobs with Python v12 SDK
• Using your data lake as a cheap time series database: do's and don'ts
• How to download blobs from Azure Storage using Python - sample code for multithreading

Recreating an Azure Synapse Link for Dataverse connection

If you encounter an exception during the initial setup of Azure Synapse Link for Dataverse, it is best that you check in Azure Synapse workspace whether the lake database was only partially created. When you want to retry the configuration, you will first need to remove the lake database. 

Previously you had to manually write a script but Microsoft has now added a handy delete button which will generate an Azure Synapse Analytics notebook for you. To be able to run the script you will however need to setup a serverless Apache Spark pool. 

The smallest default configuration (4vCores/32GB) is sufficient to run this notebook - double check the pause settings of the Spark pool after the initial setup or just delete the pool if you don't to expect to need it anymore afterwards to save costs. 

Other blog posts on Azure Synapse and Dataverse:

• Getting started with Azure Synapse Analytics on demand webinars

Storage capacity management for Dynamics 365/Dataverse - how to track storage evolution with Power BI

After the introduction of the new storage management model in Dataverse in April 2019, both Dataverse and Dynamics CRM Online are using a tiered storage model (separation between database capacity, file capacity and log capacity). 

Based on the number of licenses acquired, every customer has a specific storage capacity entitlement - you can find an overview of storage consumption within the Power Platform Admin Center - section 1 shows an overview of all storage types consumed across all environments (support and trial instances are excluded), section 2 shows the allocated capacity based on number of licenses or purchased additional capacity and section 3 shows the biggest environments. 

To see where Dataverse storage is being used across your environments, you need to click on the second tab. From this page you are also able to download a CSV file with storage allocations for all environments to which you have access. The information on this page is updated every 24 hours and unfortunately no historical data is available so it is quite hard to track storage evolution.

I currently track the evolution of storage by downloading this CSV file and storing the exported file on SharePoint - then it is quite easy to use Power BI to combine the CSV files and create your own Power BI report and dashboard based on the storage details. I added a screenshot of one of the report pages below, for those of you interested in this Power BI report send me a DM on @jopxtwits 

References:

• Storage capacity enforcement for CDS instances (June 2020)
• Microsoft Dataverse data archival (2021 Wave2) - public preview initially expected in March 2022 but postponed to a later date.
• Dynamics CRM Data Cleaning
• Administer Power Platform - Free up storage space [Microsoft docs]
• Administer Power Platform - New Microsoft Dataverse storage capacity [Microsoft docs]
• Dynamics 365 Storage Utilization | Dataverse storage | Power Platform Admin Center

Dynamics 365 and Power Platform monthly reading list February 2022

2022 Release Wave 1

• 2022 release wave 1 plans for Dynamics 365 and Power Platform now available
• 2022 Release Wave 1 - First impressions by @CarinaMClaesson
• 2022 Release Wave 1 - Top 10 in 10 minutes (Youtube) by @LisaCrosbie
• Co-presence notification now in Preview for Dataverse and Dynamics CRM
• Release 2022 Wave 1 Summary by @BergmannBene
• Modern advanced find with enhanced view management in model-driven apps
• Modern advanced find test drive by @jukkan

Technical topics (Configuration, customization and extensibility)

• Use of new configuration page UI in Power Apps solution
• Using Entity Extraction AI model with Dynamics 365 CRM/Power Apps - Part 2
• Using PCF Gallery Code components in your environment by @CRMHeidi
• Service tags now available for Power Platform
• Dynamics 365 FastTrack Architecture Insights : Power Platform & Dataverse security strategy best practices by @HenryJammes
• Dynamics 365 FastTrack Architecture Insights: Create real-time and contextualized Power BI reports in your apps by @HenryJammes
• How to free up file usage - Dataverse Storage Capacity (Dynamics 365/CRM)
• Connections and connection references explained

Topics for Dynamics 365 Business Applications Platform consultants, project managers and power users

• Building big apps in small steps with low-code
• Premium auto capture in Dynamics 365 CRM
• Zero-party data is more valuable than ever for customer experiences
• Dynamics 365 composable approach enables rapid deployment of a supply chain control tower
• 5 ways CRM can help you succeed in a customer experience economy
• Dynamics 365 drives improvement in manufacturing supply chains
• 3 characteristics to look for in a Dynamics 365 migration partner
• Use a performance dashboard to optimize seller activities
• Achieving on-going leadership buy-in for Dynamics 365 projects
• 10 benefits of using Sales Accelerator in Dynamics 365 Sales
• Achieving successful B2B selling in the digital era
  
• Customer case: Rabobank selects Power Platform as its default development platform for internal process automation
• Improve seller productivity with a sales cadence

Dataverse environmentid vs organizationid

Coming from a Dynamics CRM background, I mainly used the organizationid in the past (the organizationid is the unique identifier associated to a CRM instance). So having to use both organizationids and enviromentids isn't always easy,  and sometimes I still mix the two  - readability isn't a trait of GUIDS ... 

First some background on why we have two identifiers. With the July 2018 update of Dynamics CRM (version 9.0) Microsoft separated out the core platform features and the various business modules (Sales, Customer Service, etc...)  and this was also the point in time where CRM consultants where suddenly faced with a second identifier for a Dynamics CRM instance - the environmentid.

To find both the environmentid and organizationid of an instance, you open up the maker portal and select the developer resources link in the settings menu for an instance.

From this page you can find both the environmentid and organizationid - the Microsoft documentation still points to the legacy interface - Determine your environment's organization ID and name which starting from the maker portal requires quite a few additional steps ...

You will need to use the environmentid when working with the Power Platform Admin PowerShell cmdlets, Power Platform for Admins Cloud flow connector as well as the Power Platform API

References:

• Making sense of XRM, PowerApps and some other acronyms (CDS,CDM, ...)

Quick note - refer to previous row in Power BI

 A number of good tips in this video on how to refer to a previous row in Power BI - I am not a Power BI expert but it got the job done - I just wanted to visualize Dataverse storage growth from one day to the next.

The grey area shows the forecast for the storage increase/decrease - for more info see Use the analytics pane in Power BI and  Introducing new forecasting capabilities in Power BI for Office 365 (dated but still relevant).

Azure DevOps - historical graph for agent pools (Preview)

If you want to catch up with all features/improvements released  on Azure DevOps - you can take a look at the Azure DevOps Feature timeline. One of the cool features which went in preview in December 2020 is the historical graph for agent pools (Preview) - which is extremely useful if you are working with a large team of developers and you need to know whether you need to add additional build agents. Microsoft recently updated the documentation on Historical graph for agent pools (Preview) | Microsoft Docs .

References:

• Parallel jobs in Azure DevOps
• Historical graph for agent pools (Preview) | Microsoft Docs
• Implement a pipeline strategy (Microsoft Learn)

Update to Power Platform and Dynamics 365 request limits and allocations

Microsoft updated the Power Platform and Dynamics 365 request limits and allocations (also referred to as entitlement limits) end of last year and upped the allocated limits significantly, which is great news. At the same time Microsoft also announced their planning on enforcement of the API request limits and availability of reports to track the number of API calls at tenant level (reports expected in first quarter of 2022 - long awaited since it first announced in 2019) 

All table/entity CRUD operations - both calls that originate from internal and external sources will count towards a tenant level allocation of requests in a 24 hour window. If you go above this allocated limit you are expected to buy additional API licensing packs. 

Especially external integrations and/or business logic (even plugin executions count against these limits) might cause spikes in API calls but so far not a lot of attention was paid to it especially since the limits are currently not enforced. 

These integrations typically use non-licensed users ( e.g. application users, non-interactive users, administrative users or System) also consume allocated API requests and this is where you will need to pay more attention - especially in environments with relative high integration volume and low volume of fully licensed users.

So let's take an example to illustrate the increases in default allocated API capacity - suppose you expect 1 million API calls in a 24 hour period in your tenant and you have 100 users with Dynamics 365 licenses in your environment - then you either need to purchase a number of licensing packs - list price +/- 50€ per month per pack (but I recommend reaching out to your licensing partner to get an up to date quote) or use the new Pay as You Go subscription model which is currently in preview. 

The available capacity has been upped quite significantly with allocated base capacity now at 500.000 API calls (previously 100.000 API calls) and an additional 5.000 calls per licensed Dynamics 365 user (see Non-licensed user request limits  for details and exceptions). Also the number of API calls per licensing pack has increased to 50.000 (previously 10.000)

The list price of the licensing pack which I found seems to be 50€ per month per pack (I would recommend reaching out to your licensing partner to get an exact price) which seems to be similar to the price in the past if my memory serves my As part of the pay as you go model there will be a separate meter available - see Pay-as-you-go meters - Power Platform requests.

Currently you can only view the number API requests in the Power Platform Admin Center (PPAC) on a per environment basis in the dashboard for Dataverse API call statistics - if you can't wait for the new reporting to become available I would recommend starting from this one to get an indication of number of API calls.

References:

• What is a Microsoft Power Platform request?
• Non-licensed user request limits
• Power Apps and Power Automate capacity add-on
• Optimizing Power Platform requests by @timopertila
• New entitlement limits announced by @crmgustaf
• My previous blogposts on Power Platform and Dynamics 365 API request entitlements (February 2021) and Update on Dynamics 365/CDS request limits (January 2020)
• Power Automate Flow API Updates - 16 takeaways

Dynamics 365 and Power Platform monthly reading list January 2022

2021 Retrospective

• Looking back at 2021 - Power Platform Edition by @carinamclaesson
• Dynamics 365 & Power Platform 2021: Top 10 in 10 minutes by @lisamcrosbie
• Year 2021 in Power Platform by @jukkan

Technical topics (Configuration, customization and extensibility)

• Create mobile apps with Power Apps (preview)
• Large migrations to Dynamics 365 - lessons learned by @crmgustaf 
• Incorrect  (Dataverse) API optimizations recommendations by Microsoft
• What's new and planned in Customer Service 2021 Wave 2 
• Round up of the Ignite announcements for Power Apps!
• Preview : Using Microsoft Teams chat in Dynamics 365
• Exclusive sneak peek at the new multi-tab Panel (side panes)
• Load data in Dynamics 365 using Azure Copy Data Activity tool
• PowerFx Kernel for .NET Interactive Notebook and Playwright Dataverse Notebook by @RajYRaman  - definitely something on my todo list to take a look at.
• Tools to help you be a super system admin: XrmToolBox
• Analyzing your Dataverse environments using Application Insights
• Where should I put my data in Azure?
• Dataverse plugin telemetry with Azure Application Insights
• Power Apps and delegation - don't trust solely on the yellow triangle
• Using entity extraction AI model within Dynamics 365 CRM/Power Apps - Part 2
• Typescript for Dataverse mentoring - November 2021 playlist (Youtube) by @DanzMaverick

Topics for Dynamics 365 Business Applications Platform consultants, project managers and power users

• Tag incoming emails with AI Builder
• Ad-hoc Power BI reporting in Dynamics 365 (Preview) and Visualize data in model-driven apps with Power BI (Preview) | Power Apps Blog - you can also provide feedback on Power BI quick reports in model-driven Apps
• Revolutionizing the future for sellers in Dynamics 365 Sales
• Introducing Microsoft Customer Experience Platform: a solution for connected customer experiences
• Dynamics 365 and Microsoft Teams - the blueprint for cross-organization collaboration
• Enhancement in Email Template
• Achieving personalized engagement in B2B organizations
• Microsoft Ignite recap: Dynamics 365 news and announcements you may have missed
• Microsoft Teams - embedded chat in Dynamics 365 Sales/Customer Service
• Microsoft named a leader in 2021 Gartner Magic Quadrant for CRM Customer Engagement Center
• Microsoft Mechanics - Power Apps Updates | Pay-as-you-go model, standalone mobile app packages & Azure Integration - interview with Charles Lamanna (Youtube)
• Announcements for Power Apps at Ignite recording by @ClayBWesener
• Power Platform | Raise of the Citizen Data Analyst
• 6 ways AI-based personalization is improving the customer experience

Notes and links for the AZ-900 Azure fundamentals exam

As a Microsoft CRM technical consultant/developer/architect you should have at least a basic understanding of Azure Cloud and how you can combine Dynamics 365 CE and Azure. 

Although I have been using Azure Cloud together with multiple Dynamics 365 Customer Engagement projects since 2015 I thought it was a good idea to check my overall understanding by doing the AZ-900 exam (which I passed). The exam does not cover any topics in depth but studying for it gives you a good overview and can be used as a starting point to delve deeper in specific topics;

Listed below are some of the links and resources that I used - mainly on topics where I had less hands-on experience from projects so this is definitely not an exhaustive list. 

General training material to get started

• Microsoft Certified: Azure Fundamentals - start with the learning paths from Microsoft Learn on this page
• Azure in bullet points - AZ-900 Microsoft Azure Fundamentals
• Exam AZ-900: Microsoft Azure Fundamentals (Video) on O'Reilly Learning (subscription required) - 5 hours of training material
• Exam AZ-900 Microsoft Azure Fundamentals Study Guide - YouTube playlist by @TechTrainerTim
• Azure Tips & Tricks - great overall resource on Azure platform - probably to overwhelming as a starting point for AZ-900 but handy starting point if you are stuck on a specific topic.
• Learn Azure in a Month of Lunches (Audiobook)

Architectural components

• Availability zones fundamentals: regions and availability zones
• Subscriptions, licenses, accounts and tenants for Microsoft's cloud offering
• Azure subscription - account administrators
• Azure Resource Groups
• Azure global infrastructure - Azure geographies

Core resources

• Storage account overview
• Data redundancy - Azure storage redundancy
• App Service pricing and App Service limits
• Azure Site Recovery
• What is Azure backup service?
• What is Azure Sphere?
• What is Azure IOT Central?
• IoT concepts and Azure IoT Hub
• What is Azure Virtual Desktop?
• Maintenance for virtual machines in Azure
• Azure Disk Encryption for Windows VMs
• Choose an Azure Compute service for your applications

Management tools

• How to install the Azure CLI  and Install the Azure Az PowerShell module - both install on Windows, macOS, Linux (Debian, Ubuntu,etc...)

Security and identity management

• What is Azure Active Directory?
• What is single sign-on in Azure Active Directory?
• What is Identity Protection?
• What is Microsoft Defender for Identity formerly Advanced Threat Protection (ATP)

Networking

• Azure Network security groups
• Azure Security Center (name changed to Microsoft Defender for Cloud) - secure your management ports with just-in-time-access
• What is Microsoft Defender for Cloud (formerly known as Azure Security Center)?
• How network security groups filter network traffic
• What is Azure Firewall
• Tutorial: Create a Site-to-Site connection in the Azure portal
• Connect an on-premises network to a Microsoft Azure virtual network
• What is VPN Gateway? There are different configurations available for VPN gateway connections - see Planning table to determine which  to choose
• What is Traffic Manager?
• Tutorial: filter inbound internet traffic with Azure Firewall policy DNAT using the Azure portal
•  Azure DDoS protection - designing resilient solutions
• What is defense in depth?

Governance and compliance

• What is Azure policy?
• Quickstart: create a policy assignment to identify non-compliant resources
• What is Microsoft Compliance Manager?
• Microsoft Privacy Statement, Online Service Terms and Data Protection Addendum 
• What is Azure Blueprints?
• What is Azure Advisor? and Reduce service costs by using Azure Advisor
• What is Azure Cost Management + Billing?
• Azure support plans 
• Lock resources to prevent unexpected changes
• What is the Microsoft Cloud Adoption Framework for Azure?

Virtual tables in Microsoft Dataverse and Dynamics 365 Customer Engagement

Virtual entities in Dynamics CRM have been around for more than 4 years and were introduced with the 9.x release of Dynamics CRM Online. Virtual tables (entities) allow you to integrate external data into Dataverse without having the replicate the data or without having to write client side script to visualize the data. Virtual tables provide an abstraction layer which allow for real-time on-demand access to data wherever it resides, for end users it will look as if the data resides in Dataverse.

To be honest, I hadn't paid a lot of attention to virtual entities due to its limitations and due to the fact that there weren't  a lot of improvements from Microsoft in it before 2021 - some of these limitations of virtual tables still exists like only organization owned tables, no auditing or change tracking, no queue support, a virtual table cannot represent an activity, etc...  But a lot has changed in the last 12 months - for example CRUD support for virtual table providers went GA in March 2021 so I decided to take a look at it in the last couple of days.  Another trigger was a blog post from @jukkan on Virtual Dataverse tables with no code, via Connectors which I marked already marked for review a couple of months ago.

Virtual Table Data Providers act as the glue between the external data and your virtual entities in Dataverse.  There are currently  4 different flavors of the Virtual Table Data Providers: oData, custom-built using plugins, Cosmos DB and using the Power Platform Connector for SQL with a number of them still in preview.

Dataverse ships with an OOTB OData v4 Data Provider but I personally haven't seen any practical use case for this specific connector. So let's take a look at the current status of the 3 other data providers.

Custom Virtual Data Provider

This is really a pro-developer story, .NET developers have the option of creating custom virtual table data providers to help integrate external data source types that are not supported by an existing data provider. Sample: Custom virtual table provider with CRUD operations contains some the sample code for the plugins that you need to implement as well how to register these in Datavers but I found the walkthrough - Custom Virtual Table provider with CRUD operations: Introduction and scenario - part 1 of 6 from @capuanodanilo a lot more helpful since it outlines all of the different steps in a more elaborate manner.

Azure Cosmos DB SQL API Data Provider

The Azure Cosmos DB Data Provider has been in preview for quite a while already and you will need to install this from AppSource - Azure Cosmos DB for DocumentDB API Data Provider - based on a recent blog post from @temmy_raharjo  -  [Virtual Entity - Preview] Setup Azure Cosmos DB SQL API Data Provider - the current release seems to be working again (I got stuck with the setup but will add more details when I find time to get it working)

Virtual Connector Data Provider

The Virtual Connector Provider (VCP) for tabular connectors entered public preview in August 2021 (no GA date mentioned). The VCP preview currently supports only SQL Server (both on-premise and on-premise) but the release plan mentions that it will  allow you to use Microsoft Dataverse virtual tables using multiple supported tabular connectors (e.g. SQL Server, OneDrive, Excel, SharePoint, etc...).  Similar to the Cosmos DB Data Provider you will first need to install the solution from AppSource.

I got everything working using the walkthrough  mentioned in Create virtual tables using the virtual connector provider (preview) but overall the configuration experience is still confusing because you have to switch between the new and old solution designer experience (I think it even is safest to just do everything using the old solution designer). 

I also noticed that when you use Azure SQL with the Virtual Connector Provider you might need to add the IP addresses from the Dataverse back-end servers to the Azure SQL whitelist - maybe because my Azure SQL is in a different tenant/subscription then the Dataverse environment. 

After completing the configuration, the virtual tables initially did not work and when looking at the plugin trace logs - I noticed below exception. When adding both IP addresses (probably 2 back-end servers) to Azure SQL - everything started working.

Even though the article Create virtual tables using the virtual connector provider (preview) mentions that you can make the configuration solution aware - I think you first need to test everything especially if you want to automate solution deployment and have specific naming conventions around entities/tables.

The fact that you need to create a connection with all users in your organization is also something that Microsoft will change - a concern which is also shared in this blog post. Dataverse virtual connector provider for SQL Server. Awesome? which also provides some interesting workarounds. I however think that is should be best that in a standard config - only the service principal that you used in the initial configuration should be given access to the SQL without  a need to explicitly share the connection with everyone in the organization.

Summary

Virtual tables indeed show great promise but the Virtual Connector Provider is still a little bit to rough on the edges to use with confidence. As we near GA, I hope additional improvements will be made available especially around the security aspect. 

References:

• Enable Virtual Tables to support Dataverse events
• Custom Virtual Data Provider
• Preview feature: Azure Cosmos DB SQL API Data Provider requirements
• Create virtual tables using the virtual connector provider (preview)