ActivityParty not supported in Azure Data Factory copy activity for Dynamics 365

I must admit that I am not a frequent user of Azure Data Factory but now and then I give it a try again ( see Getting started with Azure Data Factory for Dynamics 365 CRM consultants for my previous post on the topic) when I need to import data into Dataverse/Dynamics 365 CE or export data from Dataverse/Dynamics 365. Usually I do this when I hit a snag with the out-of-the box tooling or when I need to connect to a legacy on-premise Dynamics CRM environment.

Last week I tried doing a bulk export of incoming e-mails in Dynamics 365 CE to Azure Blob Storage using the copy activity. I quickly created setup a sink and source, created by my  FetchXML query and then run the data pipeline. It immediately blew up with an exception message "ErrorCode=DataTypeNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column: from,The data type Microsoft.Xrm.Sdk.EntityCollection is not supported.,Source=,'

When looking at the mapping, I found the exception so I thought let's take a look at the blog posts of @Joejgriffin - Sink limitations with the Dynamics 365 Customer Engagement/CDS connector for Azure Data Factory seemed to be provide some good pointers but unfortunately did not get my any further so I just decided to write a console app ...

Analyze model-driven apps and Microsoft Dataverse telemetry with Application Insights

Beginning of May 2022, Microsoft announced that the ability to configure an Azure Application Insights resource to receive telemetry on diagnostics and performance of Dynamics 365/Dataverse model driven apps was generally available. 

Integration of Dynamics 365/Dataverse with Application Insights enables new monitoring strategies for Dynamics 365/Dataverse. Application Insights will allow you to detect and resolve issues before the end user notices it.  I also think this is very useful for following up on performance impact during big data loads/migrations (see screenshot below where Dataverse processed 6.1 million requests). Application Insights integration is a nice addition to the already extensive toolkit as outlined in the Monitoring the Power Platform blog series by @pfedynamics

The logs in Application Insights allow you to build queries to troubleshoot and monitor your solutions and answer questions like:

• Why are some of my users experiencing slowness on Unified Interface?
• How can I determine whether my plug-in upgrade caused a performance degradation?
• Can I drill down on errors or failures at specific times for specific users to allow for understanding the call stack?
• Why are users experiencing issues with a specific form?

If you want to learn more and see some examples - definitely take a look at the blog post from @decastroallan on Analyzing your Dataverse environment using Application Insights 

The performance insights (preview) for model-driven apps is probably leveraging the same telemetry but since you now have access to the raw data in the underlying logs it will be easier to pin point potential issues or discover ways to improve the performance and/or user experience. 

KQL (Kusto Query Language) can be used to query the logs in Application Insights in a scalable fashion  - KQL can be used to also query other Azure components like Azure Log Analytics, Microsoft Defender and Azure Data Explorer. KQL is a SQL-like query language which is powered by the Kusto Engine that allows you to query, filter, sort and aggregate data. It was built specifically built for the cloud and scales quite well. Unlike SQL, KQL can not create, update or delete data - it is purely meant to be used for query operations.

References:

• Analyzing your Dataverse environment using Application Insights by @decastroallan
• What are performance insights (preview)
• Tutorial: Use Kusto queries
• Querying telemetries from multiple Application Insights instances
• What is Azure Data Explorer and Kusto Query Language
• Learning the Kusto Query Language (KQL) with Application Insight Logs
• Introducing Performance Insights (preview) for model-driven apps
• Microsoft Learn - write your first query with Kusto Query Language

Using Python and Pandas Datareader to retrieve financial data - part 3: Yahoo Finance

Yahoo Finance is one of the most popular sources of free financial data. It does not only contain historical data but also financial statements, dividend information and calculated metrics like e.g. 50 and 200 day moving average, beta, etc ... Yahoo Finance does not have an officially supported API anymore but pandas-datareader  still allows you to access the data from Yahoo Finance in Python (other alternatives are yfinance and yahoo_fin).

This post is part of a series on using Pandas datareader to retrieve financial data:

• Using Python and Pandas Datareader to retrieve financial data - part 1: Federal Reserve Data (FRED)
• Using Python and Pandas Datareader to retrieve financial data - part 2: Fama & French library
• Using Python and Pandas Datareader to retrieve financial data - part 3: Yahoo Finance

In this post I have used version 0.10.0 of pandas-datareader  (released July 13, 2021) which is currently working with Yahoo Finance - previous versions of pandas-datareader had to be updated after Yahoo made some changes on the underlying API.

Warning: Accessing Yahoo Finance using Python libraries is quite brittle so don't try to built production trading systems using this data source.

Accessing the Yahoo Finance API using pandas-datareader is very simple as shown in the screenshot below but I would also recommend implementing a cache mechanism for your queries using the requests-cache Python library to avoid having your IP address being banned. The full source of this Jupyter notebook is available at https://github.com/jorisp/tradingnotebooks/blob/master/YahooFinancesingle.ipynb

References:

• Yahoo Finance API - A complete guide
• yfinance 
• yahoo_fin

Optionset support in the Power BI Dataverse connector

One of the gotchas of using Power BI to build reports for Dynamics 365 CE (or Dataverse) has been how to easily handle optionset columns (or choice columns as they are now called in the new Dataverse terminology). You had to use work arounds to display the optionset labels as explained in Using Power BI Desktop to build Dynamics CRM Online reports part 4 (article dating back to 2015).

As outlined in Create a Power BI report using data from Dataverse - choice columns are now handled out of the box e.g. in the account table you have a choice (optionset) column called "industrycode" - in Power BI you will now see two columns - "industrycode" (the integer value) and "industrycodename" (the friendly display name of the item)

Also consider the warnings outlined on the official Microsoft documentation though - filtering by choice value can have a performance impact and you also need to make sure that all customizations in Dataverse have been published.

Related links:

• Evaluating Power BI Desktop Connector Options for Microsoft Dataverse / Dynamics 365 Customer Engagement

Dynamics 365 and Power Platform monthly reading list May 2022

 

Technical topics (Configuration, customization and extensibility)

• Use custom connectors with the timeline control (Microsoft docs)
• Power Platform | Crawl, run & build by @Carsten_Groth
• Power CAT code components - a set of Power Apps Component Framework (PCF) controls that can be used to enhance Power Apps
• Configuring Sales Acceleration Reporting (Preview) feature in Dynamics 365 CE
• Setting up Azure Synapse Link for Dynamics 365/Dataverse
• Omnichannel for Customer Service - Outbound Voice Calling
• Azure Data Factory - SQL Server to Dataverse - Power CAT Live
• Notifications in Model Driven Apps - Power CAT Live
• Inside  Dataverse Data Storage - Power CAT Live
• Customer Service Historical Analytics Report in Dynamics 365 Customer Service - Part 2
• Announcing new data sources for the Virtual Connection Provider
• What you might not know about FetchXML Builder by @CarinaMClaesson
• Dataverse ServiceClient - expected 1.0 release in June 2022  with support for .NET Core 
• Microsoft Dynamics 365 - Success by Design implementation guide - new version released
• Latest innovations to scale automation with Microsoft Power Automate (Build 2022 announcement)
• Meet the developers combining Power Platform and Azure to go faster
• Interact with Dynamics 365 records in Teams Chat (Preview) by @D365Goddess
• EnableFolding option for Dataverse Native SQL Power Queries
• Analyze model-driven apps and Microsoft Dataverse telemetry with Application Insights (General Available)
• In Power BI, filter a Dataverse source to create a "My" records query
• Configuring  timeline (show/hide) activity display information in Dynamics 365 CE
• The benefits of using Dynamics 365 Customer Insights
• How to efficiently collaborate using built-in Microsoft Teams Chat in Dynamics 365 Sales
• Custom pages: a step towards disappearing lines between Canvas Apps and model driven apps in Power Platform/Dynamics 365 by @inogic
• Top 5 Power Platform features from Microsoft Build 2022 by @IFainberg
• Implement Power Platform Landing Zones and Enterprise Fusion Deployment Patterns (YouTube)
• Dataverse Advanced Audit Log Deletion using the BulkDelete API

Topics for Dynamics 365 Business Applications Platform consultants, project managers and power users

• Dynamics 365 Sales Premium features available for enterprise customers (D365 Fasttrack Architecture Insights recording)
• Accelerate business outcomes with Microsoft Teams meeting integration for Dynamics 365 Sales
• Get 360 degrees  view of your customer using Dynamics 365 Customer Insights
• Let's talk about funnels by @stevemordue
• What's new: the evolution of Power Virtual Agents (Build 2022) and  What's new for Microsoft's Conversational AI and Power Virtual Agents (Build 2022)
• Resolve complex cases faster using customer support swarming in Dynamics 365 Customer Service (YouTube)
• Exciting new capabilities for AI Builder (Build 2022)
• How to foster customer loyalty through personalized experiences
• Dynamics 365 Sales release wave 1: simplifying the things that matter
• Announcing the preview of Microsoft Power Pages