tag:blogger.com,1999:blog-77535772024-03-16T01:01:52.055+01:00JOPX on Microsoft Business Applications and Azure CloudOccasional rantings about Dynamics CRM/365, Power BI and Azure cloud. Taking the first small steps in machine learning, Python and algorithmic tradingjopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.comBlogger1499125tag:blogger.com,1999:blog-7753577.post-54611693887139913072024-02-23T05:43:00.002+01:002024-02-23T05:43:40.087+01:00SQL Server Integration Services Project template available for Visual Studio 2022<p> Since end of 2022, there is also a <a href="https://marketplace.visualstudio.com/items?itemName=SSIS.MicrosoftDataToolsIntegrationServices" target="_blank">SQL Server Integration Services Project template available for Visual Studio 2022</a> which you can install from the Visual Studio Marketplace. You can install it from the direct download link <a href="https://marketplace.visualstudio.com/items?itemName=SSIS.MicrosoftDataToolsIntegrationServices" target="_blank">here</a> or you can search for it in the Visual Studio 2022 extension manager and install it from there.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhx0EDtCv1cRap8OtQnT3mrzFaHVIniG5z6sAVtb-nR_kavWQDDi-NLGDrG1QYsKHakFS5nYsjuRNb58M9c28JRuWYfv2nFpOyxWwKRLQ6dM_wS7xgIGDC6mzK7NC5TMlIJr0JnsTlXy0wY8xr1EtFLCLMmjfC2cToMLAYWs64xnDVmK9Lb16px/s1411/ssis_vs2022.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="976" data-original-width="1411" height="442" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhx0EDtCv1cRap8OtQnT3mrzFaHVIniG5z6sAVtb-nR_kavWQDDi-NLGDrG1QYsKHakFS5nYsjuRNb58M9c28JRuWYfv2nFpOyxWwKRLQ6dM_wS7xgIGDC6mzK7NC5TMlIJr0JnsTlXy0wY8xr1EtFLCLMmjfC2cToMLAYWs64xnDVmK9Lb16px/w640-h442/ssis_vs2022.png" width="640" /></a></div><br /><p><br /></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-80715486796388260672024-02-22T18:25:00.003+01:002024-02-22T18:26:48.529+01:00Classic Azure Application Insights deprecated on February 29th 2024 - 7 days to go<p> If you missed it - classic Azure Application Insights will be deprecated on February 29th 2024. If you missed the different notification e-mails, you can quite easily see the warning if you navigate to an Azure Application Insights resource in Azure Portal.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQ5oAFL2uhDTf_L0ovEKx3hrI1Vi9VH_mOt6d3dl_CFkTY4cglxwcjbQ8jC5W3XR42E9QtD4Z3_fBDlDNnOE9N-I8O5dvFeeG7LUGxovXlHmJQip1SOiRzyDjkpIIgceLCwUtErMRyAgPBGTlNJvMlmuPSUKrt-FOzsdSrk-4-cjZPI64HxpJQ/s970/azure_applicationinsight1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="705" data-original-width="970" height="466" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQ5oAFL2uhDTf_L0ovEKx3hrI1Vi9VH_mOt6d3dl_CFkTY4cglxwcjbQ8jC5W3XR42E9QtD4Z3_fBDlDNnOE9N-I8O5dvFeeG7LUGxovXlHmJQip1SOiRzyDjkpIIgceLCwUtErMRyAgPBGTlNJvMlmuPSUKrt-FOzsdSrk-4-cjZPI64HxpJQ/w640-h466/azure_applicationinsight1.png" width="640" /></a></div><br /><p>Migration is actually quite easy - you just click on the link provided and this will open up the menu depicted below which allows you to associate your Azure Application Insights resource to a Log Analytics Workspace. The good news is that there are no pricing changes when moving to the workspace-based model. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZPVQ3ynu-4OMjXdQMo7wZTtr8wCyZIPPO9bB0BRyuum1FfvjwZpKDR_EbQt2s7ewq8YnErW9Ehoam1AyB7uAKi1kH9RT0U9jytsUYYx7FIBgjW1Y8tyvT1yPkB0x-rdM_VPrP28DPEt8aqE27NkZvjFIL7VyfBwf-oLO8vP_UkdtD-G15Ao38/s846/azure_applicationinsights2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="534" data-original-width="846" height="404" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZPVQ3ynu-4OMjXdQMo7wZTtr8wCyZIPPO9bB0BRyuum1FfvjwZpKDR_EbQt2s7ewq8YnErW9Ehoam1AyB7uAKi1kH9RT0U9jytsUYYx7FIBgjW1Y8tyvT1yPkB0x-rdM_VPrP28DPEt8aqE27NkZvjFIL7VyfBwf-oLO8vP_UkdtD-G15Ao38/w640-h404/azure_applicationinsights2.png" width="640" /></a></div><br /><p><br /></p><p><br /></p><p>As indicated in the migration window, this is a one way operation so plan for it in advance - the points below might impact on how you will do the migration:</p><p></p><ul style="text-align: left;"><li>You can link different Application Insight resources to a single Log Analytics workspace or you can make the split - in most case you want to consolidate it.</li><li>Instrumentation keys do not change during the migration so you don't need to worry about this</li><li>The export feature is not available on the Application Insights workspace-based resources - you need to look at diagnostic settings for exporting telemetry</li><li>There might be some schema changes - important to consider when doing KQL queries - check out <a href="https://learn.microsoft.com/en-gb/azure/azure-monitor/logs/cross-workspace-query#query-across-log-analytics-workspaces-using-workspace" target="_blank">query data across Log Analytics workspaces, applications and resources in Azure Monitor</a></li><li>Existing log data will not immediately move to the Log Analytics workspace - only new logs generated after the migration will be stored in the new log location.</li></ul><p></p><p><br /></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-26214894098965444332024-01-14T10:54:00.002+01:002024-01-14T10:54:52.619+01:00Procreate 5 handleiding video overzicht<p>Lijst van ondersteunende video's bij het boek <a href="https://www.bol.com/be/nl/p/procreate-5-handleiding/9300000009688468/?suggestionType=featured_product" target="_blank">Procreate 5 Handleiding - tekenen op de ipad (Stefan de Groot, 2020)</a></p><p><br /></p><p></p><ul style="text-align: left;"><li><a href="https://vimeo.com/450416767/2a64b13f4c" target="_blank">Hoofdstuk 2 - Gallery</a></li><li><a href="https://vimeo.com/450425204/4e70deb580" target="_blank">Hoofdstuk 3 - Canvas & brushes</a></li><li><a href="https://vimeo.com/450443909/7ecd67f0e9" target="_blank">Hoofdstuk 4 - Teken, veeg, gum en vormen</a></li><li><a href="https://vimeo.com/450517834/cfbe5f3346" target="_blank">Hoofdstuk 5 - Layers</a></li><li><a href="https://vimeo.com/450558156/0d1281c6f1" target="_blank">Hoofdstuk 6 - Kleuren</a></li><li><a href="https://vimeo.com/451154727/9eea048f86" target="_blank">Hoofdstuk 7 - Gebaren</a></li><li><a href="https://vimeo.com/450818773/b0e8669536" target="_blank">Hoofdstuk 8 - Actions</a></li><li><a href="https://vimeo.com/451100317/0d264e355c" target="_blank">Hoofdstuk 9 - Adjustments</a></li><li><a href="https://vimeo.com/451218204/82068b4e43" target="_blank">Hoofdstuk 10 - Selecteren en transformeren</a></li><li><a href="https://vimeo.com/451275463/48589856a5" target="_blank">Hoofdstuk 11 - Snelkoppelingen & Apple Pencil</a></li></ul><p></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-51595424966955966222023-12-26T08:47:00.004+01:002024-01-05T14:35:49.382+01:00Running SSIS packages in Azure Data Factory - scaling and monitoring<p>Lifting and shifting SSIS packages to Azure Data Factory (ADF) can provide several benefits. By moving your on-premises SSIS workloads to Azure, you can reduce operational costs and the burden of managing infrastructure that you have when you run SSIS on-premises or on Azure virtual machines. </p><p>You can also increase high availability with the ability to specify multiple nodes per cluster, as well as the high availability features of Azure and of Azure SQL Database. You can also increase scalability with the ability to specify multiple cores per node (scale up) and multiple nodes per cluster (scale out) - see <a href="https://learn.microsoft.com/en-us/sql/integration-services/lift-shift/ssis-azure-lift-shift-ssis-packages-overview?view=sql-server-ver16" target="_blank">Lift and shift SQL Server Integration Services workloads to the cloud</a></p><p>To lift and shift SSIS packages to ADF, you can use the SSIS Integration Runtime (IR) in ADF. The Azure SSIS-IR is a cluster of virtual machines for executing SSIS packages. You can define the number of cores and compute capacity during the initial configuration (<a href="https://www.sqlshack.com/lift-and-shift-ssis-packages-using-azure-data-factory-v2/" target="_blank">Lift and shift SSIS packages using Azure Data Factory on SQLHack</a>)</p><p>Even though there is Microsoft article which explains how to <a href="https://learn.microsoft.com/en-us/azure/data-factory/configure-azure-ssis-integration-runtime-performance" target="_blank">Configure the Azure-SSIS integration runtime for high performance</a>, there is not a lot of guidance of how to run it at the lowest possible cost but still being able to complete the jobs. So would you recommend a higher sizing running on a single node or running a lower sizing on multiple nodes? Based on experience, it seems perfectly possible to run most jobs on a single node and up until now we have been running all of them on a D4_v3, 4 cores, 16GB Standard. If you decide to run it on a lower configuration, it would recommend monitoring failures, capacity usage and throughput. (See <a href="https://learn.microsoft.com/en-us/azure/data-factory/monitor-integration-runtime" target="_blank">Monitor integration runtime in Azure Data Factory</a> for more details)</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrO1ZeVhF1eXKwHT2M7iPZ8X7Adf2rNEykI5b3NKRP2Ha0LfwF9GU_NTBSIeFTJSVwlkmM9WECPOXlsnX9VeC1buIPoCzNrRrlPoqeW7_iV6UjZ15WprZ6HkQJ3shlHWTT622pqUvoKihIc-Y7Cg5ORcrtylcIIGGUFSWPsUsozS_3tX-xWnYU/s626/azure_ssis_scaling.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="588" data-original-width="626" height="602" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrO1ZeVhF1eXKwHT2M7iPZ8X7Adf2rNEykI5b3NKRP2Ha0LfwF9GU_NTBSIeFTJSVwlkmM9WECPOXlsnX9VeC1buIPoCzNrRrlPoqeW7_iV6UjZ15WprZ6HkQJ3shlHWTT622pqUvoKihIc-Y7Cg5ORcrtylcIIGGUFSWPsUsozS_3tX-xWnYU/w640-h602/azure_ssis_scaling.png" width="640" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><p><b>Reference:</b></p><p></p><ul style="text-align: left;"><li><a href="https://learn.microsoft.com/en-us/azure/data-factory/configure-azure-ssis-integration-runtime-performance" target="_blank">Configure the Azure-SSIS integration runtime for high performance</a> </li><li><a href="https://www.timmitchell.net/post/2020/07/16/comparing-ssis-and-azure-data-factory/" target="_blank">Comparing SSIS and Azure Data Factory</a></li></ul><p></p><p><br /></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-14246248344331740472023-11-29T14:38:00.005+01:002023-12-01T11:49:35.082+01:00Dynamics 365 and Power Platform monthly reading list November 2023<p><b> 2023 Release Wave 2</b></p><p></p><ul style="text-align: left;"><li><a href="https://d365goddess.com/d365-customer-service-new-dialer-experiencepreview/" target="_blank">Dynamics 365 Customer Service: new dialer experience (preview)</a> by <a href="https://twitter.com/D365Goddess" target="_blank">@D365Goddess</a></li><li><a href="https://cloudblogs.microsoft.com/dynamics365/bdm/2023/11/02/new-sales-capabilities-in-2023-release-wave-2-helping-sellers-work-smarter/" target="_blank">New sales capabilities in 2023 release wave 2 - helping sellers work smarter</a></li><li><a href="https://www.inogic.com/blog/2023/07/dynamics-365-crm-wave-2-release-enhance-categorization-of-knowledge-search-articles/" target="_blank">Dynamics 365 CRM Wave 2 Release: enhance categorization of knowledge search articles</a></li><li><a href="https://d365goddess.com/configure-sales-copilot-preview/" target="_blank">Configure sales Copilot for Dynamics 365 Sales (Preview)</a> by <a href="https://twitter.com/DianaBirkelbach" target="_blank">@DianaBirkelbach</a></li></ul><p></p><p><b>Technical topics (Configuration, customization and extensibility)</b></p><p></p><ul style="text-align: left;"><li><a href="https://learn.microsoft.com/en-gb/power-apps/developer/data-platform/bulk-operations?tabs=sdk" target="_blank">Dataverse - use bulk operation messages (MS Learn) - </a> CreateMultiple and UpdateMultiple are now GA! Read the small print to see when to use it</li><li><a href="https://temmyraharjo.wordpress.com/2023/11/19/dataverse-lets-try-elastic-table-preview/" target="_blank">Dataverse let's try elastic tables (preview)</a> - by <a href="https://twitter.com/temmy_raharjo" target="_blank">@temmy_raharjo</a></li><li><a href="https://www.inogic.com/blog/2022/12/ways-to-deal-with-missing-power-platform-environments/" target="_blank">Ways to deal with missing Power Platform environments </a> by <a href="https://www.twitter.com/inogic" target="_blank">@inogic</a></li><li><a href="https://dianabirkelbach.wordpress.com/2023/08/05/edit-subgrids-side-by-side-with-power-apps-grid-or-editable-grid/" target="_blank">Edit subgrids side by side with Power Apps Grid or editable grid</a> by <a href="https://twitter.com/DianaBirkelbach" target="_blank">@DianaBirkelbach</a></li><li><a href="https://powerapps.microsoft.com/en-us/blog/announcing-monthly-channel-for-model-driven-apps/" target="_blank">Announcing monthly channel for model-driven apps</a></li><li><a href="https://powerapps.microsoft.com/en-us/blog/announcing-general-availability-of-custom-connectors-in-solutions-as-well-as-environment-variable-secrets/" target="_blank">Announcing general availability of custom connectors in solutions as well as environment variable secrets</a></li><li><a href="https://techcommunity.microsoft.com/t5/sharepoint-premium-blog/announcing-sharepoint-embedded-public-preview-at-espc23/ba-p/3993428" target="_blank">Announcing SharePoint Embedded Public Preview at ESPC23</a> - will be interesting to explore whether it is possible to combine this with Dynamics 365 data as an alternative for Power Pages. Head over to http://aka.ms/start-spe/ to start building your first SharePoint Embedded app. </li><li><a href="https://temmyraharjo.wordpress.com/2023/08/06/dataverse-azure-service-bus-queue-azure-function-for-processing-long-operation/" target="_blank">Dataverse + Azure Service Bus queue + Azure function for processing long operations</a></li><li><a href="https://www.inogic.com/blog/2023/06/workflow-automation-in-dynamics-365-crm-triggering-actions-on-email-send-and-receive-events/" target="_blank">Workflow automation in Dynamics 365 CRM: triggering actions on email send and receive events</a> by <a href="https://www.twitter.com/inogic" target="_blank">@inogic</a></li><li><a href="https://powerapps.microsoft.com/en-us/blog/august-2023-updates-for-modernization-and-theming-in-power-apps/" target="_blank">August 2023 updates for modernization and theming in Power Apps</a></li><li><a href="https://dreamingincrm.com/2023/08/08/connecting-to-dataverse-from-function-app-using-managed-identity-using-azd/" target="_blank">Connecting to Dataverse from Function App using Managed Identity - using azd</a></li><li><a href="https://community.fabric.microsoft.com/t5/Community-Blog/Mastering-Sales-Calculations-A-Comprehensive-Guide-to/ba-p/3388308" target="_blank">Power BI - Mastering sales calculations: a comprehensive guide to departmental analysis </a></li><li><a href="https://nishantrana.me/2023/10/18/how-to-enable-the-enhanced-email-template-editor-in-model-driven-appsdynamics-365-dataverse/" target="_blank">How to enable the enhanced email template editor in model-driven apps (Dynamics 365/Dataverse)</a></li></ul><p></p><p><b>Copilots, AI and machine learning</b></p><p></p><ul style="text-align: left;"><li><a href="https://nishantrana.me/2023/11/21/write-an-email-copilot-in-dynamics-365-customer-service/" target="_blank">Write an email with Copilot in Dynamics 365 Customer Service</a> by <a href="https://twitter.com/nishantranaCRM" target="_blank">@nishantranacrm</a></li><li><a href="https://ignite.microsoft.com/en-US/sessions/8598e920-afb0-4527-af55-544a61df73b5" target="_blank">Getting your enterprise ready for Microsoft 365 Copilot (Ignite 2023 recording)</a></li><li><a href="https://ignite.microsoft.com/en-US/sessions/964e12b9-6991-40a3-af8c-c355994f47cd?source=/schedule" target="_blank">Learn live: prepare, implement and secure Microsoft 365 Copilot (Ignite 2023 recording)</a></li><li><a href="https://blogs.microsoft.com/blog/2023/11/02/new-study-validates-the-business-value-and-opportunity-of-ai/" target="_blank">New study validates the business value and opportunity of AI</a></li><li><a href="https://learn.microsoft.com/en-gb/security/zero-trust/zero-trust-tech-illus#zero-trust-for-microsoft-365-copilot" target="_blank">Architecture and deployment diagrams for Microsoft 365 Copilot</a></li><li><a href="https://cloudblogs.microsoft.com/dynamics365/it/2023/08/29/analyze-the-impact-of-ai-enhanced-customer-service-with-copilot-analytics/" target="_blank">Analyse the impact of AI-enhanced customer service with Copilot analytics</a></li><li><a href="https://www.mckinsey.com/featured-insights/mckinsey-explainers/what-is-prompt-engineering" target="_blank">What is prompt engineering? (McKinsey)</a></li><li><a href="https://gist.github.com/veekaybee/be375ab33085102f9027853128dc5f0e" target="_blank">Anti-hype LLM reading list</a></li><li><a href="https://jussiroine.com/2023/09/microsoft-chat-copilot-vs-azure-chatgpt-which-generative-ai-capability-to-choose-for-the-enterprise/" target="_blank">Microsoft Chat Copilot vs Azure ChatGPT - which generative AI capability to choose for the enterprise</a></li><li><a href="https://www.nber.org/digest/20236/measuring-productivity-impact-generative-ai" target="_blank">Measuring the productivity impact of Generative AI (NBER digest)</a></li><li><a href="https://www.microsoft.com/en-us/research/blog/the-power-of-prompting/" target="_blank">The power of prompting (Microsoft Research)</a></li><li><a href="https://www.youtube.com/watch?v=frIA3Wof08Y" target="_blank">Get started with Microsoft Copilot Studio: how to create your first Copilot</a> by <a href="https://twitter.com/LisaCrosbie" target="_blank">@lisacrosbie</a></li><li><a href="https://nishantrana.me/2023/11/15/use-copilot-to-summarize-cases-dynamics-365-customer-service" target="_blank">Use Copilot to summarize cases - Dynamics 365 Customer Service</a></li></ul><p></p><p><b>Topics for Dynamics 365 Business Applications Platform consultants, project managers and power users</b></p><p></p><ul style="text-align: left;"><li><a href="https://msdynamicsworld.com/story/microsoft-make-dynamics-365-marketing-part-d365-customer-insights" target="_blank">Microsoft to make Dynamics 365 Marketing a part of D365 Customer Insights</a></li><li><a href="https://triciasinclair.com/2023/08/18/customer-service-in-the-age-of-ai/" target="_blank">Customer Service in the age of AI</a></li><li><a href="https://cloudblogs.microsoft.com/dynamics365/it/2023/11/13/refresh-the-sales-experience-with-dynamics-365-sales-modern-update/" target="_blank">Refresh the sales experience with Dynamics 365 Sales modern update</a></li></ul><div><br /></div><p></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-13600935041499475102023-11-26T16:00:00.005+01:002023-12-01T13:48:06.319+01:00Implementing Azure Synapse Link for Dataverse: gotchas and tips<p><a href="https://learn.microsoft.com/en-us/power-apps/maker/data-platform/export-to-data-lake" target="_blank">Azure Synapse Link for Dataverse</a> allows you to easily export data from a Dataverse (or Dynamics 365) instance to Azure Data Lake Storage Gen2 (ADLS) and/or Azure Synapse. Azure Synapse Link for Dataverse provides a continuous replication of standard and custom entities/tables to Azure Synapse and Azure Data Lake. </p><p>I highly recommend you to view the awesome <a href="https://www.youtube.com/playlist?list=PLzUAjXZBFU9NucnyOxBB-bHdhfaLiVVgn" target="_blank">YouTube playlist Azure Synapse Link and Dataverse - better together </a>from Scott Sewell (<a href="https://twitter.com/scottsewell" target="_blank">@Scottsewell</a>) as an introduction.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfHGwfSj2N2nLzgYGDNnfVBqsAY65GYuhTxpzMsCBswAGQmUDmCuOXxfrIsAwQ6h1Ec0iJXBiBtQLVh07MPYr4fss7nL5fSyJg86J8v2ipk3YU9izDmRzhQ6ZK_Jz6JDkaOE6NUKqKSfiWmuY3i68C257tM1P6c25BRffyxXkFsL3mhjQahw/s1266/synapselink_architecture.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="756" data-original-width="1266" height="382" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfHGwfSj2N2nLzgYGDNnfVBqsAY65GYuhTxpzMsCBswAGQmUDmCuOXxfrIsAwQ6h1Ec0iJXBiBtQLVh07MPYr4fss7nL5fSyJg86J8v2ipk3YU9izDmRzhQ6ZK_Jz6JDkaOE6NUKqKSfiWmuY3i68C257tM1P6c25BRffyxXkFsL3mhjQahw/w640-h382/synapselink_architecture.png" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">This blog post provides a number of tips & tricks but is not an exhaustive list - it is highly recommended to go through the links in the Microsoft documentation listed in the reference section below. You can also take a look at the presentation I delivered at Techorama in May 2023 which is available on <a href="https://github.com/jorisp/presentations/blob/main/Techorama_SynapseLinkDataverse_20230517.pdf" target="_blank">Github - Azure Synapse Link for Dataverse from 0 to 100</a></div><br /><p><b>1. Check the region of your Dataverse/Dynamics 365 instance</b></p><p>The configuration of Azure Synapse Link for Dataverse is done through the Power Platform maker portal but before you can get started you should first setup Azure Data Lake Storage Gen2 and Azure Synapse in your Azure subscription. </p><p>It is however best that you first check in the configuration screen in which region your instance is located since the storage account and Synapse Workspace must be created in the same region as the Power Apps environment for which you want to enable Azure Synapse Link. From the PPAC user interface it is currently not possible to create a Dataverse/Dynamics 365 instance in a specific region but this is possible with the PowerShell - see <a href="https://jopx.blogspot.com/2022/10/creating-dataverse-instance-in-specific.html" target="_blank">Creating a Dataverse instance in a specific Azure region using Power Apps Admin PowerShell module</a>. </p><p>If you need to move a Dataverse or Dynamics 365 instance to a different Azure region, you can open a Microsoft support tickets. Based on recent experience this specific type of Microsoft support request is handled fairly quickly (within 1-2 business days).</p><p><a href="https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction" target="_blank">Azure Data Lake Storage</a> is a set of capabilities, built on Azure Blob Storage. When you create a storage account and check the "enable hierarchical namespace" checkbox on the advanced tab, you create an Azure Data Lake Storage Gen2.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNXFI0xzQNWY1td8eKj_J7EwUiBUW_dA8e0DtLsTCLlwiDFqJDkKRwi-txCCQzobrA8TziiDFABWCdjdRbM7aPZMPDjFF1AakQaNUUVLKc6XdAhq9iRIuKl33dLCCZAJHCZdHvCKcsYvRTqSkDoIu7nUzlA0-ZMQPki1A7bbV0IjEmaNsF3A/s912/1_ADLSgen2.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="912" data-original-width="851" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNXFI0xzQNWY1td8eKj_J7EwUiBUW_dA8e0DtLsTCLlwiDFqJDkKRwi-txCCQzobrA8TziiDFABWCdjdRbM7aPZMPDjFF1AakQaNUUVLKc6XdAhq9iRIuKl33dLCCZAJHCZdHvCKcsYvRTqSkDoIu7nUzlA0-ZMQPki1A7bbV0IjEmaNsF3A/w598-h640/1_ADLSgen2.PNG" width="598" /></a></div><br /><p><b>2. Make sure all prerequisites are in place before enabling Azure Synapse Link</b></p><p>Definitely make sure that all security configuration outlined on <a href="https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-synapse" target="_blank">Create an Azure Synapse Link for Dataverse with your Azure Synapse Workspace (Microsoft docs)</a> are correctly setup. The exception messages which are shown in the Azure Synapse Link configuration pages aren't always very helpful.</p><p><b>3. Azure Synapse Link for Dataverse is a Lake Database</b></p><p>In the documentation from Microsoft (<a href="https://learn.microsoft.com/en-us/training/modules/create-metadata-objects-azure-synapse-serverless-sql-pools/2-lake-database" target="_blank">Understand lake database concepts</a>) a lake database is defined as:</p><p><i>A lake database provides a relational metadata layer over one or more files in a data lake. You can create a lake database that includes definitions for tables, including column names and data types as well as relationships between primary and foreign key columns. The tables reference files in the data lake, enabling you to apply relational semantics to working with the data and querying it using SQL. However, the storage of the data files is decoupled from the database schema; enabling more flexibility than a relational database system typically offers.</i></p><p><i><br /></i></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgy4igJ5izkyKcpKxdmRDHXaNGM3avi-yYO66hgO6yjZYbRzq52vmvZw6XfLUGthxo4kJbNY4GT9SpNAHJYsV0GOd4pH0DopZwJjLzIoezSiHWon0XCrcvYnknstWLIxxABxhjxuvFyp1BYj4pvkidYabSEU0bhQNGgg1wEzZDNoL2IB4kHvN4V/s959/synapsefolderstructuure.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="829" data-original-width="959" height="554" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgy4igJ5izkyKcpKxdmRDHXaNGM3avi-yYO66hgO6yjZYbRzq52vmvZw6XfLUGthxo4kJbNY4GT9SpNAHJYsV0GOd4pH0DopZwJjLzIoezSiHWon0XCrcvYnknstWLIxxABxhjxuvFyp1BYj4pvkidYabSEU0bhQNGgg1wEzZDNoL2IB4kHvN4V/w640-h554/synapsefolderstructuure.png" width="640" /></a></div><br /><i><br /></i><p></p><p>The data is stored ADLS Gen2 in accordance with the Common Data Model (CDM) -the folders used conform to well-defined and standardized metadata structures (mapped 1:1 with Dataverse tables/entities). At the root you will see a metadata file (called model.json) which contains semantic information about all of the entity/table records, attributes and relationships between the tables/entities.</p><p>The way the files are being written depends on the Azure Synapse Link for Dataverse configuration - both the partitioning mode and in place vs append only mode can be configured - see <a href="https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-advanced-configuration" target="_blank">Advanced Configuration Options in Azure Synapse Link </a></p><p><b>4. Synapse Link for Dataverse uses passthrough authentication using ACLs in Azure Data Lake - no support for SQL authentication</b></p><p>Since all the the data for the tables in Azure Synapse Link for Dataverse are CSV files which are stored in Azure Data Lake Storage, this also means that security needs to be set at the level of the files in Azure Data Lake Storage Gen2. There is no support for SQL authentication in the Lake DB which is created by Azure Synapse Link for Dataverse.</p><div class="separator" style="clear: both; text-align: center;"><br /></div><p><b>References:</b></p><p></p><ul><li><a href="https://learn.microsoft.com/en-us/common-data-model/data-lake" target="_blank">Use the Common Data Model to optimize Azure Data Lake Storage Gen2 (Microsoft docs)</a></li><li><a href="https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-synapse" target="_blank">Create an Azure Synapse Link for Dataverse with your Azure Synapse Workspace (Microsoft docs)</a></li><li><a href="https://www.serverlesssql.com/azure-synapse-link-for-dataverse-advanced/" target="_blank">Azure Synapse Link for Dataverse - understanding advanced configuration settings (Microsoft docs)</a></li><li><a href="https://powerapps.microsoft.com/en-us/blog/do-more-with-data-from-data-export-service-to-azure-synapse-link-for-dataverse/" target="_blank">Do more with data - from Data Export Service to Azure Synapse Link for Dataverse (Microsoft docs)</a></li><li><a href="https://nishantrana.me/2020/09/07/export-data-from-common-data-service-to-azure-data-lake-storage-gen2/" target="_blank">Use Power BI to analyze the CDS data in Azure Data Lake Storage Gen2</a></li><li><a href="https://joegill.com/synapse-link-option-sets/" target="_blank">Synapse Link for Dataverse - Option Sets</a></li><li><a href="https://joegill.com/synapse-link-dataverse-metadata/" target="_blank">Dataverse metadata in Synapse</a></li><li><a href="https://tldr-dynamics.com/blog/setup-azure-synapse-for-dataverse" target="_blank">Setting up Azure Synapse Link for Dynamics 365/Dataverse</a></li><li><a href="https://learn.microsoft.com/en-us/power-bi/guidance/powerbi-modeling-guidance-for-power-platform" target="_blank">Power BI modeling guidance for Power Platform</a></li><li><a href="https://www.serverlesssql.com/azure-synapse-serverless-sql-pools-cheat-sheet/" target="_blank">Azure Synapse Serverless SQL Pools cheat sheet</a></li><li><a href="https://www.youtube.com/watch?v=AbiYgh9RD2U" target="_blank">Controlling cost with Azure Synapse Serverless Pools</a></li></ul><div></div><p></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-50811868594184459672023-11-22T21:34:00.003+01:002023-12-01T13:55:53.796+01:00Near real-time and snapshots in Azure Synapse Link for Dataverse<p>The Azure Synapse Link for Dataverse documentation contains a section about <a href="https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-synapse#access-near-real-time-data-and-read-only-snapshot-data" target="_blank">Access near real-time data and read-only snapshot data</a> but it does not really explain why you want to use one or the other. </p><p>When you open an Azure Synapse SQL Serverless LakeDB in SQL Server Management Studio you see a clear distinction between the two versions of the table data - whereas in Azure Synapse Studio there is no obvious distinction besides the name you will see the "account" table the "account_partitioned" view:</p><p></p><ul style="text-align: left;"><li><b>Near real time data: </b>external table for all the underlying CSV files exported by the Azure Synapse Lin for Dataverse sync engine. There is a soft SLA for the data to be present in these tables within 15 minutes</li><li><b>Snapshot data/partitioned views:</b> views on top of the near-real time data which are updated on an hourly interval.</li></ul><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQLMcrGy67e_JkTGjoM2MoXFMs0Il8QcpnnLY_U1sEDzTNJf0zhTfDyGS32cPtgdSbMx4zFJ-EKnvXsDyfj3G_7ZJMFpPmpgAAHSLWtHqdEdOx-CG8T-w1NiGwvkNdV4sVPuMf0_XWhwRGspJxJSLzqyVg_1js61_IH_Zuw6WgkZ4PrPW1EH2f/s758/synapse_partitioned.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="758" height="540" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQLMcrGy67e_JkTGjoM2MoXFMs0Il8QcpnnLY_U1sEDzTNJf0zhTfDyGS32cPtgdSbMx4zFJ-EKnvXsDyfj3G_7ZJMFpPmpgAAHSLWtHqdEdOx-CG8T-w1NiGwvkNdV4sVPuMf0_XWhwRGspJxJSLzqyVg_1js61_IH_Zuw6WgkZ4PrPW1EH2f/w640-h540/synapse_partitioned.png" width="640" /></a></div><br /><p><br /></p><p>In most scenarios, it best to do queries against these partitioned views since you will avoid read conflicts and you are sure that a full transaction has been written on the CSV files in Azure Data Lake storage. </p><p>A typical exception that you might receive when doing queries directly against the "tables" is "https://`[datalakestoragegen2name].dfs.core.windows.net$$/[lakedbname]/[tablename/Snapshot/2023-05_1684234580/2023-05.csv" does not exist or you don't have file access rights)" but this also depends on your specific context. If you have a lot of create, updates or deletes on Dataverse tables this might happen more regularly. Even though, the partitioned views are update on an hourly basis - it might be that the Synapse Link engine is just refreshing the views at the same point that you perform a query, which will give you a similar exception but the changes that this occurs are more rare.</p><p>You can check the last sync timestamp and sync status in the Power Platform maker portal (see screenshot below)</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXP6EL3ECzFqYrDVemdtbCDwMvxGmK8DcEpBvB-PrLVOg4EdamT7XFbtp5Z6AkfVVY6U3PjXrbBiWQyAAQSbFDsoFkI1CUdA_KGuWhW1T2IS6eGf-7-T-lRnLEDVgaWAO21L-xKoatbNitzQOm8_ViHa57CRJ85iIhkh5EVBhw7uZdhKVZiDSn/s2141/synapselink_statuspage.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="767" data-original-width="2141" height="230" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXP6EL3ECzFqYrDVemdtbCDwMvxGmK8DcEpBvB-PrLVOg4EdamT7XFbtp5Z6AkfVVY6U3PjXrbBiWQyAAQSbFDsoFkI1CUdA_KGuWhW1T2IS6eGf-7-T-lRnLEDVgaWAO21L-xKoatbNitzQOm8_ViHa57CRJ85iIhkh5EVBhw7uZdhKVZiDSn/w640-h230/synapselink_statuspage.png" width="640" /></a></div><br /><p></p><p>For the moment, you will also have to manually check the monitoring page (which can be quite tedious if you have a lot of environments) but there is an item in the Microsoft release planner <a href="https://releaseplans.microsoft.com/?app=Microsoft+Dataverse&status=new&planID=de4a1c71-8267-ee11-9ae7-000d3a574bff" target="_blank">"Receive notifications about the state of Azure Synapse Link for Dataverse"</a> which is apparently in public preview but I haven't seen it in for environments (not in the https://make.powerapps.com and also not in https://make.preview.powerapps.com/) I have access to. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhulwBAl2zOKzmMt4h9BNRBXyL-7NUnUgJbOap8kUJOdV0Xqoh71VaY0qtzOW8kAY6bCG0cs8mkkKdJNvEbnDjPiIht2g3rV9Z99bZR6MAJnKa2nk8zRYzE8ofADlY-utU0bHdkkQVZtrA_QtOkH7TKOKR0foFB4ORd6dwg130fV9CnZvG-ghLE/s1857/synapselink_notifications.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="754" data-original-width="1857" height="260" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhulwBAl2zOKzmMt4h9BNRBXyL-7NUnUgJbOap8kUJOdV0Xqoh71VaY0qtzOW8kAY6bCG0cs8mkkKdJNvEbnDjPiIht2g3rV9Z99bZR6MAJnKa2nk8zRYzE8ofADlY-utU0bHdkkQVZtrA_QtOkH7TKOKR0foFB4ORd6dwg130fV9CnZvG-ghLE/w640-h260/synapselink_notifications.png" width="640" /></a></div><br /><p><br /></p><p>It is also not easy to see if something went wrong with the refresh of the partitioned views - up until now the easiest way to find out is running a SQL query - <i> select name,create_date from sys.views order by create_date desc</i> against the LakeDB.</p><p><br /></p><p><br /></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-27455061429287389472023-11-20T10:15:00.008+01:002023-11-20T10:16:20.996+01:00Procreate video series<p> </p><p><b>Procreate Beginners video series</b></p><p>The Beginners Series is a four-part guide to Procreate, the award-winning digital art app for iPad. Ideal for people new to Procreate, and with plenty of extra tips for advanced artists.</p><p></p><ul style="text-align: left;"><li><a href="https://www.youtube.com/watch?v=Vn8bj0YpZg4" target="_blank">Procreate Beginners series: Part One | The Fundamentals (YouTube)</a></li><li><a href="https://www.youtube.com/watch?v=e5F7em-hGwc&list=PLlpSQCrjuGkriILjGVhAMxaroOgpGDbvl&index=3" target="_blank">Procreate Beginners series: Part Two | Painting Tools (YouTube)</a></li></ul><p></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-1878736169161806362023-11-17T13:04:00.004+01:002023-11-17T13:06:34.961+01:00Quick tip: SQL Server Management Studio 19 supports AAD service principal authentication<p><a href="https://learn.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms?view=sql-server-ver16" target="_blank">SQL Server Management Studio 19.x </a>and higher now allows you to login to SQL using Azure Active Directory application ids and secrets - nice improvement and a reason for me to upgrade.</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhScVwVHzskqKxYzJIO5336cz2mvVErblu_sFVLatXk_nZ-3WyWslxFOyIewGTmFB6pBrqxtpiypfnFLY1CtUbUk9baEAiYn4OxnkvyR_krbzRMMYs4ZCMWIYmyJ8OmWdMQqM-k1q0XTKtM-HTLLYSKCudcpySNT5um9x8KTavCem4pchvTD4uV/s799/sqlserver_mgtstudio.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="573" data-original-width="799" height="458" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhScVwVHzskqKxYzJIO5336cz2mvVErblu_sFVLatXk_nZ-3WyWslxFOyIewGTmFB6pBrqxtpiypfnFLY1CtUbUk9baEAiYn4OxnkvyR_krbzRMMYs4ZCMWIYmyJ8OmWdMQqM-k1q0XTKtM-HTLLYSKCudcpySNT5um9x8KTavCem4pchvTD4uV/w640-h458/sqlserver_mgtstudio.png" width="640" /></a></div><br /><p></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-61137812161211309142023-10-12T22:00:00.001+02:002023-10-13T13:33:51.626+02:00Quick tip: prevent automatic deletion of an inactive Microsoft Dataverse environment<p> A couple of months ago Microsoft activated <a href="https://learn.microsoft.com/en-us/power-platform/admin/automatic-environment-cleanup" target="_blank">Automatic deletion of inactive Microsoft Dataverse environments</a>. To avoid that an environment gets deleted, sign in to the Power Platform Admin Center (PPAC) and select the environment. On the environment page, select Trigger environment activity.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxusZYyNWoRptI4bQwTdbcRgIM7rBdr8uTlSyi022VGTcqgqnWGf0ucdKxvcfK6IDDXFHDkLvO2mo8SU5uNNtgW2Y9JwhTaroCJhyphenhyphen2w63UOFlt6fuAFaUnLRyFEyMRRhHPwq1voqLQwxxXrUWOMjh-_143fFpsoiSy5Nm0f4jyOynue7AdQ2md/s1064/inactive_environments.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="690" data-original-width="1064" height="416" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxusZYyNWoRptI4bQwTdbcRgIM7rBdr8uTlSyi022VGTcqgqnWGf0ucdKxvcfK6IDDXFHDkLvO2mo8SU5uNNtgW2Y9JwhTaroCJhyphenhyphen2w63UOFlt6fuAFaUnLRyFEyMRRhHPwq1voqLQwxxXrUWOMjh-_143fFpsoiSy5Nm0f4jyOynue7AdQ2md/w640-h416/inactive_environments.png" width="640" /></a></div><br /><p><br /></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-84702880480253276962023-09-14T18:13:00.004+02:002023-09-15T09:35:28.334+02:00We don't talk about storage<p>When designing a solution using Dataverse or Dynamics 365 CE - Dataverse storage is rarely one of the hot topics and probably most of the times overlooked. In this post I will dive a little deeper into Dataverse storage architecture and why it is important to discuss this during a Dynamics 365 CE or Dataverse implementation </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZrgRkS9j6tHt_SuX9rI56cawXG5E4eIPep0bB8dmOJX4-ZUUz_RHjRdqQn3469hTcJX5_Qzi6OlfQIefZnmfwCMxFLN4vDpkxhFRCL4g-3EcePqr9pO1FFVArKA3Qlf8zmpbxLxaOiWZH9Gm9-5OEKJ4NsFJYxSNoBav9OoAnvo4G_jvpP4Zj/s3840/wedonttalkaboutbrunopng.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="2076" data-original-width="3840" height="346" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZrgRkS9j6tHt_SuX9rI56cawXG5E4eIPep0bB8dmOJX4-ZUUz_RHjRdqQn3469hTcJX5_Qzi6OlfQIefZnmfwCMxFLN4vDpkxhFRCL4g-3EcePqr9pO1FFVArKA3Qlf8zmpbxLxaOiWZH9Gm9-5OEKJ4NsFJYxSNoBav9OoAnvo4G_jvpP4Zj/w640-h346/wedonttalkaboutbrunopng.png" width="640" /></a></div><br /><p><br /></p><p><b>Fundamentals of Dataverse storage architecture</b></p><p>Since April 2019, both Dataverse and Dynamics 365 CE (Online) use a tiered storage model. This means that different data types in Dataverse are stored in the most optimal storage type. Azure files and blobs are used for attachments, relational data (tables) is stored in Azure SQL, audit logs are stored in Azure Cosmos DB, search is powered by Azure Cognitive Services, etc .... </p><p>From an end user perspective, this is completely transparent and administrators don't need to manage all of these underlying Azure components since Microsoft takes care of all of this.</p><p>Microsoft has however different pricing schema's for these underlying storage types and the price difference is quite significant. You can follow up on the storage capacity in the Power Platform Admin Center. Based on the number of licenses inside your tenant, you will get a specific capacity entitlement for database, file and logs. (See <a href="https://learn.microsoft.com/en-us/power-platform/admin/capacity-storage#verifying-your-new-storage-model" target="_blank">New Microsoft Dataverse storage capacity > verifying your new storage model</a> for more details)</p><p><b>Understanding the cost impact</b></p><p>Database storage in Dataverse comes at a premium price tag. Usually you don't notice this unless you will need to buy additional storage (see <a href="https://learn.microsoft.com/en-us/power-platform/admin/powerapps-flow-licensing-faq#what-dataverse-capacity-is-included-with-the-power-apps-and-power-automate-plans" target="_blank">What Dataverse capacity is included with the Power Apps and Power Automate plans?</a> for how much additional storage you get with additional licenses).</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHqDIwImKOnCgFK1IqHqSeMH4jYWTbYkH_EDB_UrCbdt402bywkBCJ5YwIXVeK4XqI68ZlOrKeO8IXMxAYWa_eL3B9n0l0YPvuUI-SMKhM6zJlWnB3t-hjjrvAxh3dfVVjloKj8fYxoiqMyxQYBLFz4WfTpHCU4HVhhY4onG0FgzAWUzvRWo1q/s1302/storage_license_cost.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="463" data-original-width="1302" height="229" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHqDIwImKOnCgFK1IqHqSeMH4jYWTbYkH_EDB_UrCbdt402bywkBCJ5YwIXVeK4XqI68ZlOrKeO8IXMxAYWa_eL3B9n0l0YPvuUI-SMKhM6zJlWnB3t-hjjrvAxh3dfVVjloKj8fYxoiqMyxQYBLFz4WfTpHCU4HVhhY4onG0FgzAWUzvRWo1q/w640-h229/storage_license_cost.png" width="640" /></a></div><br /><p>(1) Pricing based on the list price shown in my personal tenant - August 2023. Prices may vary depending on your licensing agreement. If you buy additional eligible licenses you will also get additional storage allocated, it is also possible to use <a href="https://learn.microsoft.com/en-us/power-platform/admin/pay-as-you-go-overview" target="_blank">Power Platform Pay-as-you-go plans</a> instead of buying license but the capacity pricing using PAYG is even higher.</p><p>You need to buy additional storage capacity if you are over the allocated storage capacity since storage capacity is being enforced - if you exceed storage capacity you will not be able to create a new environment (requires a minimum of 1 GB capacity) but also copy, restore and recovery operations will be blocked.</p><p><b>Archiving and data retention policies</b></p><p>Besides the impact on your budget of retaining all data in Dataverse (or Dynamics 365) forever - you however also need to consider the potential security and legal risk. Best practices dictate that data should only be kept as long as it is useful or as long as you are legally allowed to retain it - GDPR mandates to define specific data retention period's for personal data.</p><p>Defining a data retention policy helps businesses reduce legal risks, security threats and also reduce costs. Data retention policies contain the data retention period and the required actions to take when this period lapses. You should have a data retention period defined for each of entities/tables in use for your Dataverse environment.</p><p>In July 2023 finally released <a href="https://learn.microsoft.com/en-us/power-apps/maker/data-platform/data-retention-overview" target="_blank">Dataverse long term data retention overview in public preview</a>. This feature allows you to create a view on Dataverse tables/entities for data that you need to retain for a longer period. The view is used as a selection criteria to define which data is moved to long term archive storage in a Microsoft managed data lake. With this feature, you might be able to use out of the box functionality instead of having to built your own custom solution.. This functionality is still in preview and should not be used on production - also keep in mind that no pricing details are available yet (they will be announced around the GA timeframe). In a upcoming post I will delve a little deeper into the archive/long term data retention functionality but you can already take a look at <a href="https://mbeard.co.uk/2023/06/19/early-dive-into-dataverse-long-term-data-retention/" target="_blank">Early dive into Dataverse Long Term Retention</a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-gnLWctCy36DzY5GlNSMxTaw5e4lTuLSWZrf5BUdyMX_DYU8tohXEVdbRfTkBG-pLiYVw3xbxURIxA2C_AI6OhnXw6zQFDDhOwN-IXM2lYOf7v-brfLesBgUTrclMbsgiHsqWGXEfD6gl0Gx_V0Adt5D-sVtr1XdR64JRnxZ7o8RbnPMxC3aQ/s928/archives_incomplete.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="492" data-original-width="928" height="340" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-gnLWctCy36DzY5GlNSMxTaw5e4lTuLSWZrf5BUdyMX_DYU8tohXEVdbRfTkBG-pLiYVw3xbxURIxA2C_AI6OhnXw6zQFDDhOwN-IXM2lYOf7v-brfLesBgUTrclMbsgiHsqWGXEfD6gl0Gx_V0Adt5D-sVtr1XdR64JRnxZ7o8RbnPMxC3aQ/w640-h340/archives_incomplete.png" width="640" /></a></div><br /><p><br /></p><p><b>Related posts and references:</b></p><p></p><ul style="text-align: left;"><li><a href="https://jopx.blogspot.com/2022/03/storage-capacity-management-in-dynamics.html" target="_blank">Storage capacity management for Dynamics 365/Dataverse - how to track storage evolution with Power BI</a></li><li><a href="https://www.youtube.com/watch?v=bSscrGddctI" target="_blank">Inside Dataverse Data Storage -Power CAT Live YouTube video</a></li><li><a href="http://jopx.blogspot.com/2023/06/cost-optimization-older-mails-in.html" target="_blank">Cost optimization: older mails in Dataverse now stored in Dataverse file storage</a></li><li><a href="https://jopx.blogspot.com/2022/03/storage-capacity-management-in-dynamics.html" target="_blank">Storage capacity management for Dynamics 365/Dataverse - how to track storage evolution with Power BI</a></li></ul><div><br /></div><p></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-37791069324867260452023-08-07T21:10:00.006+02:002023-08-07T21:12:35.053+02:00Reducing size of the SubscriptionTrackingDeletedObject in Dynamics 365 CE or Dataverse<p>When you delete a large amount of data from Dynamics 365 CE or Dataverse, you will notice that the total storage consumed does not decrease. This is because the SubscriptionTrackingDeletedObject table stores deleted records from the Dynamics 365 database. This table is used to track deleted records for replication and synchronization purposes and records are being kept by default for quite a large amount of time.</p><p>When recently encountering this problem, Microsoft support suggested us to reduce the number of days the records are kept in the table from the default 90 to 15 days - to do this take the following steps:</p><p></p><ul style="text-align: left;"><li>Install the latest version of the OrganizationSettingsEditor solution which you can download from <a href="https://github.com/seanmcne/OrgDbOrgSettings/releases">https://github.com/seanmcne/OrgDbOrgSettings/releases</a> (The OrgDBOrgSettingsTool dates back to Dynamics CRM 2011 but is being maintained by Sean McNellis and is still quite relevant)</li></ul><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh77Ph11qpGGjp57vBqXL3QGh9LVAbAjC-COdsu-Eg0qQWS2VnV5y8lJ1Fvfv3-rJ02yziL_ss8e1AEtP_uNtYsc4VtYf00YjMS2Bf-JZ0KX3NxZChYNVwhBABUyFjjU28LpZhEZjlj7XboAQhd-AZqn8VlC6pacvZtXAjNAMRQRoWdi_mBrrs0/s861/organizationsettingseditor.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="861" data-original-width="769" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh77Ph11qpGGjp57vBqXL3QGh9LVAbAjC-COdsu-Eg0qQWS2VnV5y8lJ1Fvfv3-rJ02yziL_ss8e1AEtP_uNtYsc4VtYf00YjMS2Bf-JZ0KX3NxZChYNVwhBABUyFjjU28LpZhEZjlj7XboAQhd-AZqn8VlC6pacvZtXAjNAMRQRoWdi_mBrrs0/w572-h640/organizationsettingseditor.png" width="572" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><ul style="text-align: left;"><li>Change the ExpireChangeTrackingInDays and ExpireSubscriptionsInDays from the default setting of 90 days to 15 days. In sandbox environments, you might be able to change this value to 0 but for production environments Microsoft recommended to keep it on a minimum of 15 days. The OrgDBSettings utility utilizes the solution configuration page to provide access to the editor (in classic mode) - <a href="https://github.com/seanmcne/OrgDbOrgSettings#where-to-i-find-it-after-installing" target="_blank">see OrgDBOrgSettings - where to find it after installing?</a> for details.</li></ul><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEif364Lv0PLP8TekhydpPTYw_QKurjlCn8FB9G3kWVzju2iBkVjpMN8QILwInLF4CFvfb9kqYggaRZlXK0WBDINTLbL1GQ5L_rZzbYuGbzGgY5-P8TEvF8kkPMWoKLTAIlZW_ARVuXTHyoC_OqsWyNhhJ0f_1p0ABPEslhCw86HF3WPVGpRusXJ/s1319/organizationsettingseditor2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="737" data-original-width="1319" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEif364Lv0PLP8TekhydpPTYw_QKurjlCn8FB9G3kWVzju2iBkVjpMN8QILwInLF4CFvfb9kqYggaRZlXK0WBDINTLbL1GQ5L_rZzbYuGbzGgY5-P8TEvF8kkPMWoKLTAIlZW_ARVuXTHyoC_OqsWyNhhJ0f_1p0ABPEslhCw86HF3WPVGpRusXJ/w640-h358/organizationsettingseditor2.png" width="640" /></a></div><br /><div><br /></div><p></p><p>Afterwards - just keep patient until the lapsed number of days have been passed and then you should see a size in reduction for the SubscriptionTrackingDeletedObject</p><div><br /></div>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-52279677741975377012023-07-21T07:28:00.005+02:002023-07-21T07:33:07.672+02:00Dynamics 365 and Power Platform monthly reading list July 2023<p><b>2023 Release Wave 2</b></p><p></p><ul style="text-align: left;"><li><a href="https://learn.microsoft.com/en-us/dynamics365/release-plan/2023wave2/" target="_blank">Dynamics 365, Viva Sales (aka Microsoft Sales Copilot) and supply chain platform: 2023 release wave 2 plan</a> - early access will be available on July 31th</li><li><a href="https://learn.microsoft.com/en-us/power-platform/release-plan/2023wave2/" target="_blank">Microsoft Power Platform: 2023 release wave 2 plan</a></li><li><a href="https://www.youtube.com/watch?v=QzYMeR3xODM" target="_blank">Power Platform 2023 Release Wave 2 Highlights (YouTube)</a></li><li><a href="https://www.youtube.com/watch?v=6HSQ6daQYzQ" target="_blank">Dynamics 365 and Power Platform 2023 Release Wave 2 highlights (YouTube)</a></li><li><a href="https://www.youtube.com/watch?v=H2NEpakpjik" target="_blank">Dynamics 365 Customer Service 2023 Release Wave 2 highlights (YouTube)</a></li></ul><p></p><p><b>Technical topics (Configuration, customization and extensibility)</b></p><p></p><ul style="text-align: left;"><li><a href="https://www.youtube.com/watch?v=zNegeDa21uY" target="_blank">ALM for Power Platform (March 2023 Washington user group recording)</a></li><li><a href="https://never-stop-learning.de/analyse-dataverse-solution-dependencies/" target="_blank">Analyse your solution dependencies in Dataverse</a> with VSCode "PowerApps Helper" extension built by <a href="https://twitter.com/MMe2K" target="_blank">@MMe2K (Michael Megel)</a></li><li><a href="https://www.youtube.com/watch?v=Z0Ug29LwEzg" target="_blank">Power Apps Copilot (preview) feature ... Friend or Foe? (YouTube)</a> by <a href="https://www.twitter.com/dchristian19" target="_blank">@dchristian19</a></li><li><a href="https://github.com/chris-bowman/Azure-Cost-Reporting/tree/main/Power%20BI%20Report" target="_blank">Azure Cost Management Power BI report (Github)</a></li><li><a href="https://techcommunity.microsoft.com/t5/azure-architecture-blog/codename-project-bose-calculate-azure-cost-of-an-enterprise-by/ba-p/3741295" target="_blank">Codename Project Bose: Calculate Azure Cost of an enterprise by cost centers, divisions and projects</a></li><li><a href="https://www.youtube.com/watch?v=l3T7Tv637dk" target="_blank">KQL Visualizations (YouTube)</a></li><li><a href="https://www.inogic.com/blog/2022/03/how-to-use-invitation-id-of-customer-voice-survey-invitation-in-power-automate-flow/" target="_blank">How to use invitation id of Customer Voice Survey invitation in Power Automate Flow</a> by <a href="https://twitter.com/inogic" target="_blank">@inogic</a></li><li><a href="https://d365goddess.com/customer-360-component/" target="_blank">Use the Customer 360 component in Dynamics 365 Customer Service</a> by <a href="https://twitter.com/D365Goddess" target="_blank">@D365Goddess</a></li><li><a href="https://www.inogic.com/blog/2023/05/additional-capabilities-in-timeline-view-within-microsoft-dynamics-365-crm/" target="_blank">Additional capabilities in timeline view with Dynamics CRM</a></li><li><a href="https://learn.microsoft.com/en-us/power-apps/developer/data-platform/optimize-performance-create-update" target="_blank">Optimize performance for Create and Update operations (Microsoft Docs)</a> - interesting update for data migration scenario using <a href="https://learn.microsoft.com/en-us/power-apps/developer/data-platform/org-service/use-createmultiple-updatemultiple?tabs=sdk" target="_blank">CreateMultiple and UpdateMultiple operations</a></li><li><a href="https://learn.microsoft.com/en-us/power-apps/maker/data-platform/data-retention-overview" target="_blank">Dataverse long term data retention is finally available in preview</a> - no info available yet on whether retained Dataverse records (offloaded to Data Lake) will still be billed at the same price as Dataverse storage.</li><li>The Dataverse Accelerator is live on AppSource: <a href="https://aka.ms/dvaccelerator">https://aka.ms/dvaccelerator</a></li><li><a href="https://www.youtube.com/watch?v=tcdETeUelYQ" target="_blank">Discover the biggest Power Platform Announcements from Microsoft Build 2023 (YouTube)</a> by <a href="https://twitter.com/LisaCrosbie" target="_blank">@LisaCrosbie</a></li><li><a href="https://www.serverlesssql.com/synapse-link-for-dataverse-exporting-to-delta-lake/" target="_blank">Azure Synapse Link for Dataverse : exporting to Delta Lake</a></li><li><a href="https://detective.kusto.io/" target="_blank">Kusto Detective Agency</a> - learn KQL by playing a detective game</li><li><a href="https://www.youtube.com/watch?v=3Hv1SvAmyg0" target="_blank">Power Apps Data Sources Ranked! Which is best? YouTube video</a> by <a href="https://twitter.com/ShanesCows" target="_blank">Shane Young (@ShanesCows)</a></li><li><a href="https://powerbi.microsoft.com/en-us/blog/new-card-visual-public-preview/" target="_blank">New card visual in Power BI</a> and <a href="https://eriksvensen.wordpress.com/2023/06/17/powerbi-create-a-sparkline-in-the-new-card-visual-and-some-other-nice-formatting-options/" target="_blank">Create a sparkline in the new card visual</a></li><li><a href="https://cloudblogs.microsoft.com/dynamics365/it/2023/06/23/take-customer-service-to-new-heights-with-cross-geo-support-for-copilot/" target="_blank">Take Customer Service to new heights with cross-geo support for Copilot</a></li><li><a href="https://powerapps.microsoft.com/en-us/blog/announcing-monthly-channel-for-model-driven-apps/" target="_blank">Announcing monthly (release) channel for model-driven apps</a> - release channels provide more choice for adopting new features. It seems that Dynamics 365 and Power Platform will allow to get features activated on a monthly basis</li><li><a href="https://powerapps.microsoft.com/en-us/blog/announcing-public-preview-of-ip-internet-protocol-firewall-for-dataverse/" target="_blank">Announcing public preview of IP firewall in Dataverse</a> - only available for managed environments</li><li><a href="https://adoption.microsoft.com/en-us/copilot/" target="_blank">Microsoft Copilot readiness hub</a></li><li><a href="https://temmyraharjo.wordpress.com/2023/07/15/dataverse-the-importance-of-orderby-when-retrieve-more-than-5k-records/" target="_blank">The importance of Orderby when retrieving more than 50K records from Dataverse</a></li><li><a href="https://learn.microsoft.com/en-us/azure/cost-management-billing/costs/quick-acm-cost-analysis" target="_blank">Start using Azure Cost Management</a></li></ul><p></p><p><b>Topics for Dynamics 365 Business Applications Platform consultants, project managers and power users</b></p><p></p><ul style="text-align: left;"><li><a href="https://blogs.microsoft.com/blog/2023/03/06/introducing-microsoft-dynamics-365-copilot/" target="_blank">Introducing Microsoft Dynamics 365 copilot</a></li><li><a href="https://cloudblogs.microsoft.com/dynamics365/bdm/2023/03/16/copilot-in-viva-sales-available-as-thousands-of-customers-adopt-next-generation-ai-in-dynamics-365-copilot-and-power-platform/" target="_blank">Copilot in Viva Sales</a></li><li><a href="https://cloudblogs.microsoft.com/dynamics365/it/2023/04/04/dynamics-365-implementation-portal-now-available/" target="_blank">Dynamics 365 Implementation Portal now available</a></li><li><a href="https://cloudblogs.microsoft.com/dynamics365/bdm/2023/06/05/a-report-on-global-state-of-customer-service-sophistication/" target="_blank">A report on Global State of Customer Service sophistication</a></li><li><a href="https://cloudblogs.microsoft.com/dynamics365/bdm/2023/07/18/reshaping-the-future-of-business-with-microsoft-sales-copilot-dynamics-365-customer-insights-and-a-new-cloud-migration-program/" target="_blank">Reshaping the future of business with Microsoft Sales Copilot, Dynamics 365 Customer Insights and a new cloud migration program</a></li><li><a href="https://cloudblogs.microsoft.com/dynamics365/bdm/2023/07/18/introducing-aim-from-microsoft-future-proof-your-business-in-the-cloud/" target="_blank">Introducing AIM from Microsoft: Future-proof your business in the cloud</a></li></ul><p></p><div><br /></div>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-15553059683660995712023-06-28T15:59:00.008+02:002023-08-11T20:26:37.784+02:00Cost optimization: older mails in Dataverse now stored in Dataverse File Storage<p>Back in April 2019, Microsoft introduced a tiered storage architecture. This tiered storage architecture allows Microsoft to use the most appropriate storage type for different types of data used in Dynamics 365 and Dataverse. Last month, Microsoft started moving the storage contents of the e-mail body (email.description) from more expensive Dataverse Database storage to cheaper Dataverse File Storage - for the <a href="https://learn.microsoft.com/en-us/power-apps/developer/data-platform/email-activity-entities#email-storage" target="_blank">official documentation check out Dataverse - Email activity tables -Email storage.</a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxexRmkc8MhWBkUsQ81qPzA2cp9nPFyV-A2RUQ73HbbSlOgu4aS8q-pkn7mW9bMByXYdWkBuTaEhhfq5o-CtZLHklDf95_cOYKtWM5NW1H4KXqEvyF8KmZHe7Kn-rGBM0iR7R2_gGR8Yqsvs1THMA8UFzNbugybb9rVxiVSbvdzG9FAXnMqr4z/s949/emailstorage_docs.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="912" data-original-width="949" height="616" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxexRmkc8MhWBkUsQ81qPzA2cp9nPFyV-A2RUQ73HbbSlOgu4aS8q-pkn7mW9bMByXYdWkBuTaEhhfq5o-CtZLHklDf95_cOYKtWM5NW1H4KXqEvyF8KmZHe7Kn-rGBM0iR7R2_gGR8Yqsvs1THMA8UFzNbugybb9rVxiVSbvdzG9FAXnMqr4z/w640-h616/emailstorage_docs.PNG" width="640" /></a></div><br /><p><br /></p><p>Based on the list prices I see in my personal tenant (see screenshot below) - this cuts the unit cost for additional capacity required for e-mails down from 37,4 €/GB/month to 1,87€/GB/month (Quite interesting how they still use the older name Common Data Service :-) )</p><p>Your pricing might defer based on the type of licensing agreement that you have. If you had to buy additional storage capacity in the past, you might want to revise your storage usage and the required storage capacity add-ons.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinRjuoXDBnOuHKU5LGytP8NueXYRdaVQLpl8O_dm_zYjkkJEPp8oPa0vpLN3wIczL2XgXY_1i3z00nknWuDmV1uMLQfR3e19O0dZdQDUeY2Z7DiBWhHxzTUsBH8nN_F_S4sJXU2pmDYS6k2Xj3q2bXiP2u3Gyn_PgYJmMrVpqyF6XBD2uWmtu8/s1793/capacitybilling.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="446" data-original-width="1793" height="159" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinRjuoXDBnOuHKU5LGytP8NueXYRdaVQLpl8O_dm_zYjkkJEPp8oPa0vpLN3wIczL2XgXY_1i3z00nknWuDmV1uMLQfR3e19O0dZdQDUeY2Z7DiBWhHxzTUsBH8nN_F_S4sJXU2pmDYS6k2Xj3q2bXiP2u3Gyn_PgYJmMrVpqyF6XBD2uWmtu8/w640-h159/capacitybilling.PNG" width="640" /></a></div><br /><p><br /></p><p>You can check out the storage capacity in the Power Platform Admin Center - I added an example from a specific instance were Dataverse storage for e-mails (see ActivityPointerBase table) from 63 GB to 30GB.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDHCmLbZSD-n6eLhd0uGlq-jGev0KV-pDN-647au6fM9MenVIg6GaOBPs8aGPBdv--laIJi-y9NDJYFP6IERwc2ow14NQZrHJS50e607M24KFwZFuAM1Urim3wrldgo4sbKdbZaODv9mTeAfq7XZVrdif1EDw9u7YUmo7Hoe0RfmCRmpl-gCT2/s1280/emailstorage_dataverse.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="720" data-original-width="1280" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDHCmLbZSD-n6eLhd0uGlq-jGev0KV-pDN-647au6fM9MenVIg6GaOBPs8aGPBdv--laIJi-y9NDJYFP6IERwc2ow14NQZrHJS50e607M24KFwZFuAM1Urim3wrldgo4sbKdbZaODv9mTeAfq7XZVrdif1EDw9u7YUmo7Hoe0RfmCRmpl-gCT2/w640-h360/emailstorage_dataverse.png" width="640" /></a></div><br /><p>A the same time, we saw a new storage file type appearing called Email which grew to 3,8 GB. This is a lot less than the storage saved in the database since the data in file storage (Azure Blob Storage behind the scenes) is actually compressed.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjg5P4YmZy1vJTXFFWYZl4tIs55zPTMM3oXc3bTs0LKNLrCGco5ZEzD4NEFM6vWVqHYb4o0nc9sSH7SqZ2Rg89loY1rQ2d97jgcSY8Mwy3478kg_r5_j-WeOsCv_TcdUw3upq7TdPj6T1lX_SyKpbbtrb3OG-Vadg23lrC1pvbqe4X706swhCXN/s1280/emailstorage_dataverse2.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="720" data-original-width="1280" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjg5P4YmZy1vJTXFFWYZl4tIs55zPTMM3oXc3bTs0LKNLrCGco5ZEzD4NEFM6vWVqHYb4o0nc9sSH7SqZ2Rg89loY1rQ2d97jgcSY8Mwy3478kg_r5_j-WeOsCv_TcdUw3upq7TdPj6T1lX_SyKpbbtrb3OG-Vadg23lrC1pvbqe4X706swhCXN/w640-h360/emailstorage_dataverse2.png" width="640" /></a></div><p>All in all, a great job from Microsoft which might save you a lot of money. Don't hesitate to add a comments sharing your experience with this.... </p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-784093616014381682023-06-02T13:36:00.021+02:002023-07-27T11:11:38.015+02:00Microsoft Fabric FirehoseMicrosoft Fabric generated quite a buzz after the announcements at Microsoft Build last week but it feels a lot like drinking from a firehose. (If you can't get enough check out the <a href="https://news.microsoft.com/build-2023-book-of-news/" target="_blank">Microsoft Build 2023 - Book of News</a>)<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_ngwvkFlmqx1LOfSJwyOz4Z8cOgG2UV0BtR3OXK3pxsPMh0gV68oAAzQKkHsmpOQAllr1fw25phN5QvWISHnFPkpgwt2rr-DxLOYFTaKpze5yTKds1hGB30Wa5UsQaKuRZWwcGIN6n_5pLeQu-JEvzhYGDT1hFCZaDWpUA4tNMuxv-tCCtA/s800/1677635836919.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="694" data-original-width="800" height="278" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_ngwvkFlmqx1LOfSJwyOz4Z8cOgG2UV0BtR3OXK3pxsPMh0gV68oAAzQKkHsmpOQAllr1fw25phN5QvWISHnFPkpgwt2rr-DxLOYFTaKpze5yTKds1hGB30Wa5UsQaKuRZWwcGIN6n_5pLeQu-JEvzhYGDT1hFCZaDWpUA4tNMuxv-tCCtA/s320/1677635836919.jpg" width="320" /></a></div><div><br /></div><div><b>Microsoft documentation:</b></div><div><ul style="text-align: left;"><li><a href="https://blog.fabric.microsoft.com/en-US/blog/" target="_blank">Microsoft Fabric Blog</a></li><li><a href="https://learn.microsoft.com/en-us/training/paths/get-started-fabric/" target="_blank">Microsoft Learn Learning Path - Get started with Microsoft Fabric </a></li><li><a href="https://learn.microsoft.com/en-us/training/courses/dp-601t00" target="_blank">Microsoft Learn Learning Path - Implementing a lakehouse with Microsoft Fabric</a></li><li><a href="https://community.fabric.microsoft.com/" target="_blank">Microsoft Fabric Community</a></li><li><a href="https://azure.microsoft.com/en-us/blog/introducing-microsoft-fabric-data-analytics-for-the-era-of-ai/" target="_blank">Introducing Microsoft Fabric: data analytics for the era of AI </a></li></ul></div><div><br /></div><div>Besides the official announcements from Microsoft (which have a certain marketing fluffiness) it might make sense to check out the list of unofficial resources below.</div><br /><div><b>Link collection:</b></div><div><ul style="text-align: left;"><li><a href="https://www.reddit.com/r/MicrosoftFabric/comments/13vu6v1/ask_me_anything_ama_about_microsoft_fabric/" target="_blank">Ask me Anything about Microsoft Fabric (Reddit)</a></li><li><a href="https://radacad.com/getting-started-with-data-pipelines-in-fabric-data-factory" target="_blank">Getting started with Data Pipelines in Fabric Data Factory</a> by radacad</li><li><a href="https://learn.microsoft.com/en-us/power-bi/fundamentals/fabric-get-started" target="_blank">Microsoft Tutorial: Fabric for Power BI users</a></li><li><a href="https://mrpaulandrew.com/2023/05/31/what-is-microsoft-fabric-my-point-of-view/" target="_blank">What is Microsoft Fabric my point of view</a> by PaulAndrew</li><li><a href="https://www.infoworld.com/article/3697020/why-microsoft-is-combining-all-its-data-analytics-products-into-fabric.html" target="_blank">Why Microsoft is combining all its data analytics products into Fabric</a> by <a href="https://twitter.com/InfoWorld" target="_blank">@Infoworld</a></li><li><a href="https://www.youtube.com/watch?v=-f0XIVEP7bE" target="_blank">What is Microsoft Fabric (Public Preview) on YouTube</a> by <a href="https://twitter.com/GuyInACube" target="_blank">@guyinacube</a></li><li><a href="https://www.youtube.com/watch?v=1o_QDFq6gzE" target="_blank">Microsoft Fabric Launch Event (Day1) recording on YouTube (3h:20m)</a> and <a href="https://www.youtube.com/watch?v=_Y-XyCRE6ec" target="_blank">Microsoft Fabric Launch Event Day 2 on YouTube ( 2h:25m)</a></li><li><a href="https://endjin.com/blog/2023/05/azure-synapse-analytics-versus-microsoft-fabric-a-side-by-side-comparison" target="_blank">Azure Synapse Analytics versus Microsoft Fabric: a side by side comparison</a></li><li><a href="https://data-mozart.com/killing-me-softly-has-microsoft-fabric-just-overwritten-synapse-analytics/" target="_blank">Killing me softly has Microsoft Fabric just overwritten Synapse Analytics?</a></li><li><a href="https://cloudblogs.microsoft.com/dynamics365/it/2023/05/24/new-dataverse-enhancements-and-ai-powered-productivity-with-microsoft-365-copilot/" target="_blank">Eliminate data siloes with Fabric and Dynamics 365 (Dataverse announcements)</a></li><li><a href="https://www.youtube.com/watch?v=LkjRy2HRrks" target="_blank">Advancing Fabric - What is Microsoft Fabric? on YouTube (19m)</a> </li><li><a href="https://www.advancinganalytics.co.uk/blog/2023/5/17/what-is-microsoft-fabric" target="_blank">Microsoft Fabric introduction blog post</a> by by <a href="https://twitter.com/AdvAnalyticsUK">Advancing Analytics (@AdvAnalyticsUK) / Twitter</a></li><li><a href="https://data-marc.com/2023/05/24/overcoming-the-fear-exploring-microsoft-fabric-from-a-power-bi-angle/" target="_blank">Overcoming the Fear: exploring Microsoft Fabric from a Power BI angle</a></li><li><a href="https://data-goblins.com/power-bi/fabric-announcements" target="_blank">Microsoft Fabric - Power BI announcements a visual overview</a></li><li><a href="https://www.youtube.com/playlist?list=PLPaNVDMhUXGYU97pdqwoaociLdwyDRn39" target="_blank">Microsoft Fabric YouTube series </a></li><li><a href="https://venturebeat.com/data-infrastructure/how-microsoft-fabric-aims-to-beat-amazon-and-google-in-the-cloud-war/" target="_blank">How Microsoft Fabric aims to beat Amazon and Google in the cloud war</a> by <a href="https://twitter.com/VentureBeat" target="_blank">@Venturebeat</a></li><li><a href="https://sqlserverbi.blog/2023/05/22/how-lakehouse-architecture-is-revolutionalizing-business-intelligence/" target="_blank">How Lakehouse architecture is revolutionizing Business Intelligence</a></li><li><a href="https://www.jamesserra.com/archive/2023/06/microsoft-fabric-introduction-video/" target="_blank">Microsoft Fabric introduction video </a>by <a href="https://twitter.com/JamesSerra" target="_blank">@jameserra</a></li><li><a href="https://www.nickyvv.com/2023/05/what-is-microsoft-fabric-and-why-should-i-care.html" target="_blank">What is Microsoft Fabric and why should I care?</a> by <a href="https://twitter.com/NickyvV" target="_blank">@NickyvV</a></li><li><a href="https://www.youtube.com/watch?v=r4Yr0FzmD0k&t=1s" target="_blank">Unleashing the power of Microsoft Fabric for data engineers (YouTube)</a> - great demo by Justyna Lucznik of the data engineering capabilities in under 10 minutes</li><li><a href="https://radacad.com/microsoft-fabric-licensing-an-ultimate-guide" target="_blank">Microsoft Fabric Licensing: an ultimate guide</a> by <a href="https://twitter.com/RADACAD_COM" target="_blank">@RADACAD_COM</a></li><li><a href="https://www.youtube.com/watch?v=SFta1_70T_U" target="_blank">Create your first Lakehouse in Microsoft Fabric (YouTube)</a> by guyinacube</li><li><a href="https://microsoft.github.io/fabricnotes/" target="_blank">Microsoft Fabric Notes </a>- simple drawings illustrating the main concepts of Microsoft Fabric</li><li><a href="https://www.serverlesssql.com/productionising-fabric/" target="_blank">Microsoft Fabric in production</a> - placeholder blog with list of features that <a href="https://twitter.com/MrAndyCutler" target="_blank">@MrAndyCutler</a> considers needed before using Microsoft Fabric in production.</li><li><a href="https://radacad.com/lakehouse-vs-warehouse-vs-datamart-the-difference-between-the-three-fabric-objects" target="_blank">Lakehouse vs Warehouse vs Datamart - the difference between the three Fabric objects</a> by <a href="https://twitter.com/RADACAD_COM" target="_blank">@RADACAD_COM</a></li><li><br /></li></ul></div>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-79442210075379180292023-04-28T10:58:00.003+02:002023-04-28T10:58:48.204+02:00Azure Synapse Link for Dataverse playlist by Scott Sewell<p>A great starting point if you are new to <a href="https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-synapse" target="_blank">Azure Synapse Link for Dataverse</a> is <a href="https://www.youtube.com/playlist?list=PLM-lT-OX5zBqJ54ic7PfVJrtsgbJXbmP9" target="_blank">Scott Sewell's YouTube playlist on Azure Synapse Link for Dataverse</a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-SQsBxcVb0sLB3mVj1WBCSdELMZlcJ9iOvly_fVmc9jw6GMHgB61B65ZZD9H6ZQFj5KFJkyeTJjua5KqkJ-gbmObmpDU0jAH_w6fYFKhS3l4cNyOEkVGfnh932xo4Qjq_KcK3e6ihsXLk58AYWWKSy70XxlpajJyX1QOuDGIvOgxZwog-4Q/s1535/scott.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="710" data-original-width="1535" height="296" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-SQsBxcVb0sLB3mVj1WBCSdELMZlcJ9iOvly_fVmc9jw6GMHgB61B65ZZD9H6ZQFj5KFJkyeTJjua5KqkJ-gbmObmpDU0jAH_w6fYFKhS3l4cNyOEkVGfnh932xo4Qjq_KcK3e6ihsXLk58AYWWKSy70XxlpajJyX1QOuDGIvOgxZwog-4Q/w640-h296/scott.PNG" width="640" /></a></div><br /><p><br /></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-64933367000811668392023-04-21T18:14:00.001+02:002023-04-28T10:59:08.483+02:00SharePoint 2013 workflow retirement - the end of an era<p><a href="https://support.microsoft.com/en-us/office/sharepoint-2013-workflow-retirement-4613d9cf-69aa-40f7-b6bf-6e7831c9691e" target="_blank">Microsoft is (finally) retiring SharePoint workflows</a> - the end of an era of workflows built on top of Microsoft Workflow Foundation (<a href="https://jopx.blogspot.com/2005/10/windows-workflow-foundation-resources.html" target="_blank">Workflow Foundation got introduced in 2005 - and yes I did play around with the beta of this back then - since I though it was the greatest thing since sliced bread</a> )</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-9YCNIW3Cw6Isj9M13wxb2GcGCPzxV0i4djnKEdYg0vzt5Vp-rlpTIQ6ymY4hE_gWezySJGDBBoaRTSD0jNEYZ2m3x-pIYLiUZtgp6yrysAARvvdYncddG6X9RR4263yJondE8UokLAtu8nPCk2eUC11hFHtBmnimGi6yYsiApjNtv-9gbg/s745/sharepointworkflows.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="428" data-original-width="745" height="368" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-9YCNIW3Cw6Isj9M13wxb2GcGCPzxV0i4djnKEdYg0vzt5Vp-rlpTIQ6ymY4hE_gWezySJGDBBoaRTSD0jNEYZ2m3x-pIYLiUZtgp6yrysAARvvdYncddG6X9RR4263yJondE8UokLAtu8nPCk2eUC11hFHtBmnimGi6yYsiApjNtv-9gbg/w640-h368/sharepointworkflows.png" width="640" /></a></div><br /><p><br /></p><p>Starting April 2d 2024 all SharePoint 2013 workflows will be turned off for newly created tenants. April 2d 2026 all SharePoint 2013 workflows will stop working for existing tenants.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjS33oe4IF_PBURCcxTjZOkZkoNw-VLnCnVfZhinKxmhSa5zQQ5YlvLyQi0Q4pTsmpe76B8BuePhZ5f8avC6NvxaIzZKOs_gZIzND1lRYJK2Ct5XHQYpsT4ie1Blui1aL-C5NH5nR0pciY2Un-halwHxG7vYSW1va7gjCX0PfWomqlYTdTYMg/s727/sharepointworkflowretirement.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="727" data-original-width="431" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjS33oe4IF_PBURCcxTjZOkZkoNw-VLnCnVfZhinKxmhSa5zQQ5YlvLyQi0Q4pTsmpe76B8BuePhZ5f8avC6NvxaIzZKOs_gZIzND1lRYJK2Ct5XHQYpsT4ie1Blui1aL-C5NH5nR0pciY2Un-halwHxG7vYSW1va7gjCX0PfWomqlYTdTYMg/w380-h640/sharepointworkflowretirement.PNG" width="380" /></a></div><br /><p><b>Helpful links:</b></p><p></p><ul style="text-align: left;"><li><a href="https://pnp.github.io/pnpassessment/workflow/readme.html" target="_blank">Microsoft 365 assessment tool to scan tenants</a></li><li><a href="https://learn.microsoft.com/en-us/sharepoint/dev/business-apps/power-automate/guidance/migrate-from-classic-workflows-to-power-automate-flows" target="_blank">Guidance: migrate from classic workflows to Power Automate flows in SharePoint</a></li></ul><p></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-2295123957964647832023-04-18T21:20:00.009+02:002023-04-24T17:39:29.716+02:00Looking at historical returns of stocks and bonds with Power BI and Python<p>You might already have seen below graph taken from a study by JP Morgan Asset Management, but what if you would like to look at historical returns without going through the hassle of having to collect all the data yourself?</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQo8KooVBNOoQHRolspLY7XSaQZdFPje_vYdYbsad7FBHXhC8-jgmOzhFoiDw1HRn_ohJSk1uOjFAVbGBAlZCpzyzBN0cEPU82OnRoORdMZnYBmFmdtB7pEgU-8U4Kkjh6G5ix8qCvhnRRnP219ful4CYUf-yvbQPQiYVw5I8MHVxZdVi9BQ/s1272/annualized_returns_by_assetclass.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="636" data-original-width="1272" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQo8KooVBNOoQHRolspLY7XSaQZdFPje_vYdYbsad7FBHXhC8-jgmOzhFoiDw1HRn_ohJSk1uOjFAVbGBAlZCpzyzBN0cEPU82OnRoORdMZnYBmFmdtB7pEgU-8U4Kkjh6G5ix8qCvhnRRnP219ful4CYUf-yvbQPQiYVw5I8MHVxZdVi9BQ/w640-h320/annualized_returns_by_assetclass.png" width="640" /></a></div><br /><p>There is an interesting Excel sheet shared by <a href="https://pages.stern.nyu.edu/~adamodar/" target="_blank">Aswath Damodaran</a> (<a href="https://twitter.com/AswathDamodaran" target="_blank">@AswathDamodaran</a>) that you can download from <a href="https://pages.stern.nyu.edu/~adamodar/New_Home_Page/datafile/histretSP.html" target="_blank">Historical Returns on Stocks, Bonds and Bills: 1928-2022</a> which looks at returns of different asset classes (stocks, bonds, bills, real estate and gold) over a longer time period.</p><p>In this post I will share some tips on how you can use this data in Power BI, Python and Jupyter notebooks. </p><p>This <a href="https://pages.stern.nyu.edu/~adamodar/New_Home_Page/datafile/histretSP.html" target="_blank">Historical Returns on Stocks, Bonds and Bills: 1928-2023 - Excel file</a> file is updated in the first two weeks of every year and it is being maintained by <a href="https://pages.stern.nyu.edu/~adamodar/" target="_blank">Aswath Damodaran</a>, who is a professor of Finance at the Stern School of Business at NYU, he is also known as the "Dean of Valuation" due to his experience in this area.</p><div class="separator" style="clear: both; text-align: left;"><b>Visualizing S&P 500 and US Treasury bond returns using Power BI</b></div><div><br /></div><div>I first converted the Excel from xls to xlsx format and afterwards it is quite easy to <a href="https://learn.microsoft.com/en-us/power-bi/connect-data/service-excel-workbook-files" target="_blank"> import the data from an Excel workbook files in Power BI </a>. It is quite easy to visualize the returns of both stocks and US treasury bonds using a clustered column chart - I also added a minimum line for both stock and bond returns.</div><div><br /></div><div><div>Expected risk and expected return should go hand in hand: the higher the expected return, the higher the expected risk. Risk means means that the future actual return may vary from the expected return (and the ultimate risk is loosing all of your assets). The first visual showed a 20-year annualized return between 1999 and 2018 for the S&P 500 of 5.8%. Average returns hide however the big swings in yearly returns - e.g. in 2008 (the Great Financial Crisis), the S&P 500 had a -36.5% yearly return. Bonds on average have a lower return but also have a lower risk profile. </div><div><br /></div><div>The basic rule of thumb is to keep your “safe money” (i.e., money you don’t want to risk in stocks) in high-quality bonds. While this doesn’t give you 100% protection against losses at all times, it does provide you some peace of mind. I really like this quote: "If you can't sleep at night because of your stock market position, then you have gone too far. If this is the case, then sell your positions down to the sleeping level. (Jesse Livermoore)"<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEi8m5KdDxOWQZw1ep14AQLleBUPQONdToh2OYSyunMegeZTN7jv3PchO5GFwtAE2qaLIGP98EzcxvPF7j3fmqyLvFlQs8senpsJQrCHI8PzXoRblDaTrLV3zXyLQCCkW4CEVVmwIew-yoWsQ3AG1JC6CPZiYsFu0O6pFBlCqLGzItENMQExuw" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="140" data-original-width="5365" height="8" src="https://blogger.googleusercontent.com/img/a/AVvXsEi8m5KdDxOWQZw1ep14AQLleBUPQONdToh2OYSyunMegeZTN7jv3PchO5GFwtAE2qaLIGP98EzcxvPF7j3fmqyLvFlQs8senpsJQrCHI8PzXoRblDaTrLV3zXyLQCCkW4CEVVmwIew-yoWsQ3AG1JC6CPZiYsFu0O6pFBlCqLGzItENMQExuw" width="320" /></a></div></div><div><br /></div><div>As you can see in the visualization below, in most years with a negative return for the S&P 500, the return for bonds is positive - with two notable exceptions 1969 and 2022. A common saying is to have your age in bonds. Using that general rule, a 45-year-old might have 45% of the total portfolio in bonds. If you want to more aggressive, you would have less than your age in bonds. The last decade with interest rates very low (or even negative) this probably wasn't a very profitable asset allocation but </div><div>things might have shifted.</div></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpKAdO_ngI2MLEB2Vkb4DKfRVc9uODFSMDEXKnmAjO3cv3QjTPEaiHhisikPCdkawYwKtV4dsvT-Dd626qKzmADDBRTT9HxvyURWqSlU8kEk77EOg7EEYp3LTmLGaEYEeRyMjKU1NoutzDjpN5BeWk2K64WUKrCEOoLKkPIbIOr8q1DBZe5w/s853/Bondreturns.gif" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="853" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpKAdO_ngI2MLEB2Vkb4DKfRVc9uODFSMDEXKnmAjO3cv3QjTPEaiHhisikPCdkawYwKtV4dsvT-Dd626qKzmADDBRTT9HxvyURWqSlU8kEk77EOg7EEYp3LTmLGaEYEeRyMjKU1NoutzDjpN5BeWk2K64WUKrCEOoLKkPIbIOr8q1DBZe5w/w640-h360/Bondreturns.gif" width="640" /></a></div><br /><div><br /></div><div><br /></div><div>The US Treasury Bond used in the Excel file is the 10-year US treasury bond for which you can download the data from FRED . The yearly return has been calculated by taking the yield and the price change for a par bond with that specific yield.</div><div><br /></div><div>You can <a href="https://github.com/jorisp/powerbi/blob/main/histreturns.pbix" target="_blank">download the Power BI file histreturns.pbix from my Power BI Github repo</a></div><div><br /></div><div>In the long run (see example below for different rolling windows from a 1-year to a 20 year period) stocks will outperform bonds but this again works with averages and it ignores the tail risk which might wreak havoc in your portfolio.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdpz9H24PxbYbX72wQKkUBeEIRyzbfJ4JV6hG6d2fS7us9OXWjlMS_Jby58kSVDfxjFYgKuWyM-51e9fdSC25RDt9AYTuqld1RAZ0t32KxfmHCR7J2jr7FbXisI--BOlnNR433VXnzdELNOMgJIWdque3_fgw_oZDcmsDCZdG5Whq1hN96Gw/s1200/stock_vsbondreturns.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="799" data-original-width="1200" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdpz9H24PxbYbX72wQKkUBeEIRyzbfJ4JV6hG6d2fS7us9OXWjlMS_Jby58kSVDfxjFYgKuWyM-51e9fdSC25RDt9AYTuqld1RAZ0t32KxfmHCR7J2jr7FbXisI--BOlnNR433VXnzdELNOMgJIWdque3_fgw_oZDcmsDCZdG5Whq1hN96Gw/w640-h426/stock_vsbondreturns.jpg" width="640" /></a></div><br /><div><br /></div><div><br /></div><div><b>Reading data from Excel using Python</b></div><div><b><br /></b></div><div>Now let's take a look at how you can read and manipulate the data in this Excel sheet using Python. To read an excel file as a DataFrame, I will use the pandas read_excel() method. Internally, Pandas. .read_excel uses a library called xlrd which you also need to install but I used the <a href="https://openpyxl.readthedocs.io/en/stable/" target="_blank">openpyxl library</a> as an alternative which also works. So before you can read an excel file in pandas, you will need to install </div><div>the <a href="https://openpyxl.readthedocs.io/en/stable/" target="_blank">openpyxl library</a>. </div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKB_DM2WTyBLllxbJjRZBCnGDAwat60pRID-zSr28NPcHeitI6BxJY0cSv6kEhrN94ar2MSLwyYwKpKylEAxPWE13MdDfwl2L9DYsUeJr42UWYzJTpncCNcm8CpgNGUVEjBBEMgJfNMSrTEkAc9pWGJwILcaeGygYyF-QwaM-vyn5b2Vx3uA/s688/pandas_readexcel.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="234" data-original-width="688" height="218" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKB_DM2WTyBLllxbJjRZBCnGDAwat60pRID-zSr28NPcHeitI6BxJY0cSv6kEhrN94ar2MSLwyYwKpKylEAxPWE13MdDfwl2L9DYsUeJr42UWYzJTpncCNcm8CpgNGUVEjBBEMgJfNMSrTEkAc9pWGJwILcaeGygYyF-QwaM-vyn5b2Vx3uA/w640-h218/pandas_readexcel.PNG" width="640" /></a></div><br /><div>The above code reads only the table with data from the Excel file (which I downloaded in a subdirectory data from the Jupyter notebook) - see <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_excel.html" target="_blank">pandas.read_excel in the Pandas referencel documentation</a> for full details:</div><div><ul style="text-align: left;"><li>sheet_name: can be an integer (for the index of a worksheet in an Excel file, default to 0, the first sheet) or the name of the worksheet you want to load</li><li>nrows: number of rows to read</li><li>skiprows: number of rows to skip</li><li>usecols: by default all columns will be read but also possible to pass in a list of columns to read into the dataframe like in the example</li></ul><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEmebyz2XJORQ_egvWwuXNByFXAaehdiqWn0QmzymBuXeS8ITR3l3IaOAjJKTV-qpQ2k1wFs6MPfe5dcJQ9Yu9ojdB6z5MZ8Q4ljLT2dE7wM0NWaN9p_E2X2JZuwnXhk1Pa_lpo8InhlLsfI_2tH6j2si6tGP5Rk0FFeNguGtomQXo79kFaA/s884/bondreturns_python.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="443" data-original-width="884" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEmebyz2XJORQ_egvWwuXNByFXAaehdiqWn0QmzymBuXeS8ITR3l3IaOAjJKTV-qpQ2k1wFs6MPfe5dcJQ9Yu9ojdB6z5MZ8Q4ljLT2dE7wM0NWaN9p_E2X2JZuwnXhk1Pa_lpo8InhlLsfI_2tH6j2si6tGP5Rk0FFeNguGtomQXo79kFaA/w640-h320/bondreturns_python.PNG" width="640" /></a></div><div><br /></div><div>I just started exploring some data around stock-bond correlations and will be updating the Juypyter notebook on Github - <a href="https://github.com/jorisp/tradingnotebooks/blob/master/HistoricalReturns.ipynb" target="_blank">https://github.com/jorisp/tradingnotebooks/blob/master/HistoricalReturns.ipynb</a></div><div><br /></div><div>A couple of weeks ago I noticed this <a href="https://twitter.com/WifeyAlpha/status/1645090954943115265?s=20" target="_blank">interesting tweet on rolling one-year-stock-bond correlations for six regimes</a> from <a href="https://twitter.com/WifeyAlpha" target="_blank">@WifeyAlpha</a> - I think it would make an interesting exercise to see how to rebuild this using Python.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaVJ1SmDKiDQZePRrUrWIUP94tj4B_VKYWlTNEye_KVA1PRpBdsi3-lfI5yRZaPYvvp_4W3wZ3e6nXPCm2HmFvT8-8s7T_cLcnKgB3yUAWz68eNoRJI3r8ginQ1A2VShPB8-P9DhxcgPxRPCePbjbnY8DG7Q4JlAaIe1LrvzeA1GQlHQFHAg/s586/bondregimes.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="515" data-original-width="586" height="562" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaVJ1SmDKiDQZePRrUrWIUP94tj4B_VKYWlTNEye_KVA1PRpBdsi3-lfI5yRZaPYvvp_4W3wZ3e6nXPCm2HmFvT8-8s7T_cLcnKgB3yUAWz68eNoRJI3r8ginQ1A2VShPB8-P9DhxcgPxRPCePbjbnY8DG7Q4JlAaIe1LrvzeA1GQlHQFHAg/w640-h562/bondregimes.PNG" width="640" /></a></div><br /><div><b>References:</b></div></div><p></p><ul style="text-align: left;"><li><a href="https://lifewithdata.com/2022/02/16/pandas-read_excel-how-to-read-excel-file-in-python/" target="_blank">Pandas – read_excel() – How to read Excel file in python</a></li><li><a href="https://aswathdamodaran.blogspot.com/2022/12/a-return-to-teaching-spring-2023-edition.html" target="_blank">Musings on markets: a return to teaching - the Spring 2023 Edition</a></li><li><a href="https://am.jpmorgan.com/us/en/asset-management/adv/insights/market-insights/guide-to-the-markets/" target="_blank">Guide to the Markets | JP Morgan Asset Management</a></li></ul><div><b>Related posts:</b></div><div><ul style="text-align: left;"><li><a href="https://jopx.blogspot.com/2023/01/interactive-chart-visualizations-using.html" target="_blank">Interactive chart visualizations using Python and bqplot: visualizing S&P500 returns</a></li><li><a href="https://jopx.blogspot.com/2022/08/using-yfinance-python-package-to.html?m=1" target="_blank">Using the yFinance Python package to download financial data from Yahoo Finance - part 2</a></li><li><a href="http://jopx.blogspot.com/2022/07/using-yfinance-python-package-to.html" target="_blank">Using the yFinance Python package to download financial data from Yahoo Finance</a></li><li><a href="https://jopx.blogspot.com/2022/05/using-python-and-pandas-datareader-to_19.html" target="_blank">Using Python and Pandas Datareader to retrieve financial data - part 2: Fama & French data library</a></li><li><a href="http://jopx.blogspot.com/2022/05/using-python-and-pandas-datareader-to.html" target="_blank">Using Python and Pandas Datareader to retrieve financial data - part 1: Federal Reserve Data (FRED)</a></li></ul></div><p></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-65187170491850570122023-03-22T05:17:00.004+01:002023-04-28T10:59:41.406+02:00Upgrade to Azure Functions runtime v 4.x<p>Even though the Azure Functions runtime 2.0 and 3.0 were retired on December 13th, 2022 , I recently still found some Azure Functions using these older runtimes. See <a href="https://learn.microsoft.com/en-us/azure/azure-functions/functions-versions?tabs=v4&pivots=programming-language-csharp" target="_blank">Azure Functions runtime versions overview</a> which outlines the support for the different version.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwhaDuDa2zddpPWH8lLPT0mqDPKNJ4c9hKOSPofGvc63JGZ4l81bdtjPQ2X6kaPF2_w0R1gppeDo9jnqXYHTamerAyiP2CxFpCmRQ2tzm4qX8BH0FCD5JnPkOuJupSIu53uMPvKYJGiWw7SATCGq2LN5M2hQDQ_N-r82zKSnAbkboNnSYw2A/s1280/Upgrade_pptx.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="720" data-original-width="1280" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwhaDuDa2zddpPWH8lLPT0mqDPKNJ4c9hKOSPofGvc63JGZ4l81bdtjPQ2X6kaPF2_w0R1gppeDo9jnqXYHTamerAyiP2CxFpCmRQ2tzm4qX8BH0FCD5JnPkOuJupSIu53uMPvKYJGiWw7SATCGq2LN5M2hQDQ_N-r82zKSnAbkboNnSYw2A/w640-h360/Upgrade_pptx.png" width="640" /></a></div><br /><p>Azure Functions that use runtime version 2.x or 3.x will continue to run but Microsoft is no longer providing updates or support for applications that use these versions. There a number of reasons why you should upgrade:</p><p></p><ul style="text-align: left;"><li style="box-sizing: inherit;">Leverage the support for the latest programming languages such as .NET 6 and Node 16. Take the opportunity to upgrade your code to .NET 6 since <a href="https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-6/" target="_blank">.NET 6 includes a lot of performance improvements</a></li><li style="box-sizing: inherit;">To make sure that you keep getting security updates</li><li style="box-sizing: inherit;">Be future proof and keep technical debt to a minimum (See <a href="https://jopx.blogspot.com/2023/01/technical-debt-and-dynamics-365.html" target="_blank">Technical debt and Dynamics 365 solution architecture</a> for my view on this)</li></ul><p></p><p><b>What is the Azure Functions runtime?</b></p><p>For those not so familiar with Azure Functions, the Azure Functions runtime is the engine that powers Azure Functions to execute user-defined code (written in . It is responsible for loading the user's code, executing it in response to specific triggers, and scaling the execution as needed to handle the workload. The runtime is a key component of the Azure Functions service, and it allows developers to focus on writing and deploying their code, rather than worrying about the underlying infrastructure.</p><p><b>How to find the Azure Functions runtime version?</b></p><p>When you log in to the Azure Portal and go to the Function runtime settings tab on the configuration page of an Azure function, you will see which runtime version is in use. Instead of having to look at every individual function, you might also take a look at the<a href="https://learn.microsoft.com/en-us/azure/governance/resource-graph/overview" target="_blank"> Azure Resource Graph</a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhE_mSygCOpLgkx4duV_KeYREF1RFmz70lD2Q7vfr1dP_IfuxuPhXi5OGdzxixNqrfVEom_DgV-a352ToLeR3BHis3mrtPnGAgTtEHtxVjfyzA9YmTp0FF5RPYyAUbIa1MI5h5rdDl-jMTAAzFw7wuaOXhlKDYUlfA3sKOCvKYlsY1G9KOHiQ/s1546/azurefunctionruntimecheck.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="803" data-original-width="1546" height="332" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhE_mSygCOpLgkx4duV_KeYREF1RFmz70lD2Q7vfr1dP_IfuxuPhXi5OGdzxixNqrfVEom_DgV-a352ToLeR3BHis3mrtPnGAgTtEHtxVjfyzA9YmTp0FF5RPYyAUbIa1MI5h5rdDl-jMTAAzFw7wuaOXhlKDYUlfA3sKOCvKYlsY1G9KOHiQ/w640-h332/azurefunctionruntimecheck.PNG" width="640" /></a></div><br /><p><br /></p><p><b><br /></b></p><p><br /></p><p><br /></p><p><br /></p><p><br /></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-66761250368842611702023-03-20T07:04:00.001+01:002023-03-20T07:04:08.278+01:00Notes and links for the DP-900 Microsoft Azure Data Fundamentals<p></p><div><div>As providing reporting and analytics capabilities in a Dynamics 365 solution is becoming increasingly important I decided to brush up my knowledge about available technologies and patterns in the data area. I think that taking a certification exam is a good way to get started on such a broad topic and it also allows you to set a clear goal. So end of last year, I finally took the <a href="https://learn.microsoft.com/en-us/certifications/exams/dp-900" target="_blank">DP-900: Microsoft Azure Data Fundamentals exam</a>. </div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhP3kImowqoJX5wHuPTD3WsSkyPhFN9HokMGYZm53NJ2GucAqPKG9cOzQIViEylJRk4QtO7plwxrYAMIGb--j7qmb_2Z3nMv84XSL3RaojfBupXotAMo_ddV0dGWOVxdl6RyZAmxY2TFXxXK5BFPL89Yt32ANBIOxlNYLdCg3phjgPZnSDYPQ/s600/azure-data-fundamentals-600x600.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="600" data-original-width="600" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhP3kImowqoJX5wHuPTD3WsSkyPhFN9HokMGYZm53NJ2GucAqPKG9cOzQIViEylJRk4QtO7plwxrYAMIGb--j7qmb_2Z3nMv84XSL3RaojfBupXotAMo_ddV0dGWOVxdl6RyZAmxY2TFXxXK5BFPL89Yt32ANBIOxlNYLdCg3phjgPZnSDYPQ/s320/azure-data-fundamentals-600x600.png" width="320" /></a></div><br /><div><br /></div><div><br /></div><div>Whenever you want to attain a Microsoft certification, the first place to start is always the study guide which you can find on the certification home page. Another interesting addition from Microsoft is the ability to take a <a href="https://learn.microsoft.com/en-us/certifications/exams/dp-900/practice/assessment?assessment-type=practice&assessmentId=24" target="_blank">practice assessment for DP-900</a> (link is listed on the certification home page - there is also a <a href="https://learn.microsoft.com/en-us/certifications/practice-assessments-for-microsoft-certifications" target="_blank">full list of all practice assessments for Microsoft Certifications</a>. Listed below are some of the links and resources that I used - mainly on topics where I had less hands-on experience - so this is definitely not an exhaustive list.</div></div><div><b><br /></b></div><div><b>YouTube videos playlists</b></div><div><ul style="text-align: left;"><li><a href="https://www.youtube.com/watch?v=0gtpasITVnk" target="_blank">DP-900 Data Fundamentals Study Cram v2</a> by <a href="https://twitter.com/NTFAQGuy">John Savill (@NTFAQGuy) / Twitter</a></li><li><a href="https://www.youtube.com/watch?v=XbV0Di5ggvY" target="_blank">Microsoft Azure Data Fundamentals [Exam DP-900] Full Course</a></li></ul></div><div><b><br /></b></div><div><b>Describe core data concepts</b></div><div><ul style="text-align: left;"><li><a href="https://learn.microsoft.com/en-us/azure/architecture/data-guide/relational-data/etl" target="_blank">Extract, transform and Load - ETL vs ELT (Microsoft Data Architecture Guide) </a></li><li><a href="https://learn.microsoft.com/en-us/training/modules/explore-roles-responsibilities-world-of-data/2-explore-job-roles" target="_blank">Explore job roles in the world of data: database admins, data engineers and data analysts</a></li><li><a href="https://learn.microsoft.com/en-us/training/modules/explore-core-data-concepts/5-transactional-data-processing" target="_blank">Explore transactional data processing - ACID semantics</a></li></ul></div><div><b><br /></b></div><div><b>Describe capabilities of Azure storage</b></div><div><ul style="text-align: left;"><li><a href="https://learn.microsoft.com/en-us/azure/storage/common/storage-redundancy" target="_blank">Azure Storage redundancy</a></li><li><a href="https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-change-feed?tabs=azure-portal" target="_blank">Change feed support in Azure Blob Storage</a></li></ul></div><div><b><br /></b></div><div><b>Describe capabilities and features of Azure Cosmos DB</b></div><div><ul style="text-align: left;"><li><a href="https://learn.microsoft.com/en-us/azure/cosmos-db/use-cases" target="_blank">Common Azure Cosmos DB use cases</a></li><li><a href="https://learn.microsoft.com/en-us/training/modules/explore-non-relational-data-stores-azure/3-cosmos-db-apis" target="_blank">Identify Azure Cosmos DB APIs</a></li><li><a href="https://learn.microsoft.com/en-us/azure/cosmos-db/gremlin/introduction" target="_blank">Introduction to Azure Cosmos DB for Apache Gremlin</a></li><li><a href="https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-manage-database-account" target="_blank">Manage an Azure Cosmos DB account using the Azure portal</a></li><li><a href="https://learn.microsoft.com/en-us/azure/cosmos-db/faq" target="_blank">Frequently asked questions about different APIs in Azure Cosmos DB</a></li><li><a href="https://learn.microsoft.com/en-us/azure/cosmos-db/high-availability" target="_blank">Achieve high availability with Azure Cosmos DB</a></li></ul></div><div><b>Describe relational Azure Data services</b></div><div><ul style="text-align: left;"><li><a href="https://learn.microsoft.com/en-us/azure/azure-sql/database/features-comparison?view=azuresql" target="_blank">Features comparison: Azure SQL Database and Azure SQL Managed Instance</a></li><li><a href="https://learn.microsoft.com/en-us/azure/security/fundamentals/infrastructure-sql" target="_blank">Azure SQL Database security features</a></li><li><a href="https://learn.microsoft.com/en-us/sql/t-sql/functions/aggregate-functions-transact-sql?view=sql-server-ver16" target="_blank">Aggregate functions (Transact-SQL)</a></li><li><a href="https://learn.microsoft.com/en-us/sql/tools/overview-sql-tools?view=sql-server-ver16" target="_blank">SQL Tools overview: Azure Data Studio, SQL Server Management Studio (SSMS), SQL Server Data Tools (SSDT) and mssql extension for VS Code and command line tools</a></li><li><a href="https://learn.microsoft.com/en-us/sql/azure-data-studio/what-is-azure-data-studio?view=sql-server-ver16" target="_blank">What is Azure Data Studio?</a></li><li><a href="https://learn.microsoft.com/en-us/sql/relational-databases/indexes/clustered-and-nonclustered-indexes-described?view=sql-server-ver16" target="_blank">Clustered and non-clustered indexes described</a></li></ul></div><div><b><br /></b></div><div><b>Describe common elements of large-scale analytics</b></div><ul><li><a href="https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction" target="_blank">Introduction to Azure Data Lake Storage Gen2</a></li><li><a href="https://learn.microsoft.com/en-us/sql/relational-databases/polybase/polybase-guide?view=sql-server-ver16" target="_blank">Introducing data virtualization with PolyBase</a></li></ul><div><b>Describe data visualization in Microsoft Power BI</b></div><ul><li><a href="https://learn.microsoft.com/en-us/power-bi/consumer/end-user-dashboards" target="_blank">Dashboards for business users of the Power BI service</a></li><li><a href="https://learn.microsoft.com/en-us/power-bi/create-reports/service-dashboard-add-widget" target="_blank">Add images, videos and more to your Power BI dashboard</a></li><li><a href="https://learn.microsoft.com/en-us/power-bi/create-reports/service-dashboard-pin-tile-from-excel" target="_blank">Pin a tile to a Power BI dashboard from Excel</a></li><li><a href="https://learn.microsoft.com/en-us/power-bi/connect-data/service-datasets-across-workspaces" target="_blank">Intro to datasets across workspaces (Power BI)</a></li><li><a href="https://learn.microsoft.com/en-us/power-bi/paginated-reports/report-builder-power-bi" target="_blank">Power BI Report Builder - authoring paginated reports</a></li></ul><p></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-31049986091881294232023-03-17T06:30:00.002+01:002023-03-21T07:56:21.584+01:00Quick tip: use the new release planner for Dynamics 365 and Power Platform to track and planning upcoming changes<p>The new release planner for Dynamics 365 and Power Platform is awesome - check it out at <a href="https://aka.ms/releaseplanner">https://aka.ms/releaseplanner</a> . It allows you to track the features that are planned, coming soon or already available for trial. You have visibility across active release waves in a portal (built with Power pages) and you can personalize, filter and sort release plans when you sign in.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_W4m-phTyCIgyXDrizXX4R815IYgjBY8QzYfAllHyjydrWZf-nIK39gi1XJcSw52eeNijKj67yd7bvd0o3TrE-dbjk4RJMxhQWGpIQ48gB18tjUw3FoPW7xNZU5LQumtKBpfm4CzBtPneqKvoONVGT92C7iK_uXf6rMpbN6EihcK1A2eNaw/s1384/releaseplanner.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="840" data-original-width="1384" height="388" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_W4m-phTyCIgyXDrizXX4R815IYgjBY8QzYfAllHyjydrWZf-nIK39gi1XJcSw52eeNijKj67yd7bvd0o3TrE-dbjk4RJMxhQWGpIQ48gB18tjUw3FoPW7xNZU5LQumtKBpfm4CzBtPneqKvoONVGT92C7iK_uXf6rMpbN6EihcK1A2eNaw/w640-h388/releaseplanner.PNG" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><p><br /></p><p><br /></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-76837350503664226742023-03-16T06:25:00.000+01:002023-03-16T06:25:02.000+01:00Quick tip: resolving no hosted parallelism has been purchased or granted in Azure DevOps<p>I recently setup a new free Azure DevOps organization but when I tried running my first pipeline - I got an exception "##[error]No hosted parallelism has been purchased or granted. To request a free parallelism grant, please fill out the following form https://aka.ms/azpipelines-parallelism-request".</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQL3bWF9hwMy2ufX3P3d-usn0sKGajPG9HvJm-wWfYKL3OIC0hO1gghqoyQW4DHRPVOBTi07ENqmA9z6EPrOWc5SVippPoa4OS2FWMB_vj6MVaasHZq9WfvLHeWBotP2Zfq2O31BNJL-ngSW44pi9QvP8xnK0LiOeR3z3QxAxe-i0CLt2wag/s880/nohostedparallelism.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="119" data-original-width="880" height="86" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQL3bWF9hwMy2ufX3P3d-usn0sKGajPG9HvJm-wWfYKL3OIC0hO1gghqoyQW4DHRPVOBTi07ENqmA9z6EPrOWc5SVippPoa4OS2FWMB_vj6MVaasHZq9WfvLHeWBotP2Zfq2O31BNJL-ngSW44pi9QvP8xnK0LiOeR3z3QxAxe-i0CLt2wag/w640-h86/nohostedparallelism.png" width="640" /></a></div><p>It seems that Microsoft has made some changes in <a href="https://devblogs.microsoft.com/devops/change-in-azure-pipelines-grant-for-private-projects/?WT.mc_id=DOP-MVP-5001511" target="_blank">Azure Pipeline Grants for Private projects</a> to prevent abuse as also outlined here <a href="https://learn.microsoft.com/en-us/azure/devops/pipelines/licensing/concurrent-jobs?view=azure-devops&tabs=ms-hosted" target="_blank">Configure and pay for parallel jobs</a>.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxvwPgjT0_3QwDELFJFEBR3jBB00YyNPQsNlgn_OhQmxMrZ_lcki8DHFY9tuvVjZdqnSUq-N3nOr6GE8p26uoO4K2oeYO4DdUN4P75_8DvCAh_2rdhcFTmPyg7UnnJKB1fvsfwVQJUxvwX0mSgT8UceRG3uFiHyryIIyfjxo4qhc0khQpCHQ/s955/azuredevops.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="568" data-original-width="955" height="238" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxvwPgjT0_3QwDELFJFEBR3jBB00YyNPQsNlgn_OhQmxMrZ_lcki8DHFY9tuvVjZdqnSUq-N3nOr6GE8p26uoO4K2oeYO4DdUN4P75_8DvCAh_2rdhcFTmPyg7UnnJKB1fvsfwVQJUxvwX0mSgT8UceRG3uFiHyryIIyfjxo4qhc0khQpCHQ/w400-h238/azuredevops.PNG" width="400" /></a></div><br /><p>To be able to use the free Microsoft-hosted built agent, you will now need to fill in the form which was mentioned in the exception message - <a href="https://aka.ms/azpipelines-parallelism-request">https://aka.ms/azpipelines-parallelism-request</a>. Microsoft completed this request in 3 business days for me - which was in line with</p><p><br /></p><p><br /></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-44204067394498304652023-03-10T14:47:00.007+01:002023-04-18T21:27:35.818+02:00Quick tip: update to Dataverse developer plan<p>Great update from Microsoft on the Dataverse developer plan - check out the<a href="https://www.youtube.com/watch?v=yXRSnN2AWTs" target="_blank"> new video from Power CAT team on Dataverse Environments for everyone - New Developer Plan - Power CAT Live on YouTube</a>.</p>
<p>Interesting updates in the new developer plan:</p><p></p><ul style="text-align: left;"><li>3 developer environments available per user</li><li>Don't take up storage capacity with a maximum of 2GB data.</li><li>Premium connectors included for testing without additional licenses</li><li>Possible to play around with ALM to get a to know solution management better.</li><li>Automatic cleanup of inactive environments after 90 days.</li></ul><div>If you don't have access to the PPAC, you will need to use https://make.preview.powerapps.com/ for the moment which shows the option to create new environments from within the environment selector.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZt-OcrKQ-k_CBGx6uelVWInyStCuIqCF2wwJZyUGMjal_jiLV07Q-J_k0g3B1z6I84A9E6n1J6506IG3bV6r_reIVSr5GGkcTuEf3oXjoottG5MTeh9zUVDMPi2kgB5Zao4Ts6n0VKZxx0lYNIjQWupoVCnc16g_k_23N6WhHojxoLEMQfQ/s580/makepreview.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="572" data-original-width="580" height="316" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZt-OcrKQ-k_CBGx6uelVWInyStCuIqCF2wwJZyUGMjal_jiLV07Q-J_k0g3B1z6I84A9E6n1J6506IG3bV6r_reIVSr5GGkcTuEf3oXjoottG5MTeh9zUVDMPi2kgB5Zao4Ts6n0VKZxx0lYNIjQWupoVCnc16g_k_23N6WhHojxoLEMQfQ/w320-h316/makepreview.PNG" width="320" /></a></div><br /><div><br /></div><div><br /></div><p></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-80801122139046111982023-03-09T21:00:00.002+01:002023-07-21T07:30:42.549+02:00Dynamics 365 and Power Platform monthly reading list March 2023<p><b> Power Platform and Dynamics 365 release 2023 wave 1</b></p><p></p><ul style="text-align: left;"><li><a href="https://experience.dynamics.com/releaseplans/?app=Power+Apps&planID=dd1bf68f-539c-ed11-aad1-000d3a4e5de0" target="_blank">Test engine to support model-driven apps (Public Preview May 2023)</a></li><li><a href="https://experience.dynamics.com/releaseplans/?app=Pro+development&planID=2bf7a19d-5a9c-ed11-aad1-00224827e576" target="_blank">Make solution export source code repository friendly (Public Preview March 2023)</a></li><li><a href="https://www.youtube.com/watch?v=qaPxK7uG3Jg" target="_blank">Dynamics 365 Wave 1 2023: Top 10 features that you need to know!</a> by <a href="https://twitter.com/LisaCrosbie" target="_blank">@LisaCrosbie</a> </li><li><a href="https://www.youtube.com/watch?v=4FLoF59Zp8I" target="_blank">Power Platform Wave 1 2023: Top 10 features that you need to know!</a> by <a href="https://twitter.com/LisaCrosbie" target="_blank">@LisaCrosbie</a></li><li><a href="https://powerapps.microsoft.com/en-us/blog/virtual-tables-creation-wizard-now-in-public-preview/" target="_blank">Virtual tables creation wizard now in Public Preview</a></li><li><a href="https://powerapps.microsoft.com/en-us/blog/the-new-look-for-model-driven-power-apps-is-available-now-in-preview/" target="_blank">The new look for model driven Power Apps is available now in preview</a></li><li><a href="https://www.youtube.com/playlist?list=PLcakwueIHoT9uZkRNxgZvOXIIzrHYe0FT" target="_blank">2023 Release Wave 1 YouTube Playlist </a></li></ul><p><b>Technical topics (Configuration, customization and extensibility)</b></p><p></p><ul style="text-align: left;"><li><a href="https://www.youtube.com/playlist?list=PLY78FEdXz6KSkWcT2wWT8CrDA9PhgyZpC" target="_blank">Zero to Hero 2022 - Power Platform Governance and ALM</a> by <a href="https://twitter.com/thevictordantas" target="_blank">@thevictordantas</a></li><li><a href="https://forwardforever.com/how-does-the-flow-cope-when-a-lot-of-functions-are-done/" target="_blank">How does Power Automate flow cope when a lot of actions are executed?</a> by <a href="https://twitter.com/timopertila" target="_blank">@timopertila</a></li><li><a href="https://lowcodelewis.com/blog/using-azure-key-vault-to-store-secrets-for-use-in-environment-variables" target="_blank">Using Azure Key Vault to store secrets for use in environment variables and flows</a></li><li><a href="https://www.youtube.com/watch?v=HAhObylsYgw" target="_blank">Introduction to Pipelines for Power Platform (YouTube) </a> by <a href="https://twitter.com/rezadorrani" target="_blank">@rezadorrani</a> and <a href="https://www.youtube.com/watch?v=7t6JxOjvZ9s" target="_blank">Build your first pipeline with Power Platform pipelines</a> by <a href="https://twitter.com/laskewitz" target="_blank">@laskewitz</a></li><li><a href="https://learn.microsoft.com/en-us/power-platform/alm/set-up-pipelines" target="_blank">Setup pipelines in Power Platform (Microsoft Learn)</a></li><li><a href="https://powervirtualagents.microsoft.com/en-us/blog/signing-up-for-a-developer-plan-for-power-virtual-agents/" target="_blank">Signing up for a developer plan for Power Virtual Agents</a></li><li><a href="https://powerapps.microsoft.com/en-us/blog/power-platform-build-tools-and-azure-devops/" target="_blank">Power Platform build tools and Azure DevOps</a></li><li><a href="https://powerapps.microsoft.com/en-us/blog/pipelines-in-power-platform-is-generally-available-ga/" target="_blank">Pipelines in Power Platform is Generally Available (GA)</a></li><li><a href="https://techcommunity.microsoft.com/t5/fasttrack-for-azure/azure-app-service-patterns-and-features-for-the-azure-well/ba-p/3696156" target="_blank">Azure App Service Patterns and Features for the Azure Well-Architected Framework</a></li><li><a href="https://www.inogic.com/blog/2023/01/adding-a-filter-for-lookup-columns-made-easier-with-advanced-lookup/" target="_blank">Adding a filter for lookup columns made easier with advanced lookup</a></li><li><a href="https://www.youtube.com/watch?v=WxNqT8OqdBE" target="_blank">Every Azure Synapse Analytics Engineer Needs to Know this Essential Dataverse Feature - Synapse Link (YouTube)</a></li><li><a href="https://www.youtube.com/watch?v=D-vqNn1vcvM" target="_blank">Power BI Dynamics 365 Sales templates with Scott Sewell</a></li><li><a href="https://www.youtube.com/watch?v=yXRSnN2AWTs" target="_blank">Dataverse Environments for everyone - new developer plan - Power CAT Live</a> by <a href="https://twitter.com/topness" target="_blank">@topness</a></li><li><a href="https://benediktbergmann.eu/2022/02/08/connections-and-connection-references-explained/" target="_blank">Connections and connection references explained</a> by <a href="https://twitter.com/BergmannBene" target="_blank">@BergmannBene</a></li><li><a href="https://temmyraharjo.wordpress.com/2023/01/28/model-driven-apps-how-to-use-the-side-pane-to-show-custom-html/" target="_blank">Model-driven apps: how to use the side pane to show custom html</a> by <a href="https://twitter.com/temmy_raharjo" target="_blank">@temmy_raharjo</a></li><li><a href="https://d365demystified.com/2023/01/31/enable-managed-environments-in-power-platform-admin-center/" target="_blank">Enable managed environments in Power Platform Admin Center</a> by <a href="https://www.twitter.com/priyesh_wngman7" target="_blank">@priyesh_wngman7</a></li></ul><p></p><p><br /></p><p><b>Topics for Dynamics 365 Business Applications Platform consultants, project managers and power users</b></p><p></p><ul style="text-align: left;"><li><a href="https://www.inogic.com/blog/2022/09/sales-forecasting-increase-accuracy-with-ai-predictive-forecasting/" target="_blank">Sales forecasting increase accuracy with AI predictive forecasting</a> by <a href="https://twitter.com/inogic" target="_blank">@inogic</a></li><li><a href="https://cloudblogs.microsoft.com/dynamics365/it/2022/12/06/add-voicemail-to-your-call-queue-management/" target="_blank">Add voice mail to your call queue management</a></li><li><a href="https://cloudblogs.microsoft.com/dynamics365/bdm/2022/12/08/the-key-to-delivering-experiences-that-customers-love/" target="_blank">Microsoft and Futurum research: Discover insights to create experiences that customers love</a></li><li><a href="https://community.dynamics.com/365/dynamics-365-fasttrack/b/dynamics-365-fasttrack-blog/posts/guidance-on-sales-hub-and-custom-model-driven-apps" target="_blank">Guidance on Sales Hub and custom model driven apps</a></li><li><a href="https://cloudblogs.microsoft.com/dynamics365/it/2023/01/27/create-a-digital-contact-center-in-seconds-with-the-customer-service-free-trial/" target="_blank">Create a Digital Contact Center in seconds with the Customer Service free trial</a></li></ul><p></p><p><br /></p><p><br /></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0tag:blogger.com,1999:blog-7753577.post-28560236242157494402023-01-25T20:14:00.005+01:002023-01-25T20:23:51.410+01:00Dynamics 365 and Power Platform 2023 release wave 1 details announced<p> The 2023 release wave 1 announcements are now available:</p><p></p><ul style="text-align: left;"><li><a href="https://learn.microsoft.com/en-us/dynamics365/release-plan/2023wave1/" target="_blank">Dynamics 365, Viva Sales and supply chain platform: 2023 release wave 1 plan</a></li><li><a href="https://learn.microsoft.com/en-us/power-platform/release-plan/2023wave1/" target="_blank">Microsoft Power Platform: 2023 release wave 1 plan</a></li></ul><div>From January 30th it will be possible to enable early access - see <a href="https://learn.microsoft.com/en-us/power-platform/release-plan/2023wave1/features-ready-early-access" target="_blank">Power Platform 2023 release wave 1 features available for early access</a> and <a href="https://learn.microsoft.com/en-us/dynamics365/release-plan/2023wave1/features-ready-early-access" target="_blank">Dynamics 365 release wave 1 features available for early access</a></div><div><br /></div><div><br /></div>
<div><iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/NxBI3nx3dfE" title="YouTube video player" width="560"></iframe></div><div><br /></div><div>Details are also available in the <a href="https://experience.dynamics.com/releaseplans/?app=Marketing" target="_blank">Dynamics 365 release planner</a> and <a href="https://experience.dynamics.com/releaseplans/?app=Power+Apps" target="_blank">Power Platform release planner</a>.</div><p></p>jopxhttp://www.blogger.com/profile/03901751169502117735noreply@blogger.com0