A great starting point if you are new to Azure Synapse Link for Dataverse is Scott Sewell's YouTube playlist on Azure Synapse Link for Dataverse
Friday, April 28, 2023
Friday, April 21, 2023
SharePoint 2013 workflow retirement - the end of an era
Microsoft is (finally) retiring SharePoint workflows - the end of an era of workflows built on top of Microsoft Workflow Foundation (Workflow Foundation got introduced in 2005 - and yes I did play around with the beta of this back then - since I though it was the greatest thing since sliced bread )
Starting April 2d 2024 all SharePoint 2013 workflows will be turned off for newly created tenants. April 2d 2026 all SharePoint 2013 workflows will stop working for existing tenants.
Helpful links:
Tuesday, April 18, 2023
Looking at historical returns of stocks and bonds with Power BI and Python
You might already have seen below graph taken from a study by JP Morgan Asset Management, but what if you would like to look at historical returns without going through the hassle of having to collect all the data yourself?
There is an interesting Excel sheet shared by Aswath Damodaran (@AswathDamodaran) that you can download from Historical Returns on Stocks, Bonds and Bills: 1928-2022 which looks at returns of different asset classes (stocks, bonds, bills, real estate and gold) over a longer time period.
In this post I will share some tips on how you can use this data in Power BI, Python and Jupyter notebooks.
This Historical Returns on Stocks, Bonds and Bills: 1928-2023 - Excel file file is updated in the first two weeks of every year and it is being maintained by Aswath Damodaran, who is a professor of Finance at the Stern School of Business at NYU, he is also known as the "Dean of Valuation" due to his experience in this area.
- sheet_name: can be an integer (for the index of a worksheet in an Excel file, default to 0, the first sheet) or the name of the worksheet you want to load
- nrows: number of rows to read
- skiprows: number of rows to skip
- usecols: by default all columns will be read but also possible to pass in a list of columns to read into the dataframe like in the example
- Pandas – read_excel() – How to read Excel file in python
- Musings on markets: a return to teaching - the Spring 2023 Edition
- Guide to the Markets | JP Morgan Asset Management
- Interactive chart visualizations using Python and bqplot: visualizing S&P500 returns
- Using the yFinance Python package to download financial data from Yahoo Finance - part 2
- Using the yFinance Python package to download financial data from Yahoo Finance
- Using Python and Pandas Datareader to retrieve financial data - part 2: Fama & French data library
- Using Python and Pandas Datareader to retrieve financial data - part 1: Federal Reserve Data (FRED)
Wednesday, March 22, 2023
Upgrade to Azure Functions runtime v 4.x
Even though the Azure Functions runtime 2.0 and 3.0 were retired on December 13th, 2022 , I recently still found some Azure Functions using these older runtimes. See Azure Functions runtime versions overview which outlines the support for the different version.
Azure Functions that use runtime version 2.x or 3.x will continue to run but Microsoft is no longer providing updates or support for applications that use these versions. There a number of reasons why you should upgrade:
- Leverage the support for the latest programming languages such as .NET 6 and Node 16. Take the opportunity to upgrade your code to .NET 6 since .NET 6 includes a lot of performance improvements
- To make sure that you keep getting security updates
- Be future proof and keep technical debt to a minimum (See Technical debt and Dynamics 365 solution architecture for my view on this)
What is the Azure Functions runtime?
For those not so familiar with Azure Functions, the Azure Functions runtime is the engine that powers Azure Functions to execute user-defined code (written in . It is responsible for loading the user's code, executing it in response to specific triggers, and scaling the execution as needed to handle the workload. The runtime is a key component of the Azure Functions service, and it allows developers to focus on writing and deploying their code, rather than worrying about the underlying infrastructure.
How to find the Azure Functions runtime version?
When you log in to the Azure Portal and go to the Function runtime settings tab on the configuration page of an Azure function, you will see which runtime version is in use. Instead of having to look at every individual function, you might also take a look at the Azure Resource Graph
Monday, March 20, 2023
Notes and links for the DP-900 Microsoft Azure Data Fundamentals
- Features comparison: Azure SQL Database and Azure SQL Managed Instance
- Azure SQL Database security features
- Aggregate functions (Transact-SQL)
- SQL Tools overview: Azure Data Studio, SQL Server Management Studio (SSMS), SQL Server Data Tools (SSDT) and mssql extension for VS Code and command line tools
- What is Azure Data Studio?
- Clustered and non-clustered indexes described
Friday, March 17, 2023
Quick tip: use the new release planner for Dynamics 365 and Power Platform to track and planning upcoming changes
The new release planner for Dynamics 365 and Power Platform is awesome - check it out at https://aka.ms/releaseplanner . It allows you to track the features that are planned, coming soon or already available for trial. You have visibility across active release waves in a portal (built with Power pages) and you can personalize, filter and sort release plans when you sign in.
Thursday, March 16, 2023
Quick tip: resolving no hosted parallelism has been purchased or granted in Azure DevOps
I recently setup a new free Azure DevOps organization but when I tried running my first pipeline - I got an exception "##[error]No hosted parallelism has been purchased or granted. To request a free parallelism grant, please fill out the following form https://aka.ms/azpipelines-parallelism-request".
It seems that Microsoft has made some changes in Azure Pipeline Grants for Private projects to prevent abuse as also outlined here Configure and pay for parallel jobs.
To be able to use the free Microsoft-hosted built agent, you will now need to fill in the form which was mentioned in the exception message - https://aka.ms/azpipelines-parallelism-request. Microsoft completed this request in 3 business days for me - which was in line with
Friday, March 10, 2023
Quick tip: update to Dataverse developer plan
Great update from Microsoft on the Dataverse developer plan - check out the new video from Power CAT team on Dataverse Environments for everyone - New Developer Plan - Power CAT Live on YouTube.
Interesting updates in the new developer plan:
- 3 developer environments available per user
- Don't take up storage capacity with a maximum of 2GB data.
- Premium connectors included for testing without additional licenses
- Possible to play around with ALM to get a to know solution management better.
- Automatic cleanup of inactive environments after 90 days.
Thursday, March 09, 2023
Dynamics 365 and Power Platform monthly reading list March 2023
Power Platform and Dynamics 365 release 2023 wave 1
- Test engine to support model-driven apps (Public Preview May 2023)
- Make solution export source code repository friendly (Public Preview March 2023)
- Dynamics 365 Wave 1 2023: Top 10 features that you need to know! by @LisaCrosbie
- Power Platform Wave 1 2023: Top 10 features that you need to know! by @LisaCrosbie
- Virtual tables creation wizard now in Public Preview
- The new look for model driven Power Apps is available now in preview
- 2023 Release Wave 1 YouTube Playlist
Technical topics (Configuration, customization and extensibility)
- Zero to Hero 2022 - Power Platform Governance and ALM by @thevictordantas
- How does Power Automate flow cope when a lot of actions are executed? by @timopertila
- Using Azure Key Vault to store secrets for use in environment variables and flows
- Introduction to Pipelines for Power Platform (YouTube) by @rezadorrani and Build your first pipeline with Power Platform pipelines by @laskewitz
- Setup pipelines in Power Platform (Microsoft Learn)
- Signing up for a developer plan for Power Virtual Agents
- Power Platform build tools and Azure DevOps
- Pipelines in Power Platform is Generally Available (GA)
- Azure App Service Patterns and Features for the Azure Well-Architected Framework
- Adding a filter for lookup columns made easier with advanced lookup
- Every Azure Synapse Analytics Engineer Needs to Know this Essential Dataverse Feature - Synapse Link (YouTube)
- Power BI Dynamics 365 Sales templates with Scott Sewell
- Dataverse Environments for everyone - new developer plan - Power CAT Live by @topness
- Connections and connection references explained by @BergmannBene
- Model-driven apps: how to use the side pane to show custom html by @temmy_raharjo
- Enable managed environments in Power Platform Admin Center by @priyesh_wngman7
Topics for Dynamics 365 Business Applications Platform consultants, project managers and power users
- Sales forecasting increase accuracy with AI predictive forecasting by @inogic
- Add voice mail to your call queue management
- Microsoft and Futurum research: Discover insights to create experiences that customers love
- Guidance on Sales Hub and custom model driven apps
- Create a Digital Contact Center in seconds with the Customer Service free trial
Wednesday, January 25, 2023
Dynamics 365 and Power Platform 2023 release wave 1 details announced
The 2023 release wave 1 announcements are now available:
- Dynamics 365, Viva Sales and supply chain platform: 2023 release wave 1 plan
- Microsoft Power Platform: 2023 release wave 1 plan
Thursday, January 19, 2023
Technical debt and Dynamics 365 solution architecture
Technical debt is an important concept to understand as a solution architect. As outlined in Thoughts about the Dynamics 365 solution architect role, a solution architect has to keep the long-term use of the solution in mind and implement scalability and adaptability into the solution for the future. This is where you will need to understand the concept of technical debt. Software design and development is full of metaphors to help non-technical people understand specific concepts - technical debt is one of them.
What is technical debt?
Technical debt is created by (un)conscious actions to take shortcuts in development or design of a software solution. These shortcuts are taken due to some short-term benefits such as faster delivery time, easier implement etc. ... but they have an impact on the longer term.
Having technical debt is like having financial debt, where time spent on this "not-quite-right implementation" counts as interest on the debt and paying the interest on the loan (the debt) might slow down the project/program in the long term.
You will see 3 types of technical debt in projects:- Strategic technical debt: technical debt caused by decisions made to shorten time to market, delay development or expenses, ...
- Unavoidable technical debt: technical debt accumulated due to the fact that the platform/technology/product version might not be supported anymore, ...
- Naïve technical debt: due to business immaturity (unclear or changing requirements, misalignment between product owner and business stakeholders) or technical immaturity (no coding standards, no peer reviews, no automated tests, no refactoring, no consideration for)
It is important to grasp that technical debt is not always an indicator of low quality (but it might be). Technical debt happens because even though we do the best we can - given what we know - we subsequently learn better ways to solve a problem or better ways appear due to technical improvements in the underlying platform or technology stack. You probably heard the phrase "Ignorance is bliss" - well for technical debt this is not the case. The fact that nobody in a team has heard of technical debt should be treated as a red flag for software quality.
How to repay technical debt?
Regardless of the type of technical debt you accumulated, in the end you will always need to repay the technical debt to make sure that you don’t jeopardize your product evolution. If the cost to implement a certain change (be it a bug fix or adding an additional feature) keeps increasing over time, it might be an indicator that you have incurred technical debt on your project.
The first step is however making business aware of the existence of technical debt and the impact of it. I really like the ideas suggested in The Wall of Technical Debt on how to make technical debt visible. Building a regular habit of identifying technical debt and learning on how to negotiate with business on technical debt payoff is a good approach.
Keeping technical debt to a minimal is a team effort - make sure that delivering quality is engrained in the culture and way of working of the team.How to identify technical debt in your Dynamics 365 solution architecture?
As mentioned previously technical debt is automatically accumulated as changes are happening on the technical platform underpinning your solution. So I highly recommend to follow up on the important changes (deprecations) in Power Apps and Power Automate page and also on the Deprecation of Dynamics 365 apps page.
By writing code, you are automatically accumulating some amount of technical debt but there is tooling available to help you identify technical debt in your code and solutions:
- SonarQube - static code analysis tool which helps you identify bugs and code smells - works for multiple languages. Quite useful for .NET and Javascript development which are typically used in CRM development
- Power Apps solution checker - specific Power Platform tooling to analyze Dataverse or Dynamics CRM solutions. Can be run from the maker portal, Microsoft Power Platform Build tools for Azure DevOps or PowerShell
- Checkmarx provides static application security testing (SAST and open source composition analysis (SCA). It helps you uncover security issues in your code base or in 3d party libraries/packages that you are using. If you are looking for alternatives, you might take a look at the vulnerability scanning tools listing from OWASP
References:
- Allen Holub on Technical Debt
- What is technical debt and why is it so painful?
- Introduction to technical debt and technical debt quadrant by Martin Fowler
- Technical debt for CIO leads to organization debt for CEOs
- Is technical debt a management problem? Survey says ...
- How to manage technical debt | Lucidchart
- We invested 10% to pay back technical debt, here is what happened?
- How technical debt can cripple a company when the perfect storm hits - comment on the Southwest Airlines systems meltdown
- Armchair architects: architectural erosion and technical debt
- How to address tech debt vs product debt vs business debt
- How to manage your technical backlog
- From technical debt to technical health with healthcheck
- Defining, measuring and managing technical debt (IEEE publication, April 2023) - article based on the findings of Google engineering teams. Google uses quarterly engineering surveys to ask engineers about the types of technical debt they encounter and which hinder their work
- Debt factor - beyond technical debt
Tuesday, January 10, 2023
Interactive chart visualizations using Python and bqplot: visualizing S&P500 returns
A couple of months ago, I stumbled upon this interesting presentation Jupyter Notebooks: interactive visualization approaches. The presentation showed how you can use bqplot to build interactive visualizations.
Bqplot contains a set of 2D plotting widgets built on top of the ipywidgets framework for Jupyter notebooks. The bqplot package aims to bring d3.js visualizations to Python while retaining the flexibility and ease of use of ipywidgets and was developed by the quantitative research team at Bloomberg. You can install bqplot using conda or pip.
One of the examples built by the team that you can find on Github is a Jupyter notebook which shows US equity market performance (using the S&P 500 index) where you can select an interval on a time series chart - for the selected area you get the total return as well as a histogram of the daily returns.
References:
Monday, January 09, 2023
Why I blog and you might want to consider it as well
A couple of weeks ago I got the question why I write a blog since it would be better to do a podcast or publish videos on technical content. In this post, I will explain why it is still a good idea to start blogging. I wrote my first blog post almost 18 years ago and got inspired by Robert Scoble (Scobleizer) who worked as a technical evangelist at Microsoft at that time and had a very popular blog. 1483 blog posts and almost 2 million views of these blogs I still consider it a good idea, since I think that a lot of good things have happened to me in my professional live because I decided to start my blog and share my thoughts on technology.
Starting a blog is an easy starting point for you to be active in the technical community. Sharing is caring might sound lame but it is a good credo to live by as an IT professional/consultant. In a professional context, most persons only know the people that they directly work with - colleagues or customers.
Blogging (but also Twitter and technical communities) allows you to expand that and suddenly there are people all over the world that you can learn from and they can learn from you. Suddenly you will realize there are others who face similar challenges. By sharing your experiences and offering advice to support, you can build a community of readers who value your experience. By writing good high-quality content on a regular basis, you can establish yourself as an expert in your field which in turn will help you build your personal brand and increase your visibility. Starting a blog is a lot easier than creating a video on YouTube or doing a podcast.
The most underrated skill to have in IT is the ability to communicate efficiently. Blogging is levelling that skill up. In an era where people mostly rely on PowerPoints slides for presenting a plan or a vision, it is quite refreshing to see them back it up with a well-written, solid document afterwards. I usually cringe when seeing poorly written documents or e-mails so I think that good writing skills will set you apart.
I think of a blog as a way to write notes to myself and you should too. Throughout your daily job, you are continuously solving problems: write about how you approached solving these problems and where you found helpful information. By writing your approach down, you will remember it better and it will be easier for you to find it later (and others will benefit of this as well).
The more you explain a concept or a topic to other people, the better you will get at understanding it yourself. This might sound strange but it is true, teach to learn. When you need to teach something, you are required to understand it in a deep and comprehensive way. The process of writing, will help your own understanding and requires you to think critically and find creative ways to present the topic which further enhances your understanding and retention of the information.
Even when you are learning something new, you are probably taking notes. Blog these notes, taking these notes. Rewriting them will help you understand the content much better, while it helps others too. If you think that everything is already written, do it for your own reference. If you find it useful yourself to go back to your blog, then it is OK. Don't fuzz to much about getting little visitors to your blog in the beginning.
References:
Thursday, January 05, 2023
Dynamics 365 and Power Platform monthly reading list December 2022
First post of 2023 - so let's close with last month's suggested reading list. If you want to catch up on previous reading lists for 2022 - here is the list:
- Dynamics 365 and Power Platform monthly reading list September 2022
- Dynamics 365 and Power Platform monthly reading list July 2022
- Dynamics 365 and Power Platform monthly reading list May 2022
- Dynamics 365 and Power Platform monthly reading list April 2022
- Dynamics 365 and Power Platform monthly reading list February 2022
- Dynamics 365 and Power Platform monthly reading list January 2022
Technical topics (Configuration, customization and extensibility)
- Dataverse: simple advice on how to design your plug-in by @temmy_raharjo
- Match contacts and accounts by email domain with Power Automate
- Setting as a solution component in Power Apps by @inogic
- Dataverse group teams with Azure AD dynamic user membership groups
- Introducing Test Engine: an open platform for automated testing of canvas apps
- Adaptive cards in Power Apps
- Adaptive card designer overview (Preview) in Power Apps
- Create a card with data from Dataverse (preview)
- Build powerful Adaptive Card experiences for Teams using Cards for Power Apps by @michelcarlo
- Complete a Dataverse Task by an adaptive card in Teams
- Setup Azure DevOps projects for ALM accelerator for Power Platform
- Granular billing for Azure Data Factory
- 2022 Release Wave 2: Timeline maker enhancements
- Announcing two new white papers that dive deep into Power Pages architecture and security
- Streamline low code governance in your governance (Ignite 2022 recording BRK41)
- How to setup and connect to Dataverse environment in Postman
- A few notes on the Timeline: model-driven Power Apps form tweaks
- Power Platform Capacity Monitoring by @TheCRMNinja
- Top 25 Power Apps Tips, Tricks and Best Practices for Makers by @rezadorrani
Topics for Dynamics 365 Business Applications Platform consultants, project managers and power users
- Enable AI generated conversation summary (Preview) by @D365Goddess
- E-mail validation (Preview) in Dynamics 365 Sales
- Utilizing the swarming feature in Teams (Preview) for intelligent collaboration within Dynamics 365
- Leverage low-code to do more with less at Microsoft Ignite 2022
- Live from Microsoft Ignite 2022: Introducing new AI, automation and collaboration capabilities for Dynamics 365
- Duplicate lead detection increases sellers' productivity
- The future of IT with Charles Lammana - Power CAT Live (YouTube)
- Power BI modeling guidance for Power Platform
- How we can correct the mistakes made with Agile (YouTube)
- Exploring sales usage reporting feature in Dynamics 365 CE sales analytics
- Unleash your Dynamics 365 data with Azure Synapse Analytics and Power BI (YouTube recording)
- Price points of Power Platform (2022 pricing and licensing) by @jukkan
- One Dynamics One Platform - Introduction (FastTrack TechTalks)
- Discover new ways to analyze your customer service data
- Customer Service voice support channel launches in the UK (November 2022)
Monday, January 02, 2023
Notes on deploying and troubleshooting a Streamlit app on Azure App Services
using Azure App Services using the guidance from Deploying Streamlit Applications with Azure App Services .
- Viewing Azure App Service web server and application logs in real time on Azure Portal
- 30 days of Streamlit challenge
- Streamlit vs Dash vs Shiny vs Voila vs Flask vs Jupyter - comparing data dashboarding tools and frameworks
- Turn an Excel Sheet into an interactive dashboard using Python (Streamlit) -YouTube
- Streamlit app showcasing Plotly graphs
- Stuck learning Python? Make it fun with Streamlit!
Monday, November 21, 2022
Reminders for the occasional Power BI creator
I am not a regular Power BI user, but every couple of months I will create some Power BI reports (e.g., Storage capacity management for Dynamics 365/Dataverse ) because Power BI is simply great to visualize and analyze data.
Unfortunately, I tend to forget - maybe very obvious things to frequent Power BI users - how to perform specific tasks in Power BI.
So here it goes - my list of reminders for the occasional Power BI creator.
- Set the column to use for sorting a visual in Power BI
- Tutorial: create your own measures in Power BI Desktop
- You can create a blank table and type in data or just copy paste/data from Excel by clicking the table icon in the Power BI home tab (see image below)
- Apply conditional formatting in tables and matrixes
- Set a locale or region for data in Power Query - when you work in an international context, you might encounter multiple date formats and number formats, it is good to remember that you can specify a locale when importing data.
- Great set of Power BI visuals/diagrams explaining Power BI permissions for content creators and owners, Power BI Datamarts, etc .. from @SQLChick
- How to create a waffle chart in Power BI with a custom visual
Wednesday, November 16, 2022
Visualize S&P 500 data in Power BI using Azure Synapse Serverless SQL Pool
In Explore and analyze stock ticker data in Azure data lake with Azure Synapse serverless SQL Pool, I showed you can download stock ticker data from Yahoo Finance, stored it in Azure Data Lake and retrieve the data using standard T-SQL in Azure Synapse Studio. In this post, I will show how easy it is to consume the data from Synapse SQL Serverless using Power BI.
For the standard visual with the evolution of the S&P 500 closing price, I connected directly on SP500 external table in the Synapse SQL. You can connect to Synapse SQL Serverless using either the Azure SQL Database or Azure Synapse Analytics SQL connector and you will need to enter the Serverless SQL endpoint which looks something like this <yoursynapse>-ondemand.sql.azuresynapse.net
With the second reported I want to visualize the S&P 500 yearly return and the average return since December 1927. To make it easier, I created a separate view on top of the external table which calculates the yearly returns
You can download the synapsestockdemo.pbix file and the benchmark.csv file from my Power BI repo on GitHub
References:
Monday, November 14, 2022
Azure functions with Python: a getting started guide
In this post, we'll learn what Azure Functions are, and how you can use VS Code to write your first Azure Function in Python code.
I will show how you can create a simple Azure Function which retrieves data from Yahoo Finance (See Using Python and Pandas Datareader to retrieve financial data - part 3: Yahoo Finance) and saves the retrieved data in a CSV file in Azure blob storage. I will be using the Python v1 programming model for Azure Functions since v2 is still in preview.
Introduction to Serverless and Azure Functions
More traditional forms of cloud usage require you to provision virtual machines in the cloud, deploy your code to these VMs, manage resource usage and scaling, keep the OS up to date and the underlying stack, setup monitoring, perform backups, etc...
But if you just want to deploy some piece of code which needs to handle some kind of event, serverless compute might be the right choice for you. With serverless compute, you can develop your applications, deploy it to the serverless service like Azure Functions and you don't need to worry about the underlying hosting architecture. Serverless compute is most of the time cheaper than PAAS or IAAS hosting models.
Several versions of the Azure Functions runtime are available - see Languages by runtime version for an overview which languages are supported in each runtime version. Python 3.7, 3.8 and 3.9 are supported by Azure Functions v2, v3 and v4.
How to create an Azure Function using Azure Portal
You can deploy an Azure Function from your local machine to Azure without leaving VS Code, but I would recommend doing it first using the Azure Portal to understand what VS Code is doing behind the scenes.
To create your Azure Function, click the Create a resource link on the Azure Portal home page and next select Function App.
This brings us to the function creation screen, where we have to provide some configuration details before our function is created:
- Subscription: Azure subscription in which you want to deploy your Azure Function App
- Resource group: container that holds related resources for an Azure solution - these resources typically share the same development lifecycle, permissions and policies, ...
- Function App Name
- Runtime stack: Python
- Version: choose 3.9 (latest supported version) unless you have specific Python version dependencies.
- Region: choose the same region as other resource that you need to deploy e.g., blob storage, Cosmos DB, etc. ...
- Operating system: only Linux is supported
- Plan type: leave it to Consumption (Serverless) unless you have very specific requirements with regards to execution time limit higher than 10 minutes (see Azure functions scale and hosting - function app timeout duration for more details)
- Azure subscription
- Azure Functions Core Tools version 4.x
- Supported Python version (minimum Python 3.6 or a distro like Anaconda or Miniconda
- Visual Studio Code
- Python extension for Visual Studio Code
- Azure Functions extension for Visual Studio Code
- function.json - configuration file for our function
- sample.dat - sample data to test the function
- __init__.py - main file with the code that our function runs

- local.settings.json: stores app settings and connection strings when running locally
- requirements.txt: list of Python packages the system installs when publishing to Azure
- host.json: configuration options that affect all functions in a function app instance
- .venv: folder which contains the Python virtual environment used by local development.
References:
- Quickstart: Create a function in Azure with Python Using Visual Studio Code
- Status of Python versions - info on the Python language version support policy timeline
- Serverless Python Applications with Azure Functions - YouTube recording Build 2019
- An introduction to web scraping with Python and Azure Functions (YouTube)












