Dynamics 365, Azure webjobs, Azure Service Bus queues and deadletters

When you need to integrate Dynamics 365 (online) with other applications/services in on premise or in the cloud there are a number of challenges. One of the most commonly used integration patterns is messaging since it allows for loose coupling and it provides a reliable way of integration since both applications don’t necessarily have the same responsiveness or uptime. One of the key extensibility points from an integration perspective that you can leverage is the fact that you can connect Microsoft Dynamics 365 (both on premise and online) with the Azure Service  (See Azure integration with Dynamics 365). In an asynchronous messaging pattern, you will however still need a message processing component.  Since you are already leveraging cloud components, it might make sense to also use Azure for this message processing component.

There a number of Azure components you might take a look at:

• Azure cloud services: use an Azure worker role for message processing (for some background on this read Should I use cloud services or something else?)
• Azure app services: use  Azure web jobs for message processing.
• Azure virtual machine hosting the processing components
• Azure functions  - similar to Azure web jobs since it was built on the same code base as webjobs and has a similar API.
• Azure logic apps

If you look at the two first options (worker roles and web jobs), you will notice that they offer similar functionality payload – ability to run code repeatedly. However, there are some differences between the two - primarily in terms of environment customization. Web Jobs are good for running small tasks (not very computation heavy) and useful in scenarios where you do not have the need to depend on host OS features. Worker Roles can be used to run computation heavy workloads and their host OS can be modified a bit (by installing dependency components in OS through startup tasks). Worker roles are costly (similar to running a dedicated VM) while web jobs do not carry any additional cost. In case you need extreme customization on OS environment, then you would simply run a windows service inside custom OS image in Azure VM.  So basically you are looking at a trade off between control (and full flexibility) and agility combined with ease of management.

Now, if you look specifically at Azure webjobs, you will see that they can be triggered by a schedule, messages in Azure storage queue or blobs added to Azure storage and http call by calling the Kudu Webjobs API. But given the fact that Dynamics 365/RM already provides integration  with Azure Service Bus, this is the one that you will typically use – for a good walkthrough take a look at  How to use Azure Service Bus with the WebJobs SDK .

One of the things to keep in mind though, is that when an Azure webjob encounters an error in processing a message, it will retry processing it a specified number of times.  The number of retries is configured through the JobHostConfiguration.Queues.MaxDequeueCount property – see Azure webjobs and JobHostConfiguration for more details. This mechanism has been been built to avoid that a queue-based application gets stuck in a loop receiving and aborting the same message it can not process – this is also referred to as poison message handling.  So if this count is exceeded, the message will be move to the deadletter queue (when it is configured).

But the Azure service bus queues also have a property called MaxDeliveryCount (See QueueDescription class) which is by default set to 10 – so if this setting is lower than the setting in the JobHostConfiguration, your message might be moved to the deadletter queue (DLQ) earlier – see Overview of service bus dead-letter queues for more details.
If you open the deadletter queue with Azure Service Bus Explorer you will notice that these messages have additional properties called DeadLetterReason and DeadLetterDescription – these might be helpful in determining what causes the messages to end up in the DLQ. You can also move a message to the dead letter queue yourself by calling the BrokeredMessage.DeadLetter method.

Lessons learned:  If you are building an integration using Azure Service Bus queue, don't forget to think about how to handle messages which are deadlettered.


Technorati Tags: azure,dynamics365,service+Bus,messaging,application+integration,development

Dynamics 365 monthly reading August 2017

Technical topics (Configuration, customization and extensibility)

• Preventing folder names with GUIDs in SharePoint (Dynamics CRM/Dynamics 365 for Enterprise)
• Important changes coming in Dynamics 365 Customer Engagement
• Dynamics 365 Monthly update – July 2017 (CRM in the field – information from Dynamics PFE team)
• Dynamics 365 Monthly update – June 2017 (CRM in the field – information from Dynamics PFE team)
• Change createdby/modifiedby field at runtime using Dynamics CRM plugin
• Running Dynamics 365 Configuration Data mover jobs in Azure Functions 
• Deprecation of Dynamics 365 for Outlook
• New in V9: what’s that with Xrm.Page being deprecated?
• Analyzing Customer Insights data with Power BI
• XrmToolbox: the tools you need to get the job done (Youtube 00:52:46)
• Update Dynamics CRM Client SDK version to avoid calls to ACS
• Using Server to Server (S2S) authentication to connect your application to Dynamics 365
• Best practices when writing Dynamics CRM plugins
Using ILMerge for Dynamics CRM Plugins
• How to edit Dynamics 365 apps with the App Designer and Site Designer
• Async is smooth, smooth is fast – moving to async calls in Dynamics 365
• Where to store configuration data in Dynamics 365/CRM but also read Using Azure Key Vault to store configuration data which provides a more secure way of storing sensitive configuration data
• How to use Dynamics CRM/365 optionset labels within Power BI
• Auditing automatically disabled with unmanaged solution deployment in Dynamics 365
• Tips on submitting a Dynamics 365 App to Microsoft AppSource
• How to become Microsoft Dynamics 365 Consultant
• Refreshing web resources and iframes – the supported way

Topics for business analysts, power users and end users

• You cannot stop Dynamics 365 updating but you can be prepared
• Coming soon : Dynamics 365 for Sales, Business edition, designed for small-to-medium sized business
• 6 steps to creating a killer sales culture
• Dynamics 365 July Update 2017: Unified interface & customer service hub
• Dynamics 365 User Interface Refresh
• The evolving expectation on customer experience
• Dynamics 365 – Relationship Insights
• Relationship Insights (Youtube 00:05:35)
• Video guide to chat & co-browsing for Dynamics 365
• Comparing the new Dynamics 365 for Sales Business Edition with Enterprise Edition
• Activity timelines in Dynamics 365

Voice of the Customer for Dynamics 365

In March 2015, Microsoft announced the acquisition of Mojo Surveys – a customer feedback solution which was built solely for Microsoft Dynamics CRM (both online and on-premise).  This product was re-engineered and rebranded to Voice of the Customer (VoC). VoC was initially made available in preview in January 2016 – Preview feature: Voice of the Customer surveys and became generally available in the December 2016 release (VoC solution version 8.3.x and higher). At the moment VoC is only available for Dynamics CRM/365 (online) and apparently there are no plans to make it available for integration with on premise Dynamics CRM/365.

Voice of the Customer adds rule-based, multi-channel feedback surveys integrated with workflows, campaigns and e-mails on top of Dynamics 365. These surveys can be used in a range of scenarios from service feedback to focus groups to sales scenarios. Voice of the Customer allows you design and publish surveys from within Dynamics 365, distribute them to your target audience. Respondents can take part in a survey in different ways, depending on how you sent the invitation out, whether you want anonymous or non-anonymous responses and whether you want to integrate data from CRM into the survey. Responses can be incorporated into a business process by creating response routing rules. Voice of the Customer is available for installation as a preferred solution from the Applications tab in the Dynamics 365 Administration center.

From Voice of the Customer version 9.0 onward, you immediately receive a confirmation screen for accepting the license terms and for activating VoC when configuring it from within the Applications tab.

In previous versions, you first had to trigger the installation and afterwards go to the solution and accept the licensing terms from within the solution screen.

In the previous version, you could also Retrigger response processing to troubleshoot when you did not see responses coming in see Where are my Voice of the Customer responses? for more details. It is unclear where this setting has gone in version 9.x and higher.

Before you install the solution it is important to realize that behind the scenes Voice of the Customer uses a number of different Azure components such as Azure blog storage, Azure web and worker roles, Azure Keyvault, Azure SQL  see Install the Voice of the Customer solution for more details about this. These components are subject to other privacy and compliancy governance policies – for more information see Azure Trust Center

It is quite easy to get started with VoC and the steps are thoroughly outlined in Create a Voice of the Customer survey to get customer feedback . In previous version, I noticed that sometimes the survey definition got corrupted so it is important to quite regularly click on “Preview survey”. If you get an error like “Isv code aborted the operation”, it is best to delete the last question that you added, save the survey, run the preview and re-create the survey question.

Other good walkthroughs you can take a look at are A Step by Step Guide to Create Your First Survey with Dynamics CRM 2016 Voice of the Customer as well as Voice of the Customer for Dynamics CRM 2016 (configuration screens have changed a little bit from 9.x onwards but these articles are still worth taking a look at). One of the things, I also liked about the surveys is the fact that they are also mobile-friendly

Managed metadata column values not visible in SharePoint view

I recently encountered a problem with views in a SharePoint site collection – the values of the managed metadata column were not visible anymore for most users in the list views. However  when the user edited the item or viewed the details of an item – they could see the value. Apparently this issue was caused as a side effect of migrating Windows users to claim users. For performance reasons, managed metadata (or taxonomy items) are stored at site collection level in a hidden list called TaxonomyHiddenList which is accessible at the root level of a site collection e.g. http://servername/sites/sitecollection1/lists/taxonomyhiddenlist.  The list by default has read access to all Authenticated users but when performing the claims migration the security was mixed up, by giving all authenticated users read permission again, the issue was resolved.

Technorati Tags: SharePoint,troubleshooting,taxonomy,managed+metadata

Free CRMUG Belux webinar – What’s new in Dynamics 365 Spring 2017 release

By now, you have probably all read about the upcoming Dynamics 365 spring 2017 release. If you want to learn more about it register for the free CRMUG Belux webinar – What’s new in Dynamics 365 Spring 2017 release . This June, Microsoft announced the next Dynamics 365 release, called the “Spring 2017” release wave. During a 3-day (!) long preview session, Microsoft’s product team took a deep dive into the new capabilities that are coming to the platform, such as:

• An updated user interface, providing a uniform experience for users across web, app and mobile,
• New features in Dynamics 365 Customer Insights that leverage data across and beyond Dynamics CRM and machine learning capabilities (Also check out Introduction to Dynamics 365 Customer Insights to get started)
• New social selling capabilities, a direct result of the LinkedIn acquisition
• New features around scalability and extensibility, -
• The release of the Business Edition, a sales and marketing app triggered towards small and medium businesses.

Your CRM-UG BeLux team has managed to bundle all these exciting announcements in a 1-hour lunch webinar for you on August 28th. Our presenters, Microsoft’s Philip Verlinden and Dieter De Cock, have even promised there will be time for Q&A.

Introduction to Dynamics 365 for Customer Insights (DCI)

Dynamics 365 for Customer Insights (DCI) was introduced in December 2016 and has recently received some interesting updates in the Dynamics 365 Spring update.  DCI is an Azure-based SAAS service which enables organizations to bring together transactional, web and social customer data to visualize it in a Customer 360°. DCI also allows you apply advanced data transformation algorithms (KPI calculation and machine learning) on this data to generate role-specific insights for different roles within an organization. 

The key differentiators within Dynamics 365 for Customer Insights are:

• It provides a scalable data infrastructure which can handle millions of interactions, entities and relationships between entities. Building such an infrastructure can be a daunting task and will consume a lot of upfront investment in  IT capacity. By providing a platform where you only pay for the data that you need to handle, you will be able to shift investment to resources and more value added tasks such as integrating the intelligence within your existing business processes
• Out of the box connector for Dynamics 365 for Customer Engagement (CRM) which allows you to rapidly get going starting from the existing effort that you put into building your CRM platform
• Ability to match customer identities in different systems using conflation algorithms and predictive matching – this allows to merge/cluster customer identities at scale.
• Built-in advanced machine learning applications such as segmentation – which identifies cohorts of users with the same characteristics or at the same stage of the customer journey – and built-in predictive scoring. Business users will be able to define the business outcome they want to predict (e.g. customer churn = customer for which status becomes inactive) and DCI will automatically create a machine learning model and validate it (behind the scene it will do feature selection, machine learning selection, training and validation,…). Prediction scores will be automatically generated for new entities or they will be recalculated for entities for which one of the signals/characteristics impacting the predictive score will change. This allows you to get up and running quickly with a machine learning solution without hiring data scientists.
• Extensible platform – ability to ingest data from multiple data source by building your own connectors and ability to expose DCI entities,metrics and predictions through Power BI, Power Apps or using the REST API

Customer Insights is leveraging the same components which are also used within the Cortana Intelligence Solutions but they are taking it a number of steps further by providing out of the box connectors, configurations screens, ingest and egress paths as well as built-in visualize layer. You however have less control on the inner workings of the Customer Insights solution since this is a SAAS solutions whereas most Cortana Intelligence Solutions combine a SAAS, IAAS and PAAS platform components.

One of the customer cases that you should definitely take a look at is Marston’s and Customer Insights - Marston, is a brewing company operating more than 1700 pubs in the UK and was one of the early customers that started using this application.  In order to create a personalized pub experience for their customers and achieve a more targeted marketing they wanted to bring together customer data from social media such as Facebook, Twitter, and TripAdvisor. They also wanted to look at Wi-Fi registration data, point of sale transactions, table bookings and survey results to understand their customers.  This required a level of data integration and analysis that goes beyond just CRM alone and that’s were Customer Insights comes in. Shown below is an example of dashboard within DCI, which is tailored towards a house manager of a Marston’s inn.

Word of advice: Customer Insights for Dynamics 365 is currently in preview – available functionality is subject to change and both pricing and product naming might change before the final release. From a licensing perspective, we know see two tiers as described on the Dynamics 365 Customer Insights pricing page. Pricing is based on the amount of data which needs to be processed as well as KPIs used, predictive match policies and predictive score models.

References:

• Dynamics 365 for Customer Insights – Getting Started
• Using connectors to simplify data ingestion at scale in Dynamics 365 for Customer Insights
• Predicting business outcomes with Customer Insights (Blog post)
• Predicting business outcomes with Customer Insights (Youtube 00:01:53)
• Dynamics 365 for Customer Insights: July 2017 refresh

Introducing Cortana Intelligence Solutions

It has been a couple of months since I last looked at Cortana Intelligence Solutions (CIS) in the Cortana solution gallery so I was pleasantly surprised by a number of new solution templates which have been added.  These solution templates target different problem domains in different industries and are leveraging the full breadth of the Microsoft data platform offering – some of the scenarios covered are:

• Predictive Maintenance for Aerospace - this Predictive Maintenance solution monitors aircraft and predicts the remaining useful life of aircraft engine components. Uses Azure HDInsight, Azure Machine Learning, Azure Event Hubs, etc … the different components are outlined in a solution diagram which is provided by the solution template as well.
• Campaign optimization with SQL Server 2016 - demonstrates how to build and deploy a machine learning model with SQL Server 2016 with R Services to recommend actions to maximize the purchase rate of leads targeted by a campaign.
• Vehicle telemetry analytics - this solution demonstrates how car dealerships, automobile manufacturers and insurance companies can use the capabilities of Cortana Intelligence to gain real-time and predictive insights on vehicle health and driving habits.

Cortana Intelligence Solutions is just one of the pieces in the Cortana Intelligence Suite next to Power BI, Cognitive Services, Microsoft Bot Framework and a whole lot of Azure cloud components such as Azure Machine Learning, Azure HDInsight, Azure SQL Data Warehouse, Azure Data Factory and many more.

Cortana Intelligence Suite provides both a platform and  process guidance (Team Data Science Process) to perform advanced analytics from start to finish. The main goal of Cortana Intelligence Solutions  is to reduce the complexity of deploying advanced data analytics solutions by providing Azure building blocks to operationalize your data science process. By providing these building blocks, Microsoft tries to enable companies to make smarter decisions at a rapid pace without having to worry about the complexity of designing, deploying and operating scalable data architectures.
I truly believe in the potential of data to be able to truly transform a business, a point which has been backed by the Capturing the $1.6 Trillion Data Dividend  white paper which showed that data leaders have a competitive advantages over their peers. These data leaders have a number of characteristics in common in the way that they invest in data:

• Leverage new data such as linking transactional data with customer behavior, sensor data, social data, mobile data eetc…
• Expanded use of analytic techniques – including more predictive analytics and big data processing techniques such as Mapreduce
• Use new metrics – e.g. new ways of looking at operations or measuring performance
• Include new users – expand the number and type of users who have access to the organization’s data and analytics

The interesting part of the Cortana Intelligence Suite is that it also describes the Team Data Science Process lifecycle which provides a recommended lifecycle which can be used to structure data science projects (see Data science for the rest of us for a good starting point of your new to the field of data science).

A key point that I can’t stress enough is the fact that you need a solution understanding of the business context. Data science projects are typically not IT-driven, business should be in the lead. Data science is a team effort where close cooperation with business is required to understand and identify the business problems. The most complex part of the data science process is formulating the  questions that define the business goals and that data science techniques can target. Most companies agree that managing customer churn is important, but how you define churn is different in each industry and also the impact of churn of specific customers might be different e.g in telco customers are influenced by both friends within the network and friends of friends (See Using social network analysis to predict churn which used data from a Belgian telco provider as well as the research paper Mining telecommunication networks to enhance customer lifetime predictions)

References:

• Cortana Intelligence Solutions (James Serra)
• Cortana Intelligence Suite Overview (Channel 9 video)
• Cortana Intelligence Suite Retail Industry Solution – How to guides