Dynamics 365, Azure webjobs, Azure Service Bus queues and deadletters

When you need to integrate Dynamics 365 (online) with other applications/services in on premise or in the cloud there are a number of challenges. One of the most commonly used integration patterns is messaging since it allows for loose coupling and it provides a reliable way of integration since both applications don’t necessarily have the same responsiveness or uptime. One of the key extensibility points from an integration perspective that you can leverage is the fact that you can connect Microsoft Dynamics 365 (both on premise and online) with the Azure Service  (See Azure integration with Dynamics 365). In an asynchronous messaging pattern, you will however still need a message processing component.  Since you are already leveraging cloud components, it might make sense to also use Azure for this message processing component.

There a number of Azure components you might take a look at:

• Azure cloud services: use an Azure worker role for message processing (for some background on this read Should I use cloud services or something else?)
• Azure app services: use  Azure web jobs for message processing.
• Azure virtual machine hosting the processing components
• Azure functions  - similar to Azure web jobs since it was built on the same code base as webjobs and has a similar API.
• Azure logic apps

If you look at the two first options (worker roles and web jobs), you will notice that they offer similar functionality payload – ability to run code repeatedly. However, there are some differences between the two - primarily in terms of environment customization. Web Jobs are good for running small tasks (not very computation heavy) and useful in scenarios where you do not have the need to depend on host OS features. Worker Roles can be used to run computation heavy workloads and their host OS can be modified a bit (by installing dependency components in OS through startup tasks). Worker roles are costly (similar to running a dedicated VM) while web jobs do not carry any additional cost. In case you need extreme customization on OS environment, then you would simply run a windows service inside custom OS image in Azure VM.  So basically you are looking at a trade off between control (and full flexibility) and agility combined with ease of management.

Now, if you look specifically at Azure webjobs, you will see that they can be triggered by a schedule, messages in Azure storage queue or blobs added to Azure storage and http call by calling the Kudu Webjobs API. But given the fact that Dynamics 365/RM already provides integration  with Azure Service Bus, this is the one that you will typically use – for a good walkthrough take a look at  How to use Azure Service Bus with the WebJobs SDK .

One of the things to keep in mind though, is that when an Azure webjob encounters an error in processing a message, it will retry processing it a specified number of times.  The number of retries is configured through the JobHostConfiguration.Queues.MaxDequeueCount property – see Azure webjobs and JobHostConfiguration for more details. This mechanism has been been built to avoid that a queue-based application gets stuck in a loop receiving and aborting the same message it can not process – this is also referred to as poison message handling.  So if this count is exceeded, the message will be move to the deadletter queue (when it is configured).

But the Azure service bus queues also have a property called MaxDeliveryCount (See QueueDescription class) which is by default set to 10 – so if this setting is lower than the setting in the JobHostConfiguration, your message might be moved to the deadletter queue (DLQ) earlier – see Overview of service bus dead-letter queues for more details.
If you open the deadletter queue with Azure Service Bus Explorer you will notice that these messages have additional properties called DeadLetterReason and DeadLetterDescription – these might be helpful in determining what causes the messages to end up in the DLQ. You can also move a message to the dead letter queue yourself by calling the BrokeredMessage.DeadLetter method.

Lessons learned:  If you are building an integration using Azure Service Bus queue, don't forget to think about how to handle messages which are deadlettered.


Technorati Tags: azure,dynamics365,service+Bus,messaging,application+integration,development