Every once in a while (1 in thousands of messages) I am getting a MessageLockLostException when trying to complete a message.
I have a queue defined with lock duration set to 30s:
In code I create a listener with the following options:
var messageHandlerOptions = new MessageHandlerOptions(exceptionReceivedHandler)
{
MaxConcurrentCalls = 10,
AutoComplete = false,
MaxAutoRenewDuration = TimeSpan.FromSeconds(180)
};
var queueClient = ResolveQueueClientByQueueConfigName(queueConfigName);
_logger.Info($"Listening on queue {queueClient.ServiceBusConnection.Endpoint}{queueClient.QueueName}");
queueClient.RegisterMessageHandler(MyMessageHandler, messageHandlerOptions);And in MyMessageHandler I complete the message after successful processing:
private static async Task CompleteAsync(IReceiverClient queueClient, string lockToken)
{
await queueClient.CompleteAsync(lockToken);
}
private async Task MyMessageHandler(Message message, CancellationToken cancellationToken)
{
//Processing omitted
await CompleteAsync(queueClient, message.SystemProperties.LockToken);
}Like I said, this works fine for most of the messages, but once in a while I get the following:
2020-03-27 10:31:01,004 [64] INFO Received message: MessageId:7da4f7ec-9e27-4951-83f5-7f29d8fc93a8
2020-03-27 10:31:01,005 [64] INFO Entity Framework Core "3.1.0" initialized '"SmsDataContext"' using
provider '"Microsoft.EntityFrameworkCore.SqlServer"' with options: "None"
2020-03-27 10:31:01,087 [109] INFO Sending message to smssender, CorrelationId: 7da4f7ec-9e27-4951-83f5-7f29d8fc93a8, label: SmsSender
2020-03-27 10:31:05,205 [158] INFO Completing message '7da4f7ec-9e27-4951-83f5-7f29d8fc93a8'2020-03-27 10:31:05,206 [158] ERROR Message handler encountered an exception
Microsoft.Azure.ServiceBus.MessageLockLostException: The lock supplied is invalid. Either the lock
expired, or the message has already been removed from the queue, or was received by a different
receiver instance.
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.DisposeMessagesAsync(IEnumerable`1 lockTokens,
Outcome outcome)
at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.CompleteAsync(IEnumerable`1 lockTokens)
at Bcp.AzureServiceBusHelper.ServiceBusClientHelper.CompleteAsync(String queueName, Message message)
at Bcp.Sms.Validator.ServiceHost.HostedService.ProcessMessagesAsync(Message validatorMessage,
CancellationToken token) in ***
at Microsoft.Azure.ServiceBus.MessageReceivePump.MessageDispatchTask(Message message). Context:
Endpoint= prdhybrid.servicebus.windows.net; Entity Path: smsrequestvalidation; Executing Action:
UserCallback2020-03-27 10:31:30,949 [109] INFO Received message: MessageId:7da4f7ec-9e27-4951-83f5-7f29d8fc93a8
So in this case the lock was taken at 10:31:01 and the message complete failed 4 seconds later at 10:31:05, well within the 30 second lock duration.
The last log line indicates the message was retried after 30 seconds at 10:31:30 which tells me that the lock should have been valid at 10:31:05.
Could it be that the lock contained in message.SystemProperties.LockToken is corrupted? Also, could this be due to a short network interruption? I would imagine the LockToken stays valid in this case.
This application is written in .net core 3.1 using Microsoft.Azure.ServiceBus 4.1.2
Thanks in advance for the help!