Every once in a while (1 in thousands of messages) I am getting a MessageLockLostException when trying to complete a message.
I have a queue defined with lock duration set to 30s:
In code I create a listener with the following options:
var messageHandlerOptions = new MessageHandlerOptions(exceptionReceivedHandler) { MaxConcurrentCalls = 10, AutoComplete = false, MaxAutoRenewDuration = TimeSpan.FromSeconds(180) }; var queueClient = ResolveQueueClientByQueueConfigName(queueConfigName); _logger.Info($"Listening on queue {queueClient.ServiceBusConnection.Endpoint}{queueClient.QueueName}"); queueClient.RegisterMessageHandler(MyMessageHandler, messageHandlerOptions);
And in MyMessageHandler I complete the message after successful processing:
private static async Task CompleteAsync(IReceiverClient queueClient, string lockToken) { await queueClient.CompleteAsync(lockToken); } private async Task MyMessageHandler(Message message, CancellationToken cancellationToken) { //Processing omitted await CompleteAsync(queueClient, message.SystemProperties.LockToken); }
Like I said, this works fine for most of the messages, but once in a while I get the following:
2020-03-27 10:31:01,004 [64] INFO Received message: MessageId:7da4f7ec-9e27-4951-83f5-7f29d8fc93a8 2020-03-27 10:31:01,005 [64] INFO Entity Framework Core "3.1.0" initialized '"SmsDataContext"' using provider '"Microsoft.EntityFrameworkCore.SqlServer"' with options: "None" 2020-03-27 10:31:01,087 [109] INFO Sending message to smssender, CorrelationId: 7da4f7ec-9e27-4951-83f5-7f29d8fc93a8, label: SmsSender 2020-03-27 10:31:05,205 [158] INFO Completing message '7da4f7ec-9e27-4951-83f5-7f29d8fc93a8'2020-03-27 10:31:05,206 [158] ERROR Message handler encountered an exception Microsoft.Azure.ServiceBus.MessageLockLostException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue, or was received by a different receiver instance. at Microsoft.Azure.ServiceBus.Core.MessageReceiver.DisposeMessagesAsync(IEnumerable`1 lockTokens, Outcome outcome) at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout) at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout) at Microsoft.Azure.ServiceBus.Core.MessageReceiver.CompleteAsync(IEnumerable`1 lockTokens) at Bcp.AzureServiceBusHelper.ServiceBusClientHelper.CompleteAsync(String queueName, Message message) at Bcp.Sms.Validator.ServiceHost.HostedService.ProcessMessagesAsync(Message validatorMessage, CancellationToken token) in *** at Microsoft.Azure.ServiceBus.MessageReceivePump.MessageDispatchTask(Message message). Context: Endpoint= prdhybrid.servicebus.windows.net; Entity Path: smsrequestvalidation; Executing Action: UserCallback2020-03-27 10:31:30,949 [109] INFO Received message: MessageId:7da4f7ec-9e27-4951-83f5-7f29d8fc93a8
So in this case the lock was taken at 10:31:01 and the message complete failed 4 seconds later at 10:31:05, well within the 30 second lock duration.
The last log line indicates the message was retried after 30 seconds at 10:31:30 which tells me that the lock should have been valid at 10:31:05.
Could it be that the lock contained in message.SystemProperties.LockToken is corrupted? Also, could this be due to a short network interruption? I would imagine the LockToken stays valid in this case.
This application is written in .net core 3.1 using Microsoft.Azure.ServiceBus 4.1.2
Thanks in advance for the help!