I have a queueClient that is configured to:
- Have 200 concurrentCalls
- Have a prefetch count of 100
I have a sendClient that places 10000 messages onto the queue with:
- A max TTL for the messages (5 mins)
- A max lease (5 mins)
Given a Func that stores the 'CompleteAsync' task and then returns this, I would expect that the function would be invoked *exactly* 10000 times, and all messages are completed.
Instead, I observe the following behavior:
- At minimum 10000 messages are received, though often ~100 more
- The additional messages are of the 10000 that were sent, this indicates the additional ones are "retries".
- The additional messages have a delivery count of 2, indicating they were indeed retried.
It would appear that Completing the message is not being fully honored. Indeed, when using the ExceptionReceived event handler on the queue client. A number of exceptions are being thrown with the message, "The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue". Given the length TTL and lease times, the lock cannot possibly have expired in the 20 seconds of runtime.
My working theory is that the concurrent calls and prefetched buffer is not being handled correctly, but I am at a loss of what to do now. I'm convinced given the simplicity of this bug that is an azure queue client specific issue.