System.TimeoutException while processing service bus message

(this issue has been moved from https://social.msdn.microsoft.com/Forums/azure/en-US/a62c4956-655e-4419-bf64-d17fee8243fe/systemtimeoutexception-while-processing-service-bus-message?forum=AzureFunctions)

I have developed an Azure Function, running on the consumption plan, consuming a message from a service bus topic and publishing a new message on another topic. I sometimes see the following exception in the log:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException:Exceptionwhile executing function:UptimeChecker--->System.InvalidOperationException:Errorwhile handling parameter msg after function returned:--->System.TimeoutException:The request operation did not complete within the allotted timeout of 00:00:51.9333128.The time allotted to this operation may have been a portion of a longer timeout.For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:5cf70cdf-5558-4f3c-9599-a47cc82f52d9, Timestamp:12/15/2017 6:27:23 PM
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<>c.b__9_3(RequestAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)---End of stack trace from previous location where exception was thrown ---
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Common.AsyncResult`1.End(IAsyncResult asyncResult)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.RequestAsyncResult.<>c__DisplayClass8_1.b__4(RequestAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)---End of stack trace from previous location where exception was thrown ---
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Common.AsyncResult`1.End(IAsyncResult asyncResult)
   at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<>c.b__9_3(RequestAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)...

The function app itself is a precompiled function app with the following signature:

publicstaticvoidRun([ServiceBusTrigger("%inputTopicName%","%subscriptionName%",AccessRights.Manage,Connection="connection")]string mySbMsg,TraceWriter log,[ServiceBus("%outputTopicName%",Connection="connection",EntityType=Microsoft.Azure.WebJobs.ServiceBus.EntityType.Topic)]outstring msg)

I have the same function app running in 5 different regions. Currently, I'm not in a position where I can see which region is failing in my error log. But it is one of these 5:

• elmah-io-uptime-checker-east-us
• elmah-io-uptime-checker-north-eu
• elmah-io-uptime-checker-southeast-au
• elmah-io-uptime-checker-south-uk
• elmah-io-uptime-checker-west-us

Application Insights show the error, but I don't think that it provides me with more details than the stack trace and error message I already have in my own log.

I did notice something in the original stack trace, though: "Error while handling parameter msg after function returned". The stack trace says "msg", which would translate to the out parameter in the signature. That also matches "after function returned" from the trace.

So, it seems like my code is setting a string in the out parameter and returns. When the job host (running after my code returns), needs to deliver the message to service bus, it fails. Since this happens outside the scope of my code, I am not able to add any retry logic. I also see the following term in the stack trace: "RetrySenderAsyncResult". This indicates that the ServiceBus output binding is already implemented to retry.

An option here is to remove the out parameter and deliver the message to service bus myself, using a more consistent retry strategy. But I would rather use the features build into functions, than to roll my own implementation.

Any ideas?