We have an application that runs on theMicrosoft Azure
cloud platform. The communication between some components is made using Service Bus.
Everything was working fine, until recently we started to get the following type of timeout exceptions:
When callingQueueClient x.Send(...)
Exception rethrown at [0]:
at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult result)
at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<GetAsyncSteps>b__4(RequestAsyncResult thisPtr, IAsyncResult r)
at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
When callingNamespaceManager x.GetQueue(...)
PROGRESS queue processing failed. System.TimeoutException: The request has timed out after 60000 milliseconds. The successful completion of the request cannot be determined. Additional queries should be made to determine whether or not the operation has succeeded. TrackingId:bdffb6bd-5367-4573-aaa3-8ea9a03f5a2b,TimeStamp:5/28/2015 8:39:46 AM ---> System.Net.WebException: The request was aborted: The request was canceled.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.GetAsyncResult`1.<GetAsyncSteps>b__49(GetAsyncResult`1 thisPtr, IAsyncResult r)
at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
When callingNamespaceManager x.SubscriptionExists(...)
Exception doing periodic work: System.TimeoutException: The request has timed out after 00:10:00 milliseconds. The successful completion of the request cannot be determined. Additional queries should be made to determine whether or not the operation has succeeded.
Server stack trace:
Exception rethrown at [0]:
at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.NamespaceManager.OnEndSubscriptionExists(IAsyncResult result)
at Microsoft.ServiceBus.NamespaceManager.SubscriptionExists(String topicPath, String name)
...
When callingQueueClient x.Receive(...)
PROGRESS queue processing failed. Microsoft.ServiceBus.Messaging.MessagingCommunicationException: Error during communication with Service Bus. Check the connection information, then retry. ---> System.ServiceModel.CommunicationObjectFaultedException: Internal Server Error: The server did not provide a meaningful reply; this might be caused by a premature session shutdown. TrackingId:04ba0220-0350-4806-9c65-c2bba9671054, Timestamp:28.05.2015 13:00:55
Server stack trace:
Exception rethrown at [0]:
at Microsoft.ServiceBus.Common.ExceptionDispatcher.Throw(Exception exception)
at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult result)
at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.EndRequest(IAsyncResult result)
at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<GetAsyncSteps>b__4(RequestAsyncResult thisPtr, IAsyncResult r)
at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
….
The exceptions are clearly related to ServiceBus and they are non-deterministic. The functions throwing them, e.g., Send, GetQueue, SubscriptionExists, are called no more than 100-120 times per minute. We changed nothing in the code and increasing the timeout values (even to ridiculously high values, like 10min) did not help. Also, we do not believe it is some network related problem (on our side) since the same error occur when the application is run from different places.
Has anyone else encountered recently these kind of exceptions? Is there a problem on the side of Microsoft or we are missing something?