We make use of the Azure Service Bus Relay in our product (version 2.3.4 of theWindowsAzure.ServiceBus NuGet package). One of our customers reported that the CPU on their server was pegged at 100%. We had the customer run a performance analysis using DebugDiag which showed call stacks for the problematic threads:
[[HelperMethodFrame]
System_ni!System.Net.Security._SslStream.BeginRead(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
Microsoft.ServiceBus.ServiceBusClientWebSocket
Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1[[System.__Canon, mscorlib]].EnumerateSteps(CurrentThreadType)
Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1[[System.__Canon, mscorlib]].StepCallback(System.IAsyncResult)
Microsoft.ServiceBus.Common.AsyncResult.AsyncCompletionWrapperCallback(System.IAsyncResult)
System_ni!System.Net.LazyAsyncResult.Complete(IntPtr)
System_ni!System.Net.Security._SslStream.StartFrameBody(Int32, Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)
System_ni!System.Net.Security._SslStream.ReadHeaderCallback(System.Net.AsyncProtocolRequest)
System_ni!System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32)
System_ni!System.Net.FixedSizeReader.ReadCallback(System.IAsyncResult)
System_ni!System.Net.LazyAsyncResult.Complete(IntPtr)
mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
System_ni!System.Net.ContextAwareResult.Complete(IntPtr)
System_ni!System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
[[GCFrame]
[[DebuggerU2MCatchHandlerFrame]
[[ContextTransitionFrame]
[[DebuggerU2MCatchHandlerFrame]
User Time 00:00:12.074 (+00:00:01.747)
Kernel time 00:00:05.943 (+00:00:00.749)
And:
System_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(System.Net.SafeCloseSocket, System.Net.WSABuffer ByRef, Int32, Int32 ByRef, System.Net.Sockets.SocketFlags ByRef, System.Runtime.InteropServices.SafeHandle, IntPtr)
[[InlinedCallFrame] (System.Net.UnsafeNclNativeMethods
System_ni!System.Net.Sockets.Socket.DoBeginReceive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.OverlappedAsyncResult)
System_ni!System.Net.Sockets.Socket.BeginReceive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef, System.AsyncCallback, System.Object)
System_ni!System.Net.Sockets.NetworkStream.BeginRead(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
System_ni!System.Net.FixedSizeReader.StartReading()
System_ni!System.Net.Security._SslStream.StartFrameHeader(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)
System_ni!System.Net.Security._SslStream.StartReading(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)
System_ni!System.Net.Security._SslStream.ProcessRead(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)
System_ni!System.Net.Security._SslStream.BeginRead(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
Microsoft.ServiceBus.ServiceBusClientWebSocket
Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1[[System.__Canon, mscorlib]].EnumerateSteps(CurrentThreadType)
Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1[[System.__Canon, mscorlib]].StepCallback(System.IAsyncResult)
Microsoft.ServiceBus.Common.AsyncResult.AsyncCompletionWrapperCallback(System.IAsyncResult)
System_ni!System.Net.LazyAsyncResult.Complete(IntPtr)
System_ni!System.Net.Security._SslStream.StartFrameBody(Int32, Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)
System_ni!System.Net.Security._SslStream.ReadHeaderCallback(System.Net.AsyncProtocolRequest)
System_ni!System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32)
System_ni!System.Net.FixedSizeReader.ReadCallback(System.IAsyncResult)
System_ni!System.Net.LazyAsyncResult.Complete(IntPtr)
mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
System_ni!System.Net.ContextAwareResult.Complete(IntPtr)
System_ni!System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
[[GCFrame]
[[DebuggerU2MCatchHandlerFrame]
[[ContextTransitionFrame]
[[DebuggerU2MCatchHandlerFrame]
User Time 00:00:08.642 (+00:00:00.811)
Kernel time 00:00:04.212 (+00:00:00.406)
From the call stacks, it looks like Microsoft.ServiceBus.ServiceBusClientWebSocket is the culprit.
This issue appears to be very similar to the following issue: http://stackoverflow.com/questions/23198377/high-cpu-on-an-idle-servicehost-connected-to-azure-service-bus-relay
In the above Stack Overflow question, the person asking the question says the following:
It turns out that the high CPU is being triggered by an unexpected ACK\FIN packet. We suspect the firewall is what actually sends this, trying to close off the external connection. We were able to recreate the issue on other devices simply by injecting
the rogue ACK\FIN packet.
We are following up with Microsoft Azure team to try to get them to better handle the unexpected packet. We will also be following up with the network firewall team to try to isolate and eliminate the packet from being sent.
We haven't gotten as far as doing a packet analysis with our customer yet...
Questions:
- Is this a known issue that the Microsoft Azure team is aware of?
- Are there any workarounds?
I was wondering if using the WebStream instead of WebSocket would workaround the issue, but haven't been able to find a way to force the WebStream to be used.
I was hoping that setting ServiceBusEnvironment.SystemConnectivity.Mode to ConnectivityMode.AutoDetect and adding the following appSetting to Web.config would force it to use the WebStream:
<add key="Microsoft.ServiceBus.OverrideAutoDetectMode" value="Https" />
But it still always connects using TCP to port 5761. If I block port 5761, it always uses WebSockets.
- Is there a way to force it to use WebStreams instead of WebSockets?
Thanks!