Azure Service Bus queue is deleted when using relay binding

I'm experiencing a weird issue when using relay bindings with Azure Service Bus.
I'm running the code below

static void Main(string[] args)
{
  ServiceHost sh = new ServiceHost(typeof(RemoteService));
  sh.AddServiceEndpoint(
    typeof(IServiceEndpointPlugin), new NetTcpRelayBinding(), AppSettings["SBusEndpoint"])
    .Behaviors.Add(new TransportClientEndpointBehavior
    {
      TokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider(AppSettings["SharedAccessKeyName"], AppSettings["SharedAccessKey"])
    });

  sh.Open();

  Console.WriteLine("Press ENTER to close");
  Console.ReadLine();

  sh.Close();
}

"SBusEndpoint" is a pre-created service bus queue endpoint
"SharedAccessKeyName" is the name of the SAS key, and the "SharedAccessKey" is the key value.

As long as the program runs everything works fine, messages are received and it outputs fine, but as soon as I stop the program (both by running the Close() method and by stopping the debug session) the queue is removed from the Azure portal.
What makes it even more strange is that the queue still exists, but all of the shared access policies are gone. So if I use a shared access key defined for the bus itself then I can reconnect without any issues (the queue is still not visible from the portal, though), but if I used a shared access policy defined on the queue then that is no longer valid and throws an error on sh.Open()

If I try to re-create the queue I get a "GatewayTimeout" error saying "The gateway did not receive a response from 'Microsoft.ServiceBus' within the specified time period.", so the namespace is clearly still reserved even though the validation says the namespace is valid (green check).