When we use eventhubs we have a serious problem that events are sent out of order, 1 batch goes before another batch. This happens mainly if there are 2 paralel ampq connections. Most likely the ampq sends data and cofirms it's OK but it's not yet propagated to the partitions. (Disclaimer the partition ID is set correctly). The multiple connections can arise in following situations:
- Send Batch, Disconnect/Fast reconnect/ Send batch
- I have a stateless service that is replicated S[1,2,3,..]. I have a client C that invokes something on S. This operations has a side effect that it sends something to eventhub. If C sends something to S(1), waits for confirm, then sends something to S(2) quickly (despite waiting for confirm of all operations), different ampq connections are used for eventhub (as each S replica has its own AMPQ connection) and messages come out of order. The C invokes S must be done quickly, some load balancer randomly selects S(1,2,3,...). We are using linkerd and grpc.
I replicated this issue in golang and python. I would attach a test case but I am unable to because of your forum policy. This is a serious problem with event streaming which I hope will be fixed as soon as possible. (Also I know this is Service Bus forum but I couldnt find eventhub topic)