Hi,
I am working on sending around 1000 msg/s to an Azure Event Hub with 16 partitions and 5 TU's from an Apache Storm Cluster (Java). What we have noticed is a single EventHubClient can't keep up with this throughput so we have decided to use a pool of clients, however if we create a pool of 10 clients we are running into an odd situation where the client code freezes and does not respond. Does this have something to do with the number of TCP connections we're opening?
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-eventhubs</artifactId>
<version>0.7.8</version>
</dependency>
This is the version we're using but we have also tried 0.9.0 with no luck.
Here is an example of how we're sending, the interface for the client is just a wrapper around the EventHubClient for unit testing purposes:
@Overridepublic void sendLooping(List<Object> list) {
list.stream().parallel().forEach((object) -> {
ITmaticEventHubClient ehc = null;
try {
ehc = this.pool.borrowObject();
ehc.send(gson.toJson(object).getBytes(StandardCharsets.UTF_8));
}
catch (Exception e) {
throw new RuntimeException("Unable to send batch of messages to the event hub", e);
}
finally {
this.pool.returnObject(ehc);
}
});
}