Quantcast
Channel: Service Bus forum
Viewing all articles
Browse latest Browse all 1916

Flink Kafka connector to eventhub

$
0
0
0
<button aria-label="down vote" aria-pressed="false" class="js-vote-down-btn grid--cell s-btn s-btn__unset c-pointer" data-selected-classes="fc-theme-primary" style="margin:2px;box-sizing:inherit;font:inherit;padding:0px;border-width:initial;border-style:none;border-color:initial;border-radius:3px;background-image:none;background-background-size:initial;background-repeat:initial;background-attachment:initial;background-origin:initial;background-clip:initial;outline:none;box-shadow:none;" title="This question does not show any research effort; it is unclear or not useful"><svg aria-hidden="true" class="svg-icon m0 iconArrowDownLg" height="36" viewBox="0 0 36 36" width="36"><path d="M2 10h32L18 26z"></path></svg></button><button aria-label="favorite" aria-pressed="false" class="js-favorite-btn s-btn s-btn__unset c-pointer py8" data-selected-classes="fc-yellow-600" style="margin:0px;box-sizing:inherit;font:inherit;padding:0px;border-width:initial;border-style:none;border-color:initial;border-radius:3px;background-image:none;background-background-size:initial;background-repeat:initial;background-attachment:initial;background-origin:initial;background-clip:initial;outline:none;box-shadow:none;" title="Click to mark as favorite question (click again to undo)"><svg aria-hidden="true" class="svg-icon iconStar" height="18" viewBox="0 0 18 18" width="18"><path d="M9 12.65l-5.29 3.63 1.82-6.15L.44 6.22l6.42-.17L9 0l2.14 6.05 6.42.17-5.1 3.9 1.83 6.16z"></path></svg>
</button>

I am using Apache Flink, and trying to connect to Azure eventhub by using Apache Kafka protocol to receive messages from it. I manage to connect to Azure eventhub and receive messages, but I can't use flink feature "setStartFromTimestamp(...)" as described in Apache Flink documentation. When I am trying to get some messages from timestamp, Kafka said that the message format on the broker side is before 0.10.0. Is anybody faced with this? Apache Kafka client version is 2.0.1 Apache Flink version is 1.7.2

Tried to use Azure-Event-Hub quickstart examples in consumer package added code to get offset with timestamp, it returns null as expected if message version under 0.10.0 kafka version.

List<PartitionInfo> partitionInfos = consumer.partitionsFor(TOPIC);
        List<TopicPartition> topicPartitions = partitionInfos.stream().map(pi -> new TopicPartition(pi.topic(), pi.partition())).collect(Collectors.toList());
        Map<TopicPartition, Long> topicPartitionToTimestampMap = topicPartitions.stream().collect(Collectors.toMap(tp -> tp, tp -> 0L));
        Map<TopicPartition, OffsetAndTimestamp> offsetAndTimestamp = consumer.offsetsForTimes(topicPartitionToTimestampMap);
        System.out.println(offsetAndTimestamp);


Viewing all articles
Browse latest Browse all 1916

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>