kafka offset Management Not working

asked 2020-02-12 16:41:54 -0600

srinath_222 gravatar image


I have Kafka consumer source in my pipeline and target is Elasticsearch. If I stop and restart my pipeline, Kafka consumer should start reading messages from last stop point as per Streamsets documentation.

But my pipeline is reading all the data in that Kafka topic. Kafka Consumer origin is not setting the offset with Kafka.

Below is the screenshot from Kafka Consumer origin: image description

I tried reducing the batch size to 1 from 1000, but even then its reading all the messages from that topic.

Please advice.

edit retag flag offensive close merge delete