Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

kafka offset Management Not working


I have Kafka consumer source in my pipeline and target is Elasticsearch. If I stop and restart my pipeline, Kafka consumer should start reading messages from last stop point as per Streamsets documentation.

But my pipeline is reading all the data in that Kafka topic. Kafka Consumer origin is not setting the offset with Kafka.

Below is the screenshot from Kafka Consumer origin: image description

I tried reducing the batch size to 1 from 1000, but even then its reading all the messages from that topic.

Please advice.