Kafka origin not processing messages produced when pipeline is stopped, upon startup

asked 2020-06-29 03:32:43 -0500

Himanshu gravatar image

I am using StreamSets Data Collector 3.1.3.0 and have many pipelines with Kafka origin. Every time a pipeline is stopped for some reason and I start it again, the Kafka messages which were generated when the pipeline was not running won't be picked up upon startup.

I am unable to attach screenshots, but I would be happy to share it if there is a way to do it.

Kafka Library is streamsets-datacollector-apache-kafka_0_11-lib

edit retag flag offensive close merge delete

Comments

StreamSets Data Collector 3.1.3.0 or StreamSets Data Collector 3.13.0?

iamontheinet gravatar imageiamontheinet ( 2020-06-29 13:40:42 -0500 )edit

It's 3.1.3.0, the docker image we are pulling is streamsets/datacollector:3.1.3.0

Himanshu gravatar imageHimanshu ( 2020-07-01 10:51:07 -0500 )edit

That version is pretty old. You should definitely upgrade to the latest/GA version 3.16.1.

iamontheinet gravatar imageiamontheinet ( 2020-07-02 09:45:01 -0500 )edit