Ask Your Question

Is it possible to set offset manually for Kafka Consumer?

asked 2019-10-08 04:32:43 -0500

Gill Bates gravatar image

During development it is useful to make Kafka Consumer or Kafka Multitopic Consumer to read the same message from the same offset over and over again while making changes to the pipeline under development. I wonder - is it possible to do through the origin configuration or SDC configuration somehow?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2019-10-08 19:09:57 -0500

metadaddy gravatar image

Yes - in the Kafka Consumer or Kafka Multitopic Consumer you can set Auto Offset Reset to 'Timestamp' and then specify a starting timestamp in Auto Offset Reset Timestamp. Now, when you reset the pipeline offset, it will read from the specified timestamp - for more info see the doc section on Offset Management.

Note - if you just stop and start the pipeline, it will pick up where it left off. You must reset the origin to read from the specified timestamp.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-10-08 04:32:43 -0500

Seen: 764 times

Last updated: Oct 08 '19