Ask Your Question
1

Kafka consumer offset management

asked 2017-12-20 13:01:26 -0600

Roh gravatar image

updated 2017-12-20 16:54:26 -0600

metadaddy gravatar image

I have a pipeline with Kafka consumer as origin and HDFS as a destination I need to understand if there is a way to do the offset management.

Scenario

I have the Kafka pipeline running until 10:00 AM, but for some reason, my pipeline has an issue and stops running. If I run the job at 12:00 PM it should start broadcasting the data from 10:00 AM. Is there a way to achieve this? Basically I need to understand if there is a way to play with Kafka offset values?

Example: in the JDBC consumer when there is a STOP on the pipeline and in the next START it will start picking up the data using the last offset value, and if we reset the origin it will forget the last offset. Something like this can be done in Kafka? I know there is no offset option for the Kafka consumer origin, but i want to know if there is any workarounds for it within the stream sets?

Version : StreamSets Data Collector 3.0.0.0

As the picture speaks more, here is what I have so far:

edit retag flag offensive close merge delete

Comments

@metadaddy & @jeff tagging you guys to get your attention. Thanks in advance !!

Roh gravatar imageRoh ( 2017-12-20 13:04:27 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2017-12-20 16:57:32 -0600

metadaddy gravatar image

updated 2017-12-22 10:59:08 -0600

Roh gravatar image

The Kafka Consumer documentation says:

By default, the origin reads only incoming data, processing data from all partitions and ignoring any existing data in the topic. After the origin passes data to destinations, it saves the offset with Kafka or ZooKeeper. When you stop and restart the pipeline, processing continues based on the offset.

So your pipeline should behave exactly as you describe. The first time you run it, by default, it will pick up data from when you started it. If it stops and restarts, it should read any data that was sent to the topic while the pipeline was stopped.

edit flag offensive delete link more

Comments

Thanks, @metadaddy I will recreate/validate if it's working as it is mentioned in the document.

Roh gravatar imageRoh ( 2017-12-21 09:34:32 -0600 )edit
1

Working like charm !! thanks again.

Roh gravatar imageRoh ( 2017-12-21 14:11:53 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-12-20 13:01:26 -0600

Seen: 390 times

Last updated: Dec 22 '17