Ask Your Question

Is there a way to not commit offset to kafka when there is an error in a processor?

asked 2019-07-11 06:29:40 -0500

Nuno Passos gravatar image


I want to build a pipeline that reads events from kafka, checks the DB if the event has already been received, and if it doesn't exist, writes to the DB and process the event.

Right now, my problem is that I need to make sure that I process all the events that I receive from kafka, but if by any chance the DB is unnavailable, the record will not be processed, but the kafka offeset will be commited.

Is there any way to make sure that Kafka Consumer does not commit the offset if there was an issue during the processing?

Thank you

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2019-07-12 17:24:03 -0500

metadaddy gravatar image

Set the JDBC Producer configuration to stop the pipeline on an error and it will not commit an offset back to Kafka:

image description

You can also set the pipeline configuration to automatically retry the pipeline when it stops on an error:

image description

With the default value, -1, Data Collector retries the pipeline an infinite number of times. This allows the pipeline to continue running as soon as any temporary connection or system failures resolve.

If you want the pipeline to stop after a given number of retries, you can define the maximum number of retries to perform.

The wait time between retries begins at 15 seconds and doubles in length until reaching a maximum of 5 minutes.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-07-11 06:29:40 -0500

Seen: 204 times

Last updated: Jul 12