When a kafka consumer commit the offset.

asked 2020-04-27 03:06:36 -0500

RaviPrakash gravatar image

updated 2020-04-27 03:23:12 -0500

We have one data process pipeline and one kafka consumer pipeline. Data process pipeline process multiple (10) files. The data pipeline sends a kafka notification for every finished file. So it is sending 10 kafka notification. The Kafka consumer consume these notification.

Scenario 1 Everything is working fine when data is less.

Scenario 2 The problem starts when the file size is more for some files (around 300MB each for 3 files and 1MB to 50MB for other 7 files).

We are getting error in kafka consumer pipeline log Offset commit failed on partition. And Kafka consumer is getting same notification multiple times. We know that duplicate notification is coming because kafka consumer is not able to commit the offset.

Q1. Why we are getting this error only for huge data?
Q2. Since auto commit is not wokring how can we do manual commit in javascript evaluator?
Q3. By default when a kafka consumer commit an offset in the pipline.
Q4. If kafka consumer read a message and if we send the message to Trash, will kafka consumer commit the offset for the message?

edit retag flag offensive close merge delete