Issue with multiple kafka consumer groups reading from same topic

asked 2018-06-23 10:45:44 -0500

SUSER gravatar image

Hi,

we have a shared kafka topic that could contain messages for 40 different files. we have one pipeline per file that reads from this shared topic and writes to oracle database. we are using different consumer groups for each of these pipelines. At a given time, they all can read from same topic and write only the required information to database. EX: pipeline 1 consumes first 100 records from topic x and writes 10 records specific to table x , pipeline 2 consumes same first 100 records from topic x and writes 20 records specific to table y. The issue is, sometimes, when pipelines are run simultaneously, some of the pipelines are not reading data. Is it because data is already read by other consumer group? we are using different consumer groups for each of them, but still data is read once and second time it is not. Is there any other setting /change required to make it work as we expect for all pipelines? if I have different consumer groups, will the offset not be different for each of them and should they not be interfering each other?

edit retag flag offensive close merge delete