Why Kafka Consumer doesn't support Multithreaded Processing?

asked 2020-12-25 04:13:12 -0500

vat78 gravatar image

Hello! I need multithreaded processing for data pulled from Kafka. And unfortunately, I can't use Multitopic Kafka Origin because it has issues with offset commitment. It commits offset automatically by timeout without waiting for processing finishing. And in my case, it leads to lost records during the processing of a big amount of data. As I know there was an attempt to fix it in the latest versions but I am not sure that it was solved. Usual Kafka Consumer is good enough for me. But unfortunately, it is impossible to have multithreaded processing with it. And I have to use more instances of the pipeline to achieve appropriate performance. Is there any way to make multithreaded processing with the usual Kafka Consumer?

edit retag flag offensive close merge delete