Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Issue with kafka consumer output batch size

I am facing issue with kafka consumer output batch size.

No matter what value I set to below properties of a kafka consumer Max Batch Size (records) Batch Wait Time (ms) I am always getting a batch size of 67. production.maxBatchSize is set to default(1000)

How can I change my output batch size. I want to run my pipeline with smaller batch size so that as soon as a record gets processed it doesn't wait for whole batch to get processed before going to next stage.

Issue with kafka consumer output batch size

I am facing issue with kafka consumer output batch size.

No matter what value I set to below properties of a kafka consumer consumer

  • Max Batch Size (records) (records)
  • Batch Wait Time (ms) (ms)

I am always getting a batch size of 67. production.maxBatchSize 67.

production.maxBatchSize is set to default(1000)

How can I change my output batch size. I want to run my pipeline with smaller batch size so that as soon as a record gets processed it doesn't wait for whole batch to get processed before going to next stage.