Issue with kafka consumer output batch size
I am facing issue with kafka consumer output batch size.
No matter what value I set to below properties of a kafka consumer
- Max Batch Size (records)
- Batch Wait Time (ms)
I am always getting a batch size of 67.
production.maxBatchSize
is set to default(1000)
How can I change my output batch size. I want to run my pipeline with smaller batch size so that as soon as a record gets processed it doesn't wait for whole batch to get processed before going to next stage.