Ask Your Question

Revision history [back]

It sounds like data is being written to Kafka with 67 records per message. No matter how you set Max Batch Size, Data Collector cannot part-process a Kafka message. There's no way to tell Kafka 'I'm part way through processing this message', and, by design, Data Collector does not split Kafka messages into separate batches, since that would introduce a risk of data loss. You should look at the producer writing messages to Kafka, and see if it can write fewer records per message.