Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

You have two options. First, you can increase the size of your Elasticsearch cluster's bulk operation queue, so that it's large enough to handle your expected load. This is an ES level setting, completely outside the scope of Data Collector. See here for more information.

Secondly, you could implement a buffer (preferably using Kafka, which is well suited for cases like this). Your current pipeline origin will send records to Kafka in one pipeline. Then, another pipeline will read from Kafka and then send to your current Elasticsearch destination. This will allow you to more effectively throttle the pipeline batches/rate while still not losing any updates.