Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Is there a way to reindex error records to elasticsearch

Hi Guys, Can please someone help me with the below issue I am looking into...

I am indexing data from MySQL to elasticsearch constantly during this process when the queued tasks are more than the capacity of the queue, I am getting a stage error for few records with the below log message.

ELASTICSEARCH_16 - Could not index record ::rowCount:9': rejected execution of org.elasticsearch.transport.TransportService$7@e9a1f2c on EsThreadPoolExecutor[name = vm0pnelstsa0002/bulk, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@37a3005f[Running, pool size = 12, active threads = 12, queued tasks = 1000, completed tasks = 1861602043]]

Increasing the capacity of the queue may delay the cause of this problem, but this might not be a permanent solution. So I am thinking of something like constantly looking for records in the error stage and reindexing them. Is there a way to do this in streamsets or is there any alternate solution for this?

Thank you.