[Data collector 3] JMS to ELK, pipeline transaction

asked 2018-09-19 14:26:31 -0500

Mathieu gravatar image


I come to you with a question about the management of transactions

My need

My pipeline is very simple, i consume JMS Messages from a queue on my Tibco EMS serveur. The messages contains JSON that is directly push to Elasticsearch. (So my pipeline have only 2 stages) It works, very fast !

But, if the Elasticsearch cluster goes down or was to long to respond, the stage goes on Error.

The problem

The messages that were read in the pipeline that failed to integrate the data are still consumed and are no longer present in the queue ...

I study the Data Collector JMS Origin source code, and i found that JMS session support transaction and publish to streamset an interface with take, commit and rollback methods.

I just want the JMS Consumer stage rollback if the ELK post stage fails... Is it possible with my Streamset Data Collector 3 ?

I know that i can push errors records to other pipeline, ... but i really want to do all with one pipeline... I have this problem with a lot of pipelines and i don't want to duplicate all...

I thank you in advance for your answers,

Mathieu F. from Lille in France ;)

edit retag flag offensive close merge delete