Ask Your Question

Can pipeline stages communicate bidirectionally?

asked 2018-10-01 16:19:13 -0500

SKR gravatar image

updated 2018-10-01 19:40:08 -0500

metadaddy gravatar image

We have a TCP origin which sends records to a Mongo DB stage and a Kinesis stage, in short we have 3 stages in our pipeline. After receiving data from the external source, the TCP origin should send data to the Mongo DB stage and the Kinesis stage before responding back. Can we achieve this functionality with StreamSets? It seems the moment that control is sent to the next pipeline stage, the origin loses any control on the data flow.

edit retag flag offensive close merge delete


Hi - I edited your answer a bit to tidy up the terminology

metadaddy gravatar imagemetadaddy ( 2018-10-01 19:47:01 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2018-10-01 19:46:33 -0500

metadaddy gravatar image

In general, data cannot be passed back along the pipeline. The one exception is Microservice pipelines that use the REST Service origin: some destinations can return data that will be sent back to the REST client via the HTTP response.

Please file an issue if you would like to see similar functionality added for the TCP Server origin.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-10-01 16:19:13 -0500

Seen: 242 times

Last updated: Oct 01 '18