Ask Your Question

Setting paramaters INSIDE of pipeline

asked 2018-05-19 17:39:02 -0500

wildthought gravatar image

How can I set parameters from inside of a pipeline? I envision a Kafka Consumer Origin containing metadata about the file that has landed and its downstream pipeline parameters. Immediately after reading a Kafka Topic message, I want to read the message and assign parameters for the pipeline. These parameters would be contained within the Kafka Message itself.

Is it possible to have StreamSets read a single Kafka record and then use those values to set parameters within the instance of the current pipeline? Keep in mind that another instance of the pipeline would read the second record concurrently and set the same pipeline with different parameters than the first.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-05-21 12:40:23 -0500

metadaddy gravatar image

Pipeline parameters are currently read-only, but I think you might be able to achieve your objective by setting record header attributes. Attributes are useful for carrying metadata that you don't necessarily want to be written to the destination as part of the record.

You can set a record header attribute via the Expression Evaluator processor, and then access it via the ${record:attribute(<attribute name>)} expression language function.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-05-19 17:39:02 -0500

Seen: 602 times

Last updated: May 21 '18