Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Setting paramaters INSIDE of pipeline

How can I set parameters from inside of a pipeline? I envision a Kafka Consumer Origin containing metadata about the file that has landed and its downstream pipeline parameters. Immediately after reading a Kafka Topic message, I want to read the message and assign parameters for the pipeline. These parameters would be contained within the Kafka Message itself.

Is it possible to have StreamSets read a single Kafka record and then use those values to set parameters within the instance of the current pipeline? Keep in mind that another instance of the pipeline would read the second record concurrently and set the same pipeline with different parameters than the first.