Ask Your Question

can we configure to run the pipeline dynamically ? [closed]

asked 2019-05-22 15:51:19 -0600

anonymous user


Can pipelines be made dynamic ie. if we want to run different pipelines after reading data from DB and depending on those values run different pipelines ?

edit retag flag offensive reopen merge delete

Closed for the following reason the question is answered, right answer was accepted by KeerthiS
close date 2019-05-31 09:08:04.901252

1 Answer

Sort by ยป oldest newest most voted

answered 2019-05-22 23:37:17 -0600

Maithri gravatar image

updated 2019-05-22 23:38:26 -0600

To run pipelines after reading from db,you can create pipelines with database supported origin and then use an expression evaluator or stream selector according to your need and at the end place a shell executor where you can write pipeline start, stop commands(CLI commands )

An example pipeline structure:

image description

edit flag offensive delete link more


so can we have multiple instances of pipelines with different configurations read from DB. say we run the pipeline with a set of parameters( and one instance of the pipeline is running). can we start the pipeline with different set of parameters parallely ?

KeerthiS gravatar imageKeerthiS ( 2019-05-29 14:17:17 -0600 )edit

The only way that's possible in open source SDC is if you duplicate the pipeline. The other option is to use StreamSets Control Hub -- where you can create jobs which can run multiple instances of the same pipeline with different pipeline parameter values passed in via jobs configuration.

iamontheinet gravatar imageiamontheinet ( 2019-05-29 14:26:28 -0600 )edit

Thank you. That helps.

KeerthiS gravatar imageKeerthiS ( 2019-05-31 09:06:57 -0600 )edit

You're welcome!

iamontheinet gravatar imageiamontheinet ( 2019-05-31 11:49:33 -0600 )edit

Question Tools

1 follower


Asked: 2019-05-22 15:51:19 -0600

Seen: 204 times

Last updated: May 22 '19