Ask Your Question

How to define stage and reuse stage templates ?

asked 2019-11-12 08:32:57 -0500

Damien gravatar image


We have been testing StreamSets for several weeks with the aim of migrating our old ETL batch pipelines to streaming, but there is something we have not managed to do yet : reusing our connectors (especially Kafka ones) across pipelines.

There seem to be several related questions in the forum and most of the answers encourage the use of Runtime Parameters, but we don't understand how they help reusing the Kafka connector with its numerous parameters ? As far as we understand, runtime parameters only helps assigning values to parameters.

Is there any way to define a Kafka origin "template" with parameters (whichever the runtime values are) that we can define and reuse from the available stage sidebar or do we have to redefine the parameters for in all our pipelines ? (We are planning to design hundreds of them)

Thanks in advance for any help.


edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2019-11-12 09:08:54 -0500

jeff gravatar image

If you use Control Hub, there is the concept of pipeline fragments, which it sounds like would help in your case.

Within the Data Collector product by itself, different stages with shared runtime parameters is probably the best you can do.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-11-12 08:32:57 -0500

Seen: 43 times

Last updated: Nov 12 '19