Ask Your Question

Pipeline migration from development to production

asked 2017-11-24 06:58:55 -0500

Vivian Y gravatar image

Use case : I developing a set of pipeline in a development environment. Any suggestion that how i can move the pipeline by having the minimal changes.

For example , Pipeline A are reading from Kafka, and consume from a kafka Topic which is "topic-dev". However, when i wanna deploy the same pipeline to production, currently i would need to change kafka topic from "topic-dev" to "topic-prod".

Is there any ways that i can change the kafka-topic value without changing the pipeline? Assume this is one of the use case, there are a lot more changes to make such as api call, folder directory, etc.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2017-11-30 21:11:18 -0500

metadaddy gravatar image

Use Runtime Parameters for these configuration values. You can then specify the parameter values when you start the pipeline.

edit flag offensive delete link more


@metadaddy, in this case, user will still need manually insert those parameters value when start the pipeline. Is there any other option that streamsets can read a config file and assigned the value into parameters?

Vivian Y gravatar imageVivian Y ( 2017-12-19 05:33:45 -0500 )edit

@metadaddy, i read through the documentation and found out that i may use Will try this out.

Vivian Y gravatar imageVivian Y ( 2017-12-19 06:03:51 -0500 )edit
metadaddy gravatar imagemetadaddy ( 2017-12-19 11:46:01 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-11-24 06:58:55 -0500

Seen: 287 times

Last updated: Nov 30 '17