Ask Your Question
1

Pipeline migration from development to production

asked 2017-11-24 06:58:55 -0600

Vivian Yang gravatar image

Use case : I developing a set of pipeline in a development environment. Any suggestion that how i can move the pipeline by having the minimal changes.

For example , Pipeline A are reading from Kafka, and consume from a kafka Topic which is "topic-dev". However, when i wanna deploy the same pipeline to production, currently i would need to change kafka topic from "topic-dev" to "topic-prod".

Is there any ways that i can change the kafka-topic value without changing the pipeline? Assume this is one of the use case, there are a lot more changes to make such as api call, folder directory, etc.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2017-11-30 21:11:18 -0600

metadaddy gravatar image

Use Runtime Parameters for these configuration values. You can then specify the parameter values when you start the pipeline.

edit flag offensive delete link more

Comments

@metadaddy, in this case, user will still need manually insert those parameters value when start the pipeline. Is there any other option that streamsets can read a config file and assigned the value into parameters?

Vivian Yang gravatar imageVivian Yang ( 2017-12-19 05:33:45 -0600 )edit
1

@metadaddy, i read through the documentation and found out that i may use runtime.properties. Will try this out.

Vivian Yang gravatar imageVivian Yang ( 2017-12-19 06:03:51 -0600 )edit
metadaddy gravatar imagemetadaddy ( 2017-12-19 11:46:01 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-11-24 06:58:55 -0600

Seen: 56 times

Last updated: Nov 30 '17