Ask Your Question

Setting Environmental Variables in StreamSets...

asked 2020-07-13 13:20:44 -0500

anonymous user


I have multiple servers running multiple pipelines. Everytime I move a pipeline from one server to another, I have to go and manually change each IP number in each of those Stage Configurations.

Can I write in a .bashrc or other startup configuration file for Linux where the StreamSets server is being run on a very basic alias such as kudu001= and then invoke it in the Kudu Stage Configuration in the Kudu Masters Field by writing $Kudu001:7051? When the pipeline is migrated to another server it will have a different alias ip number in it's .bashrc file e.g kudu001=

Is this allowed?
Is there a better way to do this?

Is there a way to do this in StreamSets without having to code something in the shell ala an external pipe to a shell inside of StreamSets application: kudu001= hostname -l ?

Does a call to invoke something in the Linux operating system incur increased performance penalties versus something running natively within the JVM that the StreamSets pipeline is being executed or would it be actually more efficient to do so?

Thank you kindly for your assistance!

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-07-17 17:30:19 -0500

zac gravatar image

I'm using runtime properties. They are read from a file and you reference them in a pipeline like:


edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-07-13 13:20:44 -0500

Seen: 176 times

Last updated: Jul 17