Ask Your Question
1

Does StreamSets DC support user defined global parameters on instance level?

asked 2017-11-01 21:58:04 -0500

casel.chen gravatar image

As far as I know StreamSets DC support user defined pipeline level parameters, but how can I define those parameters used cross pipelines? Saying I need setup multiple data pipelines to sync data from multiple MySQL tables to Hive/HBase, each data pipeline handle one table data sync, there are some common parameters I want to share across those pipelines. Is there any other ways to handle better on above issue? I thought of use JDBC MultipleTableQuery origin, connect with StreamSelector, finally each StreamSelector connect with Hive/HBase destination, but there are so many tables to fork.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2020-04-22 22:29:50 -0500

J3 gravatar image

updated 2020-04-22 23:57:14 -0500

iamontheinet gravatar image

Hi Casel,

I use ${runtime:loadResource(<TEXT FILE NAME>, true)} function to store pipeline parameters that I use across multiple pipelines.

https://streamsets.com/documentation/...

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-11-01 21:58:04 -0500

Seen: 232 times

Last updated: Apr 22