Ask Your Question

Does StreamSets DC support user defined global parameters on instance level?

asked 2017-11-01 21:58:04 -0600

casel.chen gravatar image

As far as I know StreamSets DC support user defined pipeline level parameters, but how can I define those parameters used cross pipelines? Saying I need setup multiple data pipelines to sync data from multiple MySQL tables to Hive/HBase, each data pipeline handle one table data sync, there are some common parameters I want to share across those pipelines. Is there any other ways to handle better on above issue? I thought of use JDBC MultipleTableQuery origin, connect with StreamSelector, finally each StreamSelector connect with Hive/HBase destination, but there are so many tables to fork.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-04-22 22:29:50 -0600

J3 gravatar image

updated 2020-04-22 23:57:14 -0600

iamontheinet gravatar image

Hi Casel,

I use ${runtime:loadResource(<TEXT FILE NAME>, true)} function to store pipeline parameters that I use across multiple pipelines.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-11-01 21:58:04 -0600

Seen: 292 times

Last updated: Apr 22 '20