Ask Your Question

Does the JDBC Multitable consumer support writing to SDC RPC destination ?

asked 2017-10-05 19:41:34 -0600

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I need to incrementally load data from around 7 Amazon Redshift database tables and write it to Hive/HDFS.

I can connect to Redshift from only one server, which does not have access to Hive/HDFS; its basically a DMZ layer. My plan is run a pipeline on this server, reading from the JDBC Multitable Consumer and writing to an SDC RPC destination.

From my local/Hadoop cluster I want to run a pipeline on a second SDC instance, receiving the data through the SDC origin and writing it to Hive/HDFS.

Is this possible?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2017-10-09 17:03:32 -0600

metadaddy gravatar image

Yes - this should be perfectly possible - you'll just need to ensure that your first SDC instance, in the 'DMZ', has access to the correct port on the second SDC instance.

See the documentation on SDC RPC Pipelines for more detail.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-10-05 19:41:34 -0600

Seen: 31 times

Last updated: Oct 09 '17