Ask Your Question
1

Does the JDBC Multitable consumer support writing to SDC RPC destination ?

asked 2017-10-05 19:41:34 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I need to incrementally load data from around 7 Amazon Redshift database tables and write it to Hive/HDFS.

I can connect to Redshift from only one server, which does not have access to Hive/HDFS; its basically a DMZ layer. My plan is run a pipeline on this server, reading from the JDBC Multitable Consumer and writing to an SDC RPC destination.

From my local/Hadoop cluster I want to run a pipeline on a second SDC instance, receiving the data through the SDC origin and writing it to Hive/HDFS.

Is this possible?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2017-10-09 17:03:32 -0500

metadaddy gravatar image

Yes - this should be perfectly possible - you'll just need to ensure that your first SDC instance, in the 'DMZ', has access to the correct port on the second SDC instance.

See the documentation on SDC RPC Pipelines for more detail.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-10-05 19:41:34 -0500

Seen: 51 times

Last updated: Oct 09 '17