Ask Your Question
1

Hive to Hive - One cluster hive to another cluster hive

asked 2017-10-06 13:25:36 -0600

Roh gravatar image

Cluster 1 - Hive 1 - Table1 Cluster 2 - Hive 2 - Table2 (same schema as table 1)

Can I write the hive tables data from Cluster 1 to the Cluster 2 in stream sets? I have a process to do it in Hadoop which is DistCp to copy the files from one cluster to other, I was wondering if there is any way to replace that with stream sets.

Thanks in advance.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2017-10-09 16:52:16 -0600

metadaddy gravatar image

You could certainly do this at the files level with StreamSets using the Hadoop FS origin and destination. The pipeline would run in Cluster Batch mode, so you would have multiple instances of SDC running on the cluster, and you would get the benefits of being able to apply transformations to the data in flight if necessary.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-10-06 13:25:36 -0600

Seen: 236 times

Last updated: Oct 09 '17