Ask Your Question

Does StreamSets support HDFS to Google Cloud Storage?

asked 2019-05-16 08:36:06 -0600

bala divvela gravatar image

updated 2019-05-16 10:35:22 -0600

metadaddy gravatar image

Is there a way that a pipeline can be built to transfer data from Cloudera HDFS to Google Cloud Storage? Also, does it support encryption and compression during data transit or do we have to do compression and encryption before we transfer the data?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2019-05-16 10:34:26 -0600

metadaddy gravatar image

Yes - you would simply build a pipeline with one of the Hadoop FS origins (cluster or standalone) and the Google Cloud Storage destination.

Data in transit to Google Cloud Storage is encrypted with TLS by default, and you can enable Compress with Gzip in the Google Cloud Storage destination to compress data.

edit flag offensive delete link more


Hi, I tried hadoop fs standalone component. I could see in the data format option: parquet file option is not present. I wanted to read the parquet file

Aakash gravatar imageAakash ( 2020-02-13 06:23:09 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-05-16 08:36:06 -0600

Seen: 218 times

Last updated: May 16 '19