Ask Your Question

Can't separately write the files in HadoopFS

asked 2017-12-07 02:08:53 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I can read lots of files at a time from SFTP also I could write it to the HDFS but all the files are merged into single file in HDFS. So please help me for how to write it separately in HDFS.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2017-12-08 11:15:11 -0500

metadaddy gravatar image

If you don't need to process the file content, you can use Whole File data format to copy the files from SFTP to HDFS.

If you do need to process the records in each file, it's a little more complicated. You would need to use an Expression Evaluator to set the roll header attribute when record:id() ends in ::0, then configure the HDFS destination to use the roll attribute.

edit flag offensive delete link more


Hi metadady, Thanks for an update. Now I can move the files from SFTP to HDFS. Actually the files are compressed. So please tell me how can I uncompress the Whole File when it is residing on before in the HDFS?

Sam Sambath gravatar imageSam Sambath ( 2017-12-11 04:18:43 -0500 )edit

Hi metadaddy, I tried to copy the files from SFTP to HDFS by using the Whole File data format.The files are moved to hdfs but the moved files do not containing all the contents some contents are missing in hdfs file.Please hell me to configure.

Sam Sambath gravatar imageSam Sambath ( 2017-12-11 07:21:19 -0500 )edit

@Sam Sambath: Can you please share the configuration screenshots how you configure for sftp to hdfs. Please help me i try to configure from past 3 days but i'm not able to pull data from sftp. Connections and configurations all are correct.

praneeth gravatar imagepraneeth ( 2017-12-11 11:09:37 -0500 )edit

hai praneeth , You can configure by using the document which is present in the following link.

Sam Sambath gravatar imageSam Sambath ( 2017-12-11 22:38:10 -0500 )edit
Login/Signup to Answer

Question Tools



Asked: 2017-12-07 02:08:53 -0500

Seen: 190 times

Last updated: Dec 08 '17