sftp/ftp processing only one file

asked 2018-07-15 22:06:03 -0600

anonymous user

Anonymous

updated 2018-07-18 20:23:44 -0600

metadaddy gravatar image

Hi All,

I have around 7 csv files in a directory in an ftp location. i am using ftp/sftp origin and writing it to hdfs as whole file. But the pipeline is stopping after one of the files is processed.

Configuration options:

sftp/ftp origin - wholefile.
expression - roll header attribute - ${str:endsWith(record:id(), '::0')}
hdfs - filetype - whole file
          data format - whole file
          file expression - ${record:value('/fileInfo/filename')}

Please help me with this

edit retag flag offensive close merge delete

Comments

The FTP origin orders files by timestamps. Can you do an ls -l on the directory? If they're all the same it might cause this sort of problem.

metadaddy gravatar imagemetadaddy ( 2018-07-18 20:42:52 -0600 )edit

yes, they all have the same timestamp. is there a work around for this?

Robot gravatar imageRobot ( 2018-07-30 22:03:12 -0600 )edit