Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

sftp/ftp processing only one file

Hi All,

I have around 7 csv files in a directory in an ftp location. i am using ftp/sftp origin and writing it to hdfs as whole file. But the pipeline is stopping after one of the files is processed.

Configuration options:

sftp/ftp - wholefile. expression - roll header attribute - ${str:endsWith(record:id(), '::0')} hdfs - filetype - whole file data format - whole file file expression - ${record:value('/fileInfo/filename')}

Please help me with this

sftp/ftp processing only one file

Hi All,

I have around 7 csv files in a directory in an ftp location. i am using ftp/sftp origin and writing it to hdfs as whole file. But the pipeline is stopping after one of the files is processed.

Configuration options:

sftp/ftp origin - wholefile. expression - roll header attribute - ${str:endsWith(record:id(), '::0')} hdfs - filetype - whole file data format - whole file file expression - ${record:value('/fileInfo/filename')}

Please help me with this

sftp/ftp processing only one file

Hi All,

I have around 7 csv files in a directory in an ftp location. i am using ftp/sftp origin and writing it to hdfs as whole file. But the pipeline is stopping after one of the files is processed.

Configuration options:

sftp/ftp origin - wholefile.
expression - roll header attribute - ${str:endsWith(record:id(), '::0')}
hdfs - filetype - whole file
          data format - whole file
          file expression - ${record:value('/fileInfo/filename')}

${record:value('/fileInfo/filename')}

Please help me with this