Ask Your Question

Revision history [back]

If you don't need to process the file content, you can use Whole File data format to copy the files from SFTP to HDFS.

If you do need to process the records in each file, it's a little more complicated. You would need to use an Expression Evaluator to set the roll header attribute when record:id() ends in ::0, then configure the HDFS destination to use the roll attribute.