I am unable to dump JSON data from HTTP Client to a FTP server in CSV format.

asked 2019-12-26 05:45:43 -0600

arjun gravatar image

I created a pipeline with HTTP client > Field Pivoter > Field Flattenner > SFTP/FTP/FTPS Client

I am simply trying to fetch data from a HTTP API and dump its response to a FTP server.

When I am trying to preview it "Write to Destination and Executors" I am getting this error

com.streamsets.pipeline.lib.generator.DataGeneratorException: WHOLE_FILE_GENERATOR_ERROR_0 - Whole File Format Error. Reason : java.lang.IllegalArgumentException: Record does not contain the mandatory fields /fileRef,/fileInfo,/fileInfo/size for Whole File Format.

I checked the Docs, there isn't much on what it means and how to resolve this.

In the FTP client destination, which is the final block, I have specified the File Name Expressions as ${record:value('/fileInfo/filename')}.csv

If I dont check the "Write to destination and executor" checkbox while previewing, I can see all the data and its transformation. But when I am trying to dump, it shows that error.

How can I resolve this?

edit retag flag offensive close merge delete