From xml file to whole file format

asked 2019-05-08 02:53:48 -0500

set92 gravatar image

I'm trying to call a SOAP API, get the reply and save it, I'm having problems with the size of the xml file, since is too big, I tried to increase Max Record Length to the maximum, because the answer comes in one line, but I keep having some problems like sometimes it don't save the file.

So I try doing simple pipelines to check the fails or learn to use streamsets.

In the first is only directory to local fs, a simple xml of 10 lines to whole file in another folder. My idea with this is to see if I can save the xml files as whole file and that way I don't have to specify how many characters I want to save from the API request, since I want all the answer and I don't know why I have to specify it when I save it as XML.

Directory 1

  • Files
    • Files directory: /tmp
    • File Name pattern: note.xml
  • Data
    • Data format: XML

Local FS 1

  • Data Format

    • Data Format: Whole File
    • FIle name expression: ${record:value('/fileInfo/filename')}.xml
    • Permissions expression: 0777

    • File Name pattern: note.xml

  • Data
    • Data format: XML

In the second experiment I tried to call the API to get the reply and I tried to save it as text and as whole file, but when I click in preview it get stuck and then it stops the preview, and when I click in running, it finished in 3 seconds saying pipeline finished.

So, I don't know if this is possible or where to find some tutorial with this problem because it seems that sometimes saves the files but not always, in the first experiment when I see the preview I can see that the directory is able to read the file but in the output of Local FS it don't shows anything, and if I go to check the folder there aren't any folder or file.

edit retag flag offensive close merge delete