Ask Your Question
0

Large ZIP files from Directory-Orign

asked 2020-06-02 14:23:29 -0500

Peter Delaney gravatar image

I have two pipelines (copy-pipeline, unzip-pipeline). copy-pipeline starts with Directory-Origin and copies the *.zip file it finds and copies them to a Local-Directory-Destination. copy-pipeline does some filtering so it does NOT copy all of the *.zip files it finds in the Origin directory.

unzip-pipeline also starts with Directory-Origin and unzips the *.zip files to another delivery directory using custom code.

our original zip files are large so the copy-pipeline could take 10 seconds to copy to the destination that unzip-pipeline will process.

I often see error in the unzip-pipeline where it contains a _tmp_NAME-OF-ZIP.zip that it is NOT a zip file or it cannot find the file.

Is this _tmp_NAME-OF-ZIP.zip a temporary name that the Local-Directory-Destination uses while copying large files?

If so do you have a recommendation as to how to prevent our second pipeline, unzip-pipeline from reading this temp file until they are ready to be processed?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2020-06-03 21:27:07 -0500

iamontheinet gravatar image

Hi!

Yes, the _tmp files are temporary. What you could do is in the first pipeline, move fully processed files to a different folder as a post processing step. Then have second pipeline read files from that folder. That will ensure that you only process files that are ready to be processed. For details, check out Post Processing tab on Directory Origin.

Hope this helps.

Cheers, Dash

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-06-02 14:23:29 -0500

Seen: 26 times

Last updated: Jun 03