Ask Your Question

Wholefile movement does not move all files matching given glob

asked 2018-03-09 20:18:20 -0500

lampshadesdrifter gravatar image

updated 2018-03-12 17:23:15 -0500

I am trying to move two zip files from a local directory to a maprfs directory. Both are of the form, the glob used to match the files is *.zip, there is no preprocessing between the origin and destinations (just direct links), and there are no problems accessing maprfs inn the destination.

Yet, when I run the pipeline preview I only see a single record for one of the zip files being transferred, there is not even a new-file Event indicating that the 2nd file ever start being transferred at all (happens when running the pipeline "for real" as well, it just transfers the 1st one and then runs forever never getting to the next). Removing the file that is successfully getting transferred each time, then allows the other file to be transferred (making it appear as if the pipeline is only capable of transferring a single file before ending).

This does not seem to correct behavior (comparing with what is shown here: Does anyone know what could be going on here? Thanks.

The two files in question can be downloaded here (, under Under "Data Files", 201X BRFSS Data (ASCII)[ZIP – 44 MB]).

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted

answered 2018-03-12 19:48:22 -0500

lampshadesdrifter gravatar image

After trying some things and examining the logs, seemed to show that files were being ignored because they had older or equal timestamps than other files in the origin location (pipeline using read_order=last_modified_timestamp). Running "touch *" in the origin directory and rerunning the pipeline fixed the problem. I think this was due to not understanding that standalone pipelines can remain persistently "on" and scanning for new data (since have been only using cluster-batch mode until now, which don't act that way).

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-03-09 20:18:20 -0500

Seen: 5 times

Last updated: Mar 12