Ask Your Question

How to read latest file (always) automatically in sftp/ftp?

asked 2018-12-12 02:47:18 -0500

Tutgirl gravatar image

updated 2018-12-14 00:19:57 -0500

I have ftp where I want to take the file that is last dropped. I tried First File field with file pattern ABC*.csv.

But I got below exception: org.apache.commons.vfs2.FileSystemException: Could not determine the last modified timestamp of "ftp://*////*/ABC.csv" because it does not exist.

Thanks Rajashree

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-12-12 09:44:25 -0500

iamontheinet gravatar image


It appears that the Resource URL might not be evaluating correctly. For reference, see below:

image description

In the above config, all files relative to $USER_HOME/datafiles folder will be processed, including new files (with later timestamp) that are uploaded to that folder after pipeline has started.

Hope this helps.

Cheers, Dash

edit flag offensive delete link more


I have tried this, its not giving any error but it is picking the earliest file not the latest.

Tutgirl gravatar imageTutgirl ( 2018-12-13 04:34:30 -0500 )edit

It will process files in earliest to latest timestamp order. Can you update your original question and add screenshot of your config like I did in my answer? Also, are you starting and stopping the pipeline before uploading new files or are you uploading new files while the pipeline is running?

iamontheinet gravatar imageiamontheinet ( 2018-12-13 11:36:04 -0500 )edit

Hi I am not able to upload the screenshot image. The files are already uploaded and then I'm starting the pipeline. But when it go to production the files will be dropped when the pipeline is running, so I would like to know both the scenarios. Thanks in advance

Tutgirl gravatar imageTutgirl ( 2018-12-14 00:22:43 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-12-12 02:47:18 -0500

Seen: 560 times

Last updated: Dec 14 '18