Streamset not picking up files as soon as they are available on SFTP server

asked 2020-01-13 14:55:38 -0500

cvjere gravatar image

I have configured pipeline to get the data from SFTP server . I am finding that the files are not picked up by Streamsets as soon as the they are available available on SFTP, though the pipeline is running . For example We do have file for Jan 12 2020 arrived on sftp on 12th Jan 2020 on SFTP but the latest data picked up by streamsets is of 10th Jan 2020.

I have 16 pipelines running on one server getting data from SFTP server but from different directories

edit retag flag offensive close merge delete