Ask Your Question

how to move processed files from source location(file drop server) to archive folder using the SFTP/FTP origin in streamsets

asked 2019-07-23 00:29:28 -0500

dharma gravatar image

My requirement is to move files from input folder to archive folder, once the file got processed. we are using file drop server(not local) as source location.By Directory we have option(Post process) But by SFTP origin, how we can do it.

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted

answered 2019-07-23 10:46:20 -0500

iamontheinet gravatar image

updated 2019-07-24 11:52:12 -0500


In the current stable release of SDC (v 3.9.1), SFTP/FTP origin has post processing option available as well. For details, refer to the documentation.

For older versions of SDC, look into generating events by enabling Produce Events on SFTP/FTP Origin. Then use Stream Selector with its Preconditions set to ${record:eventType() == 'finished-file'}and provide your script in Shell Executor to archive the processed file. (See attached screenshots.)

image description

image description

image description

Hope this helps.

Cheers, Dash

edit flag offensive delete link more


Thanks to provide solution in sort time. But Right now we are using SDC (v 3.5), so is any other way to do this with SDC (v 3.5) .

dharma gravatar imagedharma ( 2019-07-24 05:18:42 -0500 )edit

I've updated my answer with alternative solution.

iamontheinet gravatar imageiamontheinet ( 2019-07-24 11:49:55 -0500 )edit

Thanks alot "iamontheinet", i tried with this, but file didn't moved & no any's seams nothing happen. is this solution is working only for localhost or for different server also? For us file location is on different server,so may be need to provide credential .

dharma gravatar imagedharma ( 2019-07-25 01:13:03 -0500 )edit

Yes, you will need credentials, permissions, etc. to move files.

iamontheinet gravatar imageiamontheinet ( 2019-07-25 10:10:21 -0500 )edit

how/where do we provide credentials for the Shell Script so that it runs with same credentials we used in the SFTP/FTP origin ?

Carol gravatar imageCarol ( 2020-03-24 15:01:23 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-07-23 00:29:28 -0500

Seen: 1,198 times

Last updated: Jul 24 '19