Ask Your Question
1

how to move processed files from source location(file drop server) to archive folder using the SFTP/FTP origin in streamsets

asked 2019-07-23 00:29:28 -0500

dharma gravatar image

My requirement is to move files from input folder to archive folder, once the file got processed. we are using file drop server(not local) as source location.By Directory we have option(Post process) But by SFTP origin, how we can do it.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-07-23 10:46:20 -0500

iamontheinet gravatar image

updated 2019-07-24 11:52:12 -0500

Hi!

In the current stable release of SDC (v 3.9.1), SFTP/FTP origin has post processing option available as well. For details, refer to the documentation.

For older versions of SDC, look into generating events by enabling Produce Events on SFTP/FTP Origin. Then use Stream Selector with its Preconditions set to ${record:eventType() == 'finished-file'}and provide your script in Shell Executor to archive the processed file. (See attached screenshots.)

image description

image description

image description

Hope this helps.

Cheers, Dash

edit flag offensive delete link more

Comments

Thanks to provide solution in sort time. But Right now we are using SDC (v 3.5), so is any other way to do this with SDC (v 3.5) .

dharma gravatar imagedharma ( 2019-07-24 05:18:42 -0500 )edit

I've updated my answer with alternative solution.

iamontheinet gravatar imageiamontheinet ( 2019-07-24 11:49:55 -0500 )edit

Thanks alot "iamontheinet", i tried with this, but file didn't moved & no any error.it's seams nothing happen. is this solution is working only for localhost or for different server also? For us file location is on different server,so may be need to provide credential .

dharma gravatar imagedharma ( 2019-07-25 01:13:03 -0500 )edit

Yes, you will need credentials, permissions, etc. to move files.

iamontheinet gravatar imageiamontheinet ( 2019-07-25 10:10:21 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-07-23 00:29:28 -0500

Seen: 881 times

Last updated: Jul 24