Ask Your Question

How to process ftp files and move it to archive folder after processing?

asked 2018-12-16 09:38:22 -0600

Tutgirl gravatar image

I am using sftp/ftp to read the file, please suggest me how to move the file to archive after successful processing and to error folder if any error occurs?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-12-17 14:43:16 -0600

iamontheinet gravatar image

updated 2018-12-17 14:45:30 -0600


Here are the steps you can follow:

Step 1: Enable SFTP/FTP origin to produce events by checking Produce Events checkbox.

image description

Step 2: Add Stream Selector processor to handle finished-file events using condition expression ${record:eventType() == "finished-file"}

image description

Step 3: Add Shell executor and on Environment tab set the current file name and path in environment variable, for example, file_to_archive using expression ${record:value('/filepath')}.

image description

Step 4: On Script tab add shell script to archive the file. For example:

exit 0

Note that $file_to_archive in the script matches variable name set in Step 3.

image description

Hope this helps!

Cheers, Dash

edit flag offensive delete link more


Env variables: sourcefile: ${record:value('/filepath')}, archivefile: ${str:replaceAll(record:value('/filepath'),"Latest","Archive")} My script: #!/bin/bash HOST='' USER='' PASSWORD='*' File_To_Archive=$sourcefile Arc=$archivefile ftp -inv $HOST <<eof user="" $user="" password="" rename="" $file_to_archive="" $arc<="" p="">

Tutgirl gravatar imageTutgirl ( 2019-01-03 07:02:35 -0600 )edit

But my script from Streamsets is not working, whereas if i execute the same commands from command prompt, it is working. Below are the streamset logs: stdout: Connected to **.*** (**.***) stdout: 220 Microsoft FTP Service stdout: Remote system type is Windows_NT. stdout: ?Invalid command stdout: 530

Tutgirl gravatar imageTutgirl ( 2019-01-03 07:05:36 -0600 )edit

Is it appropriate to write ftp commands in shell executor for the Windows File Server?

Tutgirl gravatar imageTutgirl ( 2019-01-03 07:07:06 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-12-16 09:38:22 -0600

Seen: 40 times

Last updated: Dec 17 '18