Ask Your Question
0

How to process ftp files and move it to archive folder after processing?

asked 2018-12-16 09:38:22 -0500

Tutgirl gravatar image

I am using sftp/ftp to read the file, please suggest me how to move the file to archive after successful processing and to error folder if any error occurs?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2018-12-17 14:43:16 -0500

iamontheinet gravatar image

updated 2018-12-17 14:45:30 -0500

Hi!

Here are the steps you can follow:

Step 1: Enable SFTP/FTP origin to produce events by checking Produce Events checkbox.

image description

Step 2: Add Stream Selector processor to handle finished-file events using condition expression ${record:eventType() == "finished-file"}

image description

Step 3: Add Shell executor and on Environment tab set the current file name and path in environment variable, for example, file_to_archive using expression ${record:value('/filepath')}.

image description

Step 4: On Script tab add shell script to archive the file. For example:

#!/bin/bash
FILE_TO_ARCHIVE=$file_to_archive
ARCHIVE_DIR="/path/to/archive/folder"
mv "$FILE_TO_ARCHIVE" $ARCHIVE_DIR
exit 0

Note that $file_to_archive in the script matches variable name set in Step 3.

image description

Hope this helps!

Cheers, Dash

edit flag offensive delete link more

Comments

Env variables: sourcefile: ${record:value('/filepath')}, archivefile: ${str:replaceAll(record:value('/filepath'),"Latest","Archive")} My script: #!/bin/bash HOST='' USER='' PASSWORD='*' File_To_Archive=$sourcefile Arc=$archivefile ftp -inv $HOST <<eof user="" $user="" password="" rename="" $file_to_archive="" $arc<="" p="">

Tutgirl gravatar imageTutgirl ( 2019-01-03 07:02:35 -0500 )edit

But my script from Streamsets is not working, whereas if i execute the same commands from command prompt, it is working. Below are the streamset logs: stdout: Connected to **.*** (**.***) stdout: 220 Microsoft FTP Service stdout: Remote system type is Windows_NT. stdout: ?Invalid command stdout: 530

Tutgirl gravatar imageTutgirl ( 2019-01-03 07:05:36 -0500 )edit

Is it appropriate to write ftp commands in shell executor for the Windows File Server?

Tutgirl gravatar imageTutgirl ( 2019-01-03 07:07:06 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-12-16 09:38:22 -0500

Seen: 84 times

Last updated: Dec 17 '18