Ask Your Question
0

How to process ftp files and move it to archive folder after processing?

asked 2018-12-16 09:38:22 -0500

Tutgirl gravatar image

I am using sftp/ftp to read the file, please suggest me how to move the file to archive after successful processing and to error folder if any error occurs?

edit retag flag offensive close merge delete

2 Answers

Sort by » oldest newest most voted
0

answered 2019-08-29 13:05:07 -0500

iamontheinet gravatar image

Actually, if you get the latest release (3.10.1), SFTP/FTP origin has built-in way of post processing non-whole file data format files -- https://streamsets.com/documentation/...

edit flag offensive delete link more
1

answered 2018-12-17 14:43:16 -0500

iamontheinet gravatar image

updated 2018-12-17 14:45:30 -0500

Hi!

Here are the steps you can follow:

Step 1: Enable SFTP/FTP origin to produce events by checking Produce Events checkbox.

image description

Step 2: Add Stream Selector processor to handle finished-file events using condition expression ${record:eventType() == "finished-file"}

image description

Step 3: Add Shell executor and on Environment tab set the current file name and path in environment variable, for example, file_to_archive using expression ${record:value('/filepath')}.

image description

Step 4: On Script tab add shell script to archive the file. For example:

#!/bin/bash
FILE_TO_ARCHIVE=$file_to_archive
ARCHIVE_DIR="/path/to/archive/folder"
mv "$FILE_TO_ARCHIVE" $ARCHIVE_DIR
exit 0

Note that $file_to_archive in the script matches variable name set in Step 3.

image description

Hope this helps!

Cheers, Dash

edit flag offensive delete link more

Comments

1

Env variables: sourcefile: ${record:value('/filepath')}, archivefile: ${str:replaceAll(record:value('/filepath'),"Latest","Archive")} My script: #!/bin/bash HOST='' USER='' PASSWORD='*' File_To_Archive=$sourcefile Arc=$archivefile ftp -inv $HOST <<eof user="" $user="" password="" rename="" $file_to_archive="" $arc<="" p="">

Tutgirl gravatar imageTutgirl ( 2019-01-03 07:02:35 -0500 )edit

But my script from Streamsets is not working, whereas if i execute the same commands from command prompt, it is working. Below are the streamset logs: stdout: Connected to **.*** (**.***) stdout: 220 Microsoft FTP Service stdout: Remote system type is Windows_NT. stdout: ?Invalid command stdout: 530

Tutgirl gravatar imageTutgirl ( 2019-01-03 07:05:36 -0500 )edit

Is it appropriate to write ftp commands in shell executor for the Windows File Server?

Tutgirl gravatar imageTutgirl ( 2019-01-03 07:07:06 -0500 )edit

I am working in a development cloud environment, trying to remove the file from SFTP. As could result obvious I am getting the message stderr: mv: failed to access ‘/efs/home/b553851/streamset/destination’: Permission denied, because it is in the sftp and I cannot send the password to authenticate.

bcd2682 gravatar imagebcd2682 ( 2019-08-29 16:05:00 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-12-16 09:38:22 -0500

Seen: 1,442 times

Last updated: Dec 17 '18