# Shell executor not executing AWS cli commands as expected

Hi,

I am using a Shell executor to preform some tasks on AWS S3 via AWS cli.

e.g. I want to delete some files from S3.

I define S3Bucket, S3CommonPrefix and FileType Environment variables based on the data in the pipeline and attempt to execute the following from a Shell Executor:

echo "aws s3 rm s3://$S3Bucket/$S3CommonPrefix/ --recursive --exclude '*' --include 'MyFile_$FileType-*.avro'" > /tmp/awscmd.txt aws s3 rm s3://$S3Bucket/$S3CommonPrefix/ --recursive --exclude '*' --include 'MyFile_$FileType-*.avro'
exit 0


This does not preform the S3 delete for me.

If I use the text of the command written to /tmp/awscmd.txt from a command prompt outside of Streamsets then the files are deleted, so my AWS cli syntax is correct and the relevant environment variables are being populated correctly.

After some playing around I worked out that the following will syntax work:

echo "aws s3 rm s3://$S3Bucket/$S3CommonPrefix/ --recursive --exclude '*' --include 'MyFile_$FileType-*.avro'" | sh exit 0  Can anyone advise what is going on here ? Is it something to do with the single quotes ? Thanks, Eoin. edit retag close merge delete ## 1 Answer Sort by » oldest newest most voted Try capturing the stderr and stdout from the command and see if you are missing output. e.g. aws s3 rm s3://$S3Bucket/$S3CommonPrefix/ --recursive --exclude '' --include 'MyFile_$FileType-.avro' 2>&1 > /tmp/awscmd.txt

I think more than likely you are executing the code in two different environments. Either you are running SDC under one user login and testing the command under another. Or there are environment variables missing from the StreamSets shell that are being picked up from your test login. For example, ~/.bash_profile is read when you log in but may not be read when a shell is executed from an application.

Best of luck,

Tim

more