Ask Your Question

Shell executor not executing AWS cli commands as expected

asked 2018-09-28 05:28:45 -0500

eohehirA gravatar image


I am using a Shell executor to preform some tasks on AWS S3 via AWS cli.

e.g. I want to delete some files from S3.

I define S3Bucket, S3CommonPrefix and FileType Environment variables based on the data in the pipeline and attempt to execute the following from a Shell Executor:

echo "aws s3 rm s3://$S3Bucket/$S3CommonPrefix/ --recursive --exclude '*' --include 'MyFile_$FileType-*.avro'" > /tmp/awscmd.txt
aws s3 rm s3://$S3Bucket/$S3CommonPrefix/ --recursive --exclude '*' --include 'MyFile_$FileType-*.avro'
exit 0

This does not preform the S3 delete for me.

If I use the text of the command written to /tmp/awscmd.txt from a command prompt outside of Streamsets then the files are deleted, so my AWS cli syntax is correct and the relevant environment variables are being populated correctly.

After some playing around I worked out that the following will syntax work:

echo "aws s3 rm s3://$S3Bucket/$S3CommonPrefix/ --recursive --exclude '*' --include 'MyFile_$FileType-*.avro'" | sh
exit 0

Can anyone advise what is going on here ? Is it something to do with the single quotes ?



edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-09-28 09:36:04 -0500

Tim gravatar image

updated 2018-09-28 10:05:29 -0500

Try capturing the stderr and stdout from the command and see if you are missing output. e.g.

aws s3 rm s3://$S3Bucket/$S3CommonPrefix/ --recursive --exclude '' --include 'MyFile_$FileType-.avro' 2>&1 > /tmp/awscmd.txt

I think more than likely you are executing the code in two different environments. Either you are running SDC under one user login and testing the command under another. Or there are environment variables missing from the StreamSets shell that are being picked up from your test login. For example, ~/.bash_profile is read when you log in but may not be read when a shell is executed from an application.

Best of luck,


edit flag offensive delete link more


You are spot on Tim. Thanks. I was testing using my own login but Streamsets runs under a different user (sdc) meaning different environment settings. In this case the sdc user had different proxy settings so could not access aws.

eohehirA gravatar imageeohehirA ( 2018-10-02 11:05:20 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-09-28 05:28:45 -0500

Seen: 616 times

Last updated: Sep 28 '18