Ask Your Question

Change S3 prefix based on filename

asked 2018-09-13 08:24:25 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.


I am retrieving a file from sftp with a date in the filename (e.g.

I want to place this file in S3 with a common prefix based on the date in the filename.

e.g. will be stored in S3 at s3://MyS3Bucket/year=2018/month=09/day=13/

I tried to create the "year=2018/month=09/day=13" string from the sftp client Filename header attribute using an Expression Evaluator, placing the result in a /prefix header attribute. I then put the value "{record:attribute(\prefix)}" in the Common Prefix field in the S3 destination. This results in the file being copied to the following in S3


i.e. the expression is not evaluated but taken as a literal string

Is it possible to dynamically set the Common Prefix at runtime ?



edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-09-13 14:19:16 -0500

metadaddy gravatar image

Common Prefix is a literal string, but Partition Prefix is evaluated. If you set Partition Prefix to ${record:attribute(\prefix)} it should work.

edit flag offensive delete link more


That did the trick. Thanks.

eohehirA gravatar imageeohehirA ( 2018-09-14 03:19:40 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-09-13 08:24:25 -0500

Seen: 123 times

Last updated: Sep 13 '18