Ask Your Question
1

Change S3 prefix based on filename

asked 2018-09-13 08:24:25 -0600

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

Hi,

I am retrieving a file from sftp with a date in the filename (e.g. myfile_2018_09_13.zip).

I want to place this file in S3 with a common prefix based on the date in the filename.

e.g. myfile_2018_09_13.zip will be stored in S3 at s3://MyS3Bucket/year=2018/month=09/day=13/myfile.zip

I tried to create the "year=2018/month=09/day=13" string from the sftp client Filename header attribute using an Expression Evaluator, placing the result in a /prefix header attribute. I then put the value "{record:attribute(\prefix)}" in the Common Prefix field in the S3 destination. This results in the file being copied to the following in S3

s3://MyS3Bucket/{record:attribute(\prefix)}/myfile.zip

i.e. the expression is not evaluated but taken as a literal string

Is it possible to dynamically set the Common Prefix at runtime ?

Thanks,

Eoin.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2018-09-13 14:19:16 -0600

metadaddy gravatar image

Common Prefix is a literal string, but Partition Prefix is evaluated. If you set Partition Prefix to ${record:attribute(\prefix)} it should work.

edit flag offensive delete link more

Comments

That did the trick. Thanks.

eohehirA gravatar imageeohehirA ( 2018-09-14 03:19:40 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-09-13 08:24:25 -0600

Seen: 22 times

Last updated: Sep 13