Ask Your Question
0

Custom filename for aws s3

asked 2018-12-10 17:51:34 -0600

sandeeplv gravatar image

MY origin is kafka topic from there i get json data which i want to move to aws s3 with the filename same as one of the field in my json body.

The suffix or prefix field do not take expressions. Also the common prefix takes only literal.Could some one suggest any possible options.

Any help appreciated.

Thanks Sandeep

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-12-13 03:52:34 -0600

Alexander gravatar image

You can write the file first to your directory (pipeline 1), from there you can connect a file source to S3 and select the option "whole file" (pipeline 2). In this case the filename does not change and also no timestamp is added in the filename.

edit flag offensive delete link more

Comments

I tried doing the way you mentioned but in the second pipeline[local FS to s3] i am getting the below exception when data flows com.streamsets.pipeline.api.base.OnRecordErrorException: S3_52 - Cannot Write Record : java.lang.IllegalArgumentException: Record does not contain the mandatory fields /fil

sandeeplv gravatar imagesandeeplv ( 2018-12-13 10:29:01 -0600 )edit

com.streamsets.pipeline.api.base.OnRecordErrorException: S3_52 - Cannot Write Record : java.lang.IllegalArgumentException: Record does not contain the mandatory fields /fileRef,/fileInfo,/fileInfo/size for Whole File Format. at com.streamsets.pipeline.stage.destination.s3.WholeFileHelper.handle(Who

sandeeplv gravatar imagesandeeplv ( 2018-12-13 10:29:47 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-12-10 17:51:34 -0600

Seen: 28 times

Last updated: Dec 13 '18