Ask Your Question
0

S3 Destination WARN "No content length specified..."

asked 2019-04-05 09:34:52 -0600

daveh gravatar image

I have a pipeline that splits a record into three different types of S3 destinations, one being an AVRO conversion. I am receiving a "WARN" in the logs as follows:

No content length specified for stream data. Stream contents will be buffered in memory and could result in out of memory errors. The logs also say its for the AmazonS3Client.

I cannot find any reference to this for the S3 Destination in the Streamsets documentation or elsewhere on line. I do not see anywhere in the S3 Destinaton configuration to set a content length for writing to S3.

Any insight would be greatly appreciated.

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted
0

answered 2019-04-05 11:26:05 -0600

iamontheinet gravatar image

Hi!

The message is coming directly from the AWS client library code. Basically when streaming data to/from S3 they don't send a content-length header along with the payload; since it’s streaming mode. This should not impact your pipeline.

Cheers, Dash

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-04-05 09:34:52 -0600

Seen: 986 times

Last updated: Apr 05