How to increase reader limit for http server with JSON data?

asked 2019-12-12 07:48:38 -0600

I'm running SDC 3.2.0.0 and I'm using HTTP Server Origin with data format JSON. When I'm trying to sent to the server an array of 200K objects, I'm getting an error: CONTAINER_0001 - com.streamsets.pipeline.api.ext.io.OverrunException: Reader exceeded the read limit '100000000'.

I read in the forum I should set parser.limit in sdc.properties. I put the value parser.limit=2147483647, however it didn't help. Even if the correct value is visible on host:/collect/configuration. Then I also set Max Object Length (chars) to 2147483647. Am I right that these two properties (parser.limit and Max Object Length) are related and limited by Integer maximal value?

I'm also wandering if OverrunException may be connected with HTTP property: Max Request Size(MB)? I tried to increase this value, but I've got the message: HTTP_SERVER_PUSH_00 - Max request size '1000MB' exceeds limit '100MB'. Is it possible to change it in sdc.properties file?

Data Format:

Compression Format None,

JSON Content JSON array of objects,

Max Object Length(chars) 2147483647,

Charset UTF-8,

additional information:

sent Content-Length: 184200001

file size: 196 600 003B

I've also tried with the SDC 3.11.0 with the same result.

edit retag flag offensive close merge delete