Ask Your Question
1

`com.streamsets.pipeline.stage.origin.s3.BadSpoolObjectException: com.streamsets.pipeline.api.ext.io.OverrunException: Reader exceeded the read limit '1048576'` when parsing a json file

asked 2017-06-13 08:52:39 -0500

bob gravatar image

Hello, we have been getting com.streamsets.pipeline.stage.origin.s3.BadSpoolObjectException: com.streamsets.pipeline.api.ext.io.OverrunException: Reader exceeded the read limit '1048576' on a json file for a while now and cannot bypass this limitation. Any workarounds ?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2017-06-13 08:58:22 -0500

bob gravatar image

updated 2017-06-19 12:47:47 -0500

LC gravatar image

There are 2 ways to get around this problem:

  • in version 2.6 and before - add the following to your ${SDC_JVM_OPTS}:

    -DDataFactoryBuilder.OverRunLimit=10485760 -Doverrun.reader.read.limit=10485760
    

    these parameters set the buffer size to 10MiB - adjust as appropriate. also, set the Max Object Length in the Data Format tab.

  • in our next version after 2.6 (so in 2.6.X or 2.7), you'll be able to fix this by setting a new property - parser.limit in the sdc.properties file eg: parser.limit=10485760

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-06-13 08:52:39 -0500

Seen: 1,078 times

Last updated: Jun 19 '17