The maximum batch size set in $SDC_CONF/sdc.properties overrides the value in the origin. Increase the production.maxBatchSize property there and you will be able to configure bigger batch sizes.
Note - there are trade-offs in configuring large batch sizes. For example, a batch of records must be able to fit into memory.
You'll want to keep an eye on your JVM heap size if you increase the batch size significantly. I bumped it up to 10,000 records at one point and found that StreamSets would occasionally crash on the machine and I would have to hard restart it. You might also need to increase the JVM heap size itself.
A bit risky to increase this a lot of you have other important pipelines on the same box because they all share the JVM heap.
Asked: 2017-03-22 23:02:58 -0500
Seen: 4,074 times
Last updated: May 17 '17
