Ask Your Question
0

Azure IoT Hub origin batch size

asked 2018-12-14 05:55:50 -0500

hapination gravatar image

There doesn't seem to be any batch size setting for the Azure IoT Hub origin. It is always returning me just 10 records in a batch. How can I increase this?

If I try to increment the thread count I start getting errors 'New receiver with higher epoch of '37' is created hence current receiver with epoch '36' is getting disconnected.' which implies that there can only be one thread at a time running. Although the Streamsets documentation says that the origin can use multithreaded processing

edit retag flag offensive close merge delete
add a comment

1 Answer

Sort by ยป oldest newest most voted
1

answered 2018-12-14 13:58:02 -0500

iamontheinet gravatar image

Hi!

The pipeline preview batch size is capped at the Data Collector's preview batch size--which is set to 10 by default. So in order to increase pipeline preview batch size you'll first need to change Data Collector's preview batch size by editing preview.maxBatchSize property in sdc.properties file. (Note: for the change to take effect you'll need to restart Data Collector.)

Note: For thread count issue please "ask" a new question.

Cheers, Dash

edit flag offensive delete link more

Comments

I'm not running the preview but the normal production execution. I have already tried increasing the preview batch size also to 1000 but it didn't have any effect even after restart.

hapination gravatar imagehapination ( 2018-12-16 13:07:16 -0500 )edit

Try increasing production.maxBatchSize in sdc.properties and restart data collector.

iamontheinet gravatar imageiamontheinet ( 2018-12-22 22:41:47 -0500 )edit

I have already set both the production.maxbatchsize and the preview.maxbatchsize to 1000 and restarted, but the issue remains. I'm only having this issue with the azure iot/event hub origin, other origins are working fine with large batch sizes.

hapination gravatar imagehapination ( 2019-01-07 01:30:53 -0500 )edit
add a comment
Login/Signup to Answer

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2018-12-14 05:53:55 -0500

Seen: 19 times

Last updated: Dec 14 '18

Copyright StreamSets, Inc., 2017.
Licensed under Creative Commons Attribution Share Alike 3.0
about | faq | help | privacy policy | give feedback
Powered by Askbot version 0.7.58