Ask Your Question

Can we set a lower configuration value for the pipeline than that is set by the Cloudera Manager?

asked 2018-03-02 04:46:02 -0600

nimmy bosco gravatar image

when data collector is configured with Cloudera Manager,what is the impact of setting a property manually? I have seen in the document that "Manual changes to Data Collector configuration files can be overwritten by Cloudera Manager" . So if I set property value less than that in the cloudera manager which will be actually taken by the pipeline ? for example my max batch size is set to 10000 in cloudera manager and I try to run the pipeline with batch size set to 100 in streamsets UI, in this case what will be the actual batch size of the pipeline when it runs?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-03-16 09:47:23 -0600

jeff gravatar image

The production.maxBatchSize property, which you're referring to in the context of Cloudera Manager, is a global maximum batch size. The purpose is to prevent issues that can arise when using extremely large batches, especially in multiple pipelines. If you set a max batch size to a lower value for a particular origin in a particular pipeline (ex: the Directory origin), then that setting will take effect, provided it is lower than the global property setting.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-03-02 04:46:02 -0600

Seen: 480 times

Last updated: Mar 16 '18