Ask Your Question

sdc kudu destination buffer too big

asked 2019-07-09 20:17:58 -0600

lichanghong gravatar image

updated 2019-07-09 20:18:52 -0600

Hello : I use kudu as the destination of my pipeline 。But occationally in the pipeline's log it reads that

"Errors while interacting with Kudu: org.apache.kudu.client.NonRecoverableException: MANUAL_FLUSH is enabled but the buffer is too big" 。

How should I figure out the problem 。

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted

answered 2019-07-10 21:17:28 -0600

lichanghong gravatar image

Finally I get the answer . while interacting with Kudu ,kudu client has a default config----"Mutation Buffer Space " which is 1000.this config control the size of one flush of kudu。 At ordinary times,the pipeline runs smoothly,But when DBA operate the databases Manully,delete many rows accordingly to some conditions once. If the number of deleted rows exceed the config of "Mutation Buffer Space " .then the above issue occurs.

edit flag offensive delete link more


Thanks for coming back and posting the solution!

metadaddy gravatar imagemetadaddy ( 2019-07-10 21:24:52 -0600 )edit

@metadaddy, @lichanghong....We are also getting similar error while loading data to Kudu destination. Where do we need to change Mutation Buffer Space property ??

srinath_222 gravatar imagesrinath_222 ( 2020-08-04 15:35:12 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-07-09 20:17:58 -0600

Seen: 491 times

Last updated: Jul 10 '19