MySQL Binary Log Origin Running Slow

asked 2018-07-02 09:36:55 -0500

Pradip gravatar image

updated 2018-07-02 10:07:35 -0500

metadaddy gravatar image

I have pipeline using MySQL Binlog Origin -> Kafka Producer, and database where binlog is reading having lot of transactions happening and found that binlog reader is very much behind on reading the binlogs from the database. Over time it almost got 20 minutes delay.

Also tried to improve the performance playing with Max Batch Size/Batch Wait Time etc but that did not help much.

Also available pipeline runners are always 1/1 so could not increase the pipeline runners as well.

Any suggestions on how to tune this SDC pipeline so that binlog is read immediately after it is written to binlog file on the db server and avoid such lag?

edit retag flag offensive close merge delete