pipeline run OutOfMemoryError

asked 2019-09-08 20:56:17 -0500

MarkLuo gravatar image

updated 2019-09-09 03:53:23 -0500

My pipeline is about sync mysql binary log to postgresql jdbc database, but when continuity run the pipeline , pipeline occur error ,error info :

Sep 06 09:06:45 iZbp1b5nzqo34m1gp31pjjZ streamsets[10863]: Exception in thread "blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306" java.lang.OutOfMemoryError: Java heap space Sep 06 09:07:00 iZbp1b5nzqo34m1gp31pjjZ streamsets[10863]: Exception in thread "blc-rds212fncxa86kk8p315631.mysql.rds.aliyuncs.com:3306" java.lang.OutOfMemoryError: Java heap space Sep 06 09:07:21 iZbp1b5nzqo34m1gp31pjjZ streamsets[10863]: Exception in thread "blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306" Exception in thread "blc-rm-bp1j97avoql9nu788942.mysql.rd...va heap space Sep 06 09:07:47 iZbp1b5nzqo34m1gp31pjjZ streamsets[10863]: java.lang.OutOfMemoryError: Java heap space Sep 06 09:08:33 iZbp1b5nzqo34m1gp31pjjZ streamsets[10863]: Exception in thread "blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306" Exception in thread "blc-rm-bp1j97avoql9nu788942.mysql.rd...va heap space Sep 06 09:09:00 iZbp1b5nzqo34m1gp31pjjZ streamsets[10863]: java.lang.OutOfMemoryError: Java heap space Sep 06 09:09:00 iZbp1b5nzqo34m1gp31pjjZ streamsets[10863]: Exception in thread "blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306" Exception in thread "blc-rm-bp1j97avoql9nu788942.mysql.rd...va heap space Sep 06 09:09:03 iZbp1b5nzqo34m1gp31pjjZ streamsets[10863]: java.lang.OutOfMemoryError: Java heap space Sep 06 09:09:16 iZbp1b5nzqo34m1gp31pjjZ streamsets[10863]: Exception in thread "blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306" java.lang.OutOfMemoryError: Java heap space Sep 06 09:09:33 iZbp1b5nzqo34m1gp31pjjZ streamsets[10863]: Exception in thread "blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306" java.lang.OutOfMemoryError: Java heap space

my pc info: 4 cpu 8Gb memory , java heap size is 4Gb . The version of Streamset is 3.9.0 ( built by ubuntu on 2019-06-05T21:35Z) In sdc Metrics's window,After 3 hours ,JAVA Heap Memory Usage from 600Mb to 2.3GB, What's wrong? my friends.

The pipeline is frequently restart with MYSQL_006 - MySql server error: Client requested master to start replication from position > file size; the first event 'mysql-bin.003489' at 1835234060, the last event read from '/home/mysql/data3006/mysql/mysql-bin.003489' at 4, the last byte read from '/home/mysql/data3006/mysql/mysql-bin.003489' at 4.

Is that the reason? Who tell me the reason and resolve this question?

edit retag flag offensive close merge delete