MySQL Binary Log throw EOFException and Pipeline not work but status is RUNNING

asked 2020-07-17 01:19:58 -0500

gavin gravatar image

updated 2020-07-31 21:35:53 -0500

SDC Version :3.16

i have config pipline that read from mysqlblog and write to postgresql

srouce :MySQL Binary Log

desc:jdbc producer

this pipeline is Uptime 3 days unitl throw EOFException, pipeline looks like blocked or choked, and running icon stop on [MySQL Binary Log] stage. and [MySQL Binary Log] stage OutPut count not change.

only restart this pipline, record can read from [mysql Binary log].

what happened ? i guess that thead deadlock or endless loop or thread is destroyed ?

How to prevent this problem from happening again?

How to config the pipeline automatically restart in this case?

How to get notification and restart by manual?

mysql binary log.config

"configuration" : [ { "name" : "config.hostname", "value" : "rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com" }, { "name" : "config.port", "value" : "3306" }, { "name" : "config.username", "value" : "" }, { "name" : "config.password", "value" : "" }, { "name" : "config.serverId", "value" : "20086" }, { "name" : "config.maxBatchSize", "value" : 100 }, { "name" : "config.maxWaitTime", "value" : 2000 }, { "name" : "config.connectTimeout", "value" : 5000 }, { "name" : "config.enableKeepAlive", "value" : false }, { "name" : "config.keepAliveInterval", "value" : 600000 }, { "name" : "config.useSsl", "value" : false }, { "name" : "config.startFromBeginning", "value" : false }, { "name" : "config.initialOffset", "value" : "88b548c5-6c42-11e7-9a22-6c92bf2c1831:1-131672166,a6fc3562-6c42-11e7-9a23-7cd30ac4dabc:1-329938921" }, { "name" : "config.includeTables", "value" : "wms_gzrz_%.customer,wms_gzrz_%.vendor,wms_gzrz_%.shipping_order,wms_gzrz_%.shipping_trade_partner,wms_gzrz_%.sys_code_detail,wms_gzrz_%.shipping_order_detail,wms_gzrz_%.shipping_pick_detail,wms_gzrz_%.lot_attribute,wms_gzrz_%.sku,wms_gzrz_%.sku_rule,wms_gzrz_%.sku_udf,wms_gzrz_%.shipping_udf,wms_gzrz_%.receipt,wms_gzrz_%.receipt_detail,wms_gzrz_%.receipt_detail_udf,wms_gzrz_%.receipt_detail_history,wms_gzrz_%.pack,wms_gzrz_%.inv_trans,wms_gzrz_%.pack_detail,scmoms7_gzrz.receipt,scmoms7_gzrz.shipping_detail,scmoms7_gzrz.sale_order_detail,scmoms7_gzrz.sale_order_udf,scmoms7_gzrz.receipt_detail,scmoms7_gzrz.receipt_detail_lot,scmoms7_gzrz.balance_route_order,scmoms7_gzrz.balance_route_delivery_circle,scmoms7_gzrz.balance_route_cus_detail,scmoms7_gzrz.purchase_order,scmoms7_gzrz.receipt_udf,rolemanage.wh_config,wms_gzrz_%.inv_lot_loc_lpn_case,wms_gzrz_%.location,wms_gzrz_%.OWNER,wms_gzrz_%.receipt_udf,wms_gzrz_%.sku_barcode,wms_gzrz_%.wh_config,scmoms7_gzrz.sale_detail_udf,scmoms7_gzrz.sale_order,scmoms7_gzrz.shipping,scmoms7_gzrz.status_value,sync_wms.time_sync" }, { "name" : "config.ignoreTables", "value" : null }, { "name" : "stageOnRecordError", "value" : "TO_ERROR" } ]

logs:

after throw error , not record generate , unitl i stop and start the pipeline by manual.

2020-07-16 09:44:21,624 data sync - gzrz-wms-sum-to-1-wh/datasync6dba2d3c-c1b8-4e2d-adb3-fdfb4913d4a4 WARN Encountered unsupported mysql type char(10) MysqlType *admin blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306

2020-07-16 10:24:11,530 data sync - gzrz-wms-sum-to-1-wh/datasync6dba2d3c-c1b8-4e2d-adb3-fdfb4913d4a4 WARN Encountered unsupported mysql type char(1) MysqlType *admin blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306

2020-07-16 10:24:45,408 data sync - gzrz-wms-sum-to-1-wh/datasync6dba2d3c-c1b8-4e2d-adb3-fdfb4913d4a4 WARN Encountered unsupported mysql type char(36) MysqlType *admin blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306

2020-07-16 10:24:45,452 data sync - gzrz-wms-sum-to-1-wh/datasync6dba2d3c-c1b8-4e2d-adb3-fdfb4913d4a4 WARN Encountered unsupported mysql type char(36) MysqlType *admin blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306

2020-07-16 10:25:49,561 data sync - gzrz-wms-sum-to-1-wh/datasync6dba2d3c-c1b8-4e2d-adb3-fdfb4913d4a4 WARN Encountered unsupported mysql type char(36) MysqlType *admin blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306

2020-07-16 10:53:00,853 data sync - gzrz-wms-sum-to-1-wh/datasync6dba2d3c-c1b8-4e2d-adb3-fdfb4913d4a4 WARN Encountered unsupported mysql type char(36) MysqlType *admin blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306

2020-07-16 10:58:57,871 data sync - gzrz-wms-sum-to-1-wh/datasync6dba2d3c-c1b8-4e2d-adb3-fdfb4913d4a4 WARN Encountered unsupported mysql type char(36) MysqlType *admin blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306

2020-07-16 11:22:33,296 data sync - gzrz-wms-sum-to-1-wh/datasync6dba2d3c-c1b8-4e2d-adb3-fdfb4913d4a4 WARN Encountered unsupported mysql type char(36) MysqlType *admin blc-rm-bp1j97avoql9nu788942.mysql.rds.aliyuncs.com:3306

2020-07-16 11:31:04,886 data sync - gzrz-wms-sum-to-1-wh/datasync6dba2d3c-c1b8-4e2d-adb3-fdfb4913d4a4 WARN Encountered unsupported mysql ... (more)

edit retag flag offensive close merge delete