Ask Your Question
0

postgresql cdc client loss some data

asked 2020-01-09 06:29:00 -0600

shixinbao gravatar image

updated 2020-01-09 20:14:26 -0600

hello, I use sdc(v3.9.1) to collect data of postgresql cdc(use default params).But when postgresql generated a large number of logs in a short time, the processor loss some data within the time difference between batches. At first I thought that wal2json didn't collect logs, and then I used debezium and streamsets to collect data from this database at the same time. The results showed that debezium collected all data change records, while streamsets lost some data. Interestingly, I found that the missing data was generated between batches.

It seems that "query timeout" doesn't work

image description

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2020-01-10 15:51:54 -0600

metadaddy gravatar image

Please file a P1 bug at issues.streamsets.com with as much detail as possible.

edit flag offensive delete link more

Comments

hello pat! I have already raised the issue at issues.streamsets.com. I think this is a very serious bug. This issue is bothering us. I hope you can solve it as soon as possible

shixinbao gravatar imageshixinbao ( 2020-01-13 05:40:35 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-01-09 06:29:00 -0600

Seen: 32 times

Last updated: Jan 10