Ask Your Question
0

sdc 3.9.1 Postgres CDC Client report Error

asked 2019-07-11 01:19:43 -0600

casel.chen gravatar image

updated 2019-07-11 01:23:22 -0600

I installed postgresql 11.2 on CentOS 7 and followed wal2json guide https://github.com/streamsets/wal2jso... to install wal2json plugin, then I created a pipeline consumed from postgres cdc client to hdfs, after start it the sdc reported the following error, anything wrongly config?

com.streamsets.pipeline.api.StageException: JDBC_00 - Cannot connect to specified database: forceUpdateStatus failed :Database connection failed when writing to copy at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.setLsnFlushed(PostgresCDCWalReceiver.java:332) at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCSource.produce(PostgresCDCSource.java:229) at com.streamsets.pipeline.api.base.configurablestage.DSource.produce(DSource.java:38) at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:295) at com.streamsets.pipeline.api.impl.CreateByRef.call(CreateByRef.java:40) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:243) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:310) at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:219) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:850) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:584) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:389) at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:533) at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:110) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75) at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:704) at com.streamsets.datacollector.execution.AbstractRunner.lambda$scheduleForRetries$0(AbstractRunner.java:361) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226) at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.postgresql.util.PSQLException: Database connection failed when writing to copy at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1013) at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:23) at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:190) at org.postgresql.core.v3.replication.V3PGReplicationStream.forceUpdateStatus(V3PGReplicationStream.java:109) at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.setLsnFlushed(PostgresCDCWalReceiver.java:329) ... 25 more Caused by: java.net.SocketException: 断开的管道 (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at org.postgresql.core.PGStream.flush(PGStream.java:514) at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1010) ... 29 more

edit retag flag offensive close merge delete

Comments

Same issue with me also. postgres 11.4

ahmed_alashrafy gravatar imageahmed_alashrafy ( 2019-07-28 23:58:51 -0600 )edit

1 Answer

Sort by » oldest newest most voted
0

answered 2020-01-11 21:42:31 -0600

Dean Han gravatar image

same here. When this issue occurred, were you fetching CDC and also inserting large data into PG at the same time?

edit flag offensive delete link more

Comments

looks like I found the reason. Try to increase the shared_buffers in postgresql.conf, I increase this value from 128M to 1024M, this error was gone.

Dean Han gravatar imageDean Han ( 2020-01-11 22:07:14 -0600 )edit

and also try to increase the CPU and memory at your running server machine

Dean Han gravatar imageDean Han ( 2020-01-12 08:16:43 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-07-11 01:19:43 -0600

Seen: 130 times

Last updated: Jan 11