Ask Your Question
1

ERROR PostgresWalRunner - Error reading PostgreSQL replication stream: Database connection failed when reading from copy [closed]

asked 2020-02-07 20:27:23 -0600

umanohar gravatar image

updated 2020-02-07 20:38:12 -0600

metadaddy gravatar image

I am trying to use the Postgres CDC client. Followed the instructions in the documentation. I see the following error - running the pipeline,

com.streamsets.pipeline.api.StageException: JDBC_00 - Cannot connect to specified database:  forceUpdateStatus failed :Database connection failed when writing to copy
com.streamsets.pipeline.api.StageException: JDBC_00 - Cannot connect to specified database:  forceUpdateStatus failed :Database connection failed when writing to copy
    at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.setLsnFlushed(PostgresCDCWalReceiver.java:332)
    at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCSource.produce(PostgresCDCSource.java:232)
    at com.streamsets.pipeline.api.base.configurablestage.DSource.produce(DSource.java:38)
    at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:296)
    at com.streamsets.pipeline.api.impl.CreateByRef.call(CreateByRef.java:40)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:244)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:311)
    at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:220)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:854)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:588)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:390)
    at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:516)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:112)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75)
    at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:720)
    at com.streamsets.datacollector.execution.AbstractRunner.lambda$scheduleForRetries$0(AbstractRunner.java:363)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: Database connection failed when writing to copy
    at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1013)
    at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:23)
    at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:190)
    at org.postgresql.core.v3.replication.V3PGReplicationStream.forceUpdateStatus(V3PGReplicationStream.java:109)
    at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.setLsnFlushed(PostgresCDCWalReceiver.java:329)
    ... 25 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
    at org.postgresql.core.PGStream.flush(PGStream.java:514)
    at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1010

What am I missing? Thank you

edit retag flag offensive reopen merge delete

Closed for the following reason the question is answered, right answer was accepted by umanohar
close date 2020-02-10 07:12:10.962001

Comments

Could you add some details on the version of PostgreSQL you are working with. Is it a local installation? In the cloud? Did you follow the instructions in the documentation to build and install wal2json? https://github.com/streamsets/wal2json/blob/master/README.md .

metadaddy gravatar imagemetadaddy ( 2020-02-07 20:43:36 -0600 )edit

The version is postgres 9.6.2 (Postgres Pro 9.6 from aws marketplace). Followed the instructions on that page (Except that they do not work as is and had to modify makefile as well as other configuration parameters such as max_wal_senders that are not mentioned there)

umanohar gravatar imageumanohar ( 2020-02-07 20:49:31 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2020-02-10 07:10:35 -0600

umanohar gravatar image

Appears that I had to increase the wal_sender_timeout. The error message from streamsets is not very clear

edit flag offensive delete link more

Question Tools

1 follower

Stats

Asked: 2020-02-07 20:27:23 -0600

Seen: 30 times

Last updated: Feb 10