Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

sdc 3.9.1 Postgres CDC Client report Error

I installed postgresql 11.2 on CentOS 7 and followed wal2json guide https://github.com/streamsets/wal2json/blob/master/README.md to install wal2json plugin, then I created a pipeline consumed from postgres cdc client to hdfs, after start it the sdc reported the following error, anything wrongly config?

com.streamsets.pipeline.api.StageException: JDBC_00 - Cannot connect to specified database: forceUpdateStatus failed :Database connection failed when writing to copy at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.setLsnFlushed(PostgresCDCWalReceiver.java:332) at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCSource.produce(PostgresCDCSource.java:229) at com.streamsets.pipeline.api.base.configurablestage.DSource.produce(DSource.java:38) at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:295) at com.streamsets.pipeline.api.impl.CreateByRef.call(CreateByRef.java:40) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:243) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:310) at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:219) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:850) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:584) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:389) at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:533) at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:110) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75) at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:704) at com.streamsets.datacollector.execution.AbstractRunner.lambda$scheduleForRetries$0(AbstractRunner.java:361) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226) at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.postgresql.util.PSQLException: Database connection failed when writing to copy at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1013) at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:23) at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:190) at org.postgresql.core.v3.replication.V3PGReplicationStream.forceUpdateStatus(V3PGReplicationStream.java:109) at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.setLsnFlushed(PostgresCDCWalReceiver.java:329) ... 25 more Caused by: java.net.SocketException: Broken Pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at org.postgresql.core.PGStream.flush(PGStream.java:514) at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1010) ... 29 more

sdc 3.9.1 Postgres CDC Client report Error

I installed postgresql 11.2 on CentOS 7 and followed wal2json guide https://github.com/streamsets/wal2json/blob/master/README.md to install wal2json plugin, then I created a pipeline consumed from postgres cdc client to hdfs, after start it the sdc reported the following error, anything wrongly config?

com.streamsets.pipeline.api.StageException: com.streamsets.pipeline.api.StageException: JDBC_00 - Cannot connect to specified specified database: forceUpdateStatus failed failed :Database connection failed when when writing to copy at copy at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.setLsnFlushed(PostgresCDCWalReceiver.java:332) at at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCSource.produce(PostgresCDCSource.java:229) at at com.streamsets.pipeline.api.base.configurablestage.DSource.produce(DSource.java:38) at at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:295) at at com.streamsets.pipeline.api.impl.CreateByRef.call(CreateByRef.java:40) at at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:243) at at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:310) at at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:219) at at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:850) at at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:584) at at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:389) at at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:533) at at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:110) at at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75) at at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:704) at at com.streamsets.datacollector.execution.AbstractRunner.lambda$scheduleForRetries$0(AbstractRunner.java:361) at at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226) at at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33) at at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222) at at java.util.concurrent.FutureTask.run(FutureTask.java:266) at at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100) at at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at at java.lang.Thread.run(Thread.java:748) Caused by: org.postgresql.util.PSQLException: by: org.postgresql.util.PSQLException: Database connection failed when when writing to copy at copy at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1013) at at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:23) at at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:190) at at org.postgresql.core.v3.replication.V3PGReplicationStream.forceUpdateStatus(V3PGReplicationStream.java:109) at at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.setLsnFlushed(PostgresCDCWalReceiver.java:329) ... 25 more more Caused by: by: java.net.SocketException: Broken Pipe Pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at at org.postgresql.core.PGStream.flush(PGStream.java:514) at at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1010) ... 29 more

sdc 3.9.1 Postgres CDC Client report Error

I installed postgresql 11.2 on CentOS 7 and followed wal2json guide https://github.com/streamsets/wal2json/blob/master/README.md to install wal2json plugin, then I created a pipeline consumed from postgres cdc client to hdfs, after start it the sdc reported the following error, anything wrongly config?

com.streamsets.pipeline.api.StageException:

com.streamsets.pipeline.api.StageException: JDBC_00 - Cannot connect to specified
  specified database:  forceUpdateStatus failed
  failed :Database connection failed when
  when writing to copy   at
  copy
at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.setLsnFlushed(PostgresCDCWalReceiver.java:332)
    at
  at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCSource.produce(PostgresCDCSource.java:229)
    at
  at com.streamsets.pipeline.api.base.configurablestage.DSource.produce(DSource.java:38)
    at
  at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:295)
    at
  at com.streamsets.pipeline.api.impl.CreateByRef.call(CreateByRef.java:40)
    at
  at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:243)
    at
  at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:310)
    at
  at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:219)
    at
  at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:850)
    at
  at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:584)
    at
  at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:389)
    at
  at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:533)
    at
  at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:110)
    at
  at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75)
    at
  at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:704)
    at
  at com.streamsets.datacollector.execution.AbstractRunner.lambda$scheduleForRetries$0(AbstractRunner.java:361)
    at
  at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at
  at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at
  at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
  at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
    at
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at
  at java.lang.Thread.run(Thread.java:748)
 

Caused by: org.postgresql.util.PSQLException: by: org.postgresql.util.PSQLException: Database connection failed when when writing to copy at copy at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1013) at at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:23) at at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:190) at at org.postgresql.core.v3.replication.V3PGReplicationStream.forceUpdateStatus(V3PGReplicationStream.java:109) at at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.setLsnFlushed(PostgresCDCWalReceiver.java:329) ... 25 more more Caused by: by: java.net.SocketException: Broken Pipe 断开的管道 (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at at org.postgresql.core.PGStream.flush(PGStream.java:514) at at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1010) ... 29 more

sdc 3.9.1 Postgres CDC Client report Error

I installed postgresql 11.2 on CentOS 7 and followed wal2json guide https://github.com/streamsets/wal2json/blob/master/README.md to install wal2json plugin, then I created a pipeline consumed from postgres cdc client to hdfs, after start it the sdc reported the following error, anything wrongly config?

com.streamsets.pipeline.api.StageException: JDBC_00 - Cannot connect to specified database: forceUpdateStatus failed :Database connection failed when writing to copy at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.setLsnFlushed(PostgresCDCWalReceiver.java:332) at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCSource.produce(PostgresCDCSource.java:229) at com.streamsets.pipeline.api.base.configurablestage.DSource.produce(DSource.java:38) at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:295) at com.streamsets.pipeline.api.impl.CreateByRef.call(CreateByRef.java:40) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:243) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:310) at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:219) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:850) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:584) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:389) at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:533) at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:110) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75) at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:704) at com.streamsets.datacollector.execution.AbstractRunner.lambda$scheduleForRetries$0(AbstractRunner.java:361) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226) at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Caused by: org.postgresql.util.PSQLException: Database connection failed when writing to copy at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1013) at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:23) at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:190) at org.postgresql.core.v3.replication.V3PGReplicationStream.forceUpdateStatus(V3PGReplicationStream.java:109) at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.setLsnFlushed(PostgresCDCWalReceiver.java:329) ... 25 more Caused by: java.net.SocketException: 断开的管道 (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at org.postgresql.core.PGStream.flush(PGStream.java:514) at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1010) ... 29 more