Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

PostgreSQL CDC - UTF-8 Error

Hi ! I am trying to use the Postgres CDC client. Followed the instructions in the documentation. I see the following error - running the pipeline :

org.postgresql.util.PSQLException: Database connection failed when writing to copy
at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1013)
at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:23)
at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:190)
at org.postgresql.core.v3.replication.V3PGReplicationStream.forceUpdateStatus(V3PGReplicationStream.java:109)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.createReplicationStream(PostgresCDCWalReceiver.java:174)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCSource.init(PostgresCDCSource.java:156)
at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48)
at com.streamsets.pipeline.api.base.configurablestage.DStage.init(DStage.java:36)
at com.streamsets.datacollector.runner.StageRuntime.lambda$init$0(StageRuntime.java:220)
at com.streamsets.datacollector.util.LambdaUtil.withClassLoaderInternal(LambdaUtil.java:148)
at com.streamsets.datacollector.util.LambdaUtil.withClassLoader(LambdaUtil.java:44)
at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:218)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:107)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:44)
at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:392)
at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:297)
at com.streamsets.datacollector.runner.preview.PreviewPipeline.run(PreviewPipeline.java:49)
at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:230)
at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$1(AsyncPreviewer.java:98)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Illegal UTF-8 sequence: byte 2 of 3 byte sequence is not 10xxxxxx: 98
    at org.postgresql.core.UTF8Encoding.checkByte(UTF8Encoding.java:28)
    at org.postgresql.core.UTF8Encoding.decode(UTF8Encoding.java:113)
    at org.postgresql.core.PGStream.receiveString(PGStream.java:341)
    at org.postgresql.core.v3.QueryExecutorImpl.receiveNoticeResponse(QueryExecutorImpl.java:2444)
    at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1091)
    at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1011)
    ... 31 more

PostgreSQL CDC - UTF-8 Error

Hi ! I am trying to use the Postgres CDC client. Followed the instructions in the documentation. I see the following error - running the pipeline :

DataCollector Version : 3.13.0 PostgreSQL Version : 10.12 (wal2json installed)

org.postgresql.util.PSQLException: Database connection failed when writing to copy
at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1013)
at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:23)
at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:190)
at org.postgresql.core.v3.replication.V3PGReplicationStream.forceUpdateStatus(V3PGReplicationStream.java:109)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.createReplicationStream(PostgresCDCWalReceiver.java:174)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCSource.init(PostgresCDCSource.java:156)
at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48)
at com.streamsets.pipeline.api.base.configurablestage.DStage.init(DStage.java:36)
at com.streamsets.datacollector.runner.StageRuntime.lambda$init$0(StageRuntime.java:220)
at com.streamsets.datacollector.util.LambdaUtil.withClassLoaderInternal(LambdaUtil.java:148)
at com.streamsets.datacollector.util.LambdaUtil.withClassLoader(LambdaUtil.java:44)
at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:218)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:107)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:44)
at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:392)
at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:297)
at com.streamsets.datacollector.runner.preview.PreviewPipeline.run(PreviewPipeline.java:49)
at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:230)
at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$1(AsyncPreviewer.java:98)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Illegal UTF-8 sequence: byte 2 of 3 byte sequence is not 10xxxxxx: 98
    at org.postgresql.core.UTF8Encoding.checkByte(UTF8Encoding.java:28)
    at org.postgresql.core.UTF8Encoding.decode(UTF8Encoding.java:113)
    at org.postgresql.core.PGStream.receiveString(PGStream.java:341)
    at org.postgresql.core.v3.QueryExecutorImpl.receiveNoticeResponse(QueryExecutorImpl.java:2444)
    at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1091)
    at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1011)
    ... 31 more

PostgreSQL CDC - UTF-8 Error

Hi ! I am trying to use the Postgres CDC client. Followed the instructions in the documentation. I see the following error - running the pipeline :

DataCollector Version : 3.13.0 3.13.0 (Host Docker) PostgreSQL Version : 10.12 (wal2json (Host Windows 10 - wal2json compiled and installed)

org.postgresql.util.PSQLException: Database connection failed when writing to copy
at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1013)
at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:23)
at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:190)
at org.postgresql.core.v3.replication.V3PGReplicationStream.forceUpdateStatus(V3PGReplicationStream.java:109)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.createReplicationStream(PostgresCDCWalReceiver.java:174)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCSource.init(PostgresCDCSource.java:156)
at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48)
at com.streamsets.pipeline.api.base.configurablestage.DStage.init(DStage.java:36)
at com.streamsets.datacollector.runner.StageRuntime.lambda$init$0(StageRuntime.java:220)
at com.streamsets.datacollector.util.LambdaUtil.withClassLoaderInternal(LambdaUtil.java:148)
at com.streamsets.datacollector.util.LambdaUtil.withClassLoader(LambdaUtil.java:44)
at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:218)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:107)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:44)
at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:392)
at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:297)
at com.streamsets.datacollector.runner.preview.PreviewPipeline.run(PreviewPipeline.java:49)
at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:230)
at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$1(AsyncPreviewer.java:98)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Illegal UTF-8 sequence: byte 2 of 3 byte sequence is not 10xxxxxx: 98
    at org.postgresql.core.UTF8Encoding.checkByte(UTF8Encoding.java:28)
    at org.postgresql.core.UTF8Encoding.decode(UTF8Encoding.java:113)
    at org.postgresql.core.PGStream.receiveString(PGStream.java:341)
    at org.postgresql.core.v3.QueryExecutorImpl.receiveNoticeResponse(QueryExecutorImpl.java:2444)
    at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1091)
    at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1011)
    ... 31 more

EDIT (2020-04-26) :

PostgreSQL Version : 9.6.17 (Host Windows 10 - wal2json compiled and installed) DataCollector Version : 3.13.0 (Host Docker)

After installing PostgreSQL 9.6.17, recompiled wal2json, it seems to work.

I'll make some testing with PostgreSQL 11 and 12 for Windows.

PostgreSQL CDC - UTF-8 Error

Hi ! I am trying to use the Postgres CDC client. Followed the instructions in the documentation. I see the following error - running the pipeline :

DataCollector Version : 3.13.0 (Host Docker)
PostgreSQL Version : 10.12 (Host Windows 10 - wal2json compiled and installed)

installed)

org.postgresql.util.PSQLException: Database connection failed when writing to copy
at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1013)
at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:23)
at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:190)
at org.postgresql.core.v3.replication.V3PGReplicationStream.forceUpdateStatus(V3PGReplicationStream.java:109)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.createReplicationStream(PostgresCDCWalReceiver.java:174)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCSource.init(PostgresCDCSource.java:156)
at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48)
at com.streamsets.pipeline.api.base.configurablestage.DStage.init(DStage.java:36)
at com.streamsets.datacollector.runner.StageRuntime.lambda$init$0(StageRuntime.java:220)
at com.streamsets.datacollector.util.LambdaUtil.withClassLoaderInternal(LambdaUtil.java:148)
at com.streamsets.datacollector.util.LambdaUtil.withClassLoader(LambdaUtil.java:44)
at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:218)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:107)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:44)
at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:392)
at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:297)
at com.streamsets.datacollector.runner.preview.PreviewPipeline.run(PreviewPipeline.java:49)
at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:230)
at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$1(AsyncPreviewer.java:98)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Illegal UTF-8 sequence: byte 2 of 3 byte sequence is not 10xxxxxx: 98
    at org.postgresql.core.UTF8Encoding.checkByte(UTF8Encoding.java:28)
    at org.postgresql.core.UTF8Encoding.decode(UTF8Encoding.java:113)
    at org.postgresql.core.PGStream.receiveString(PGStream.java:341)
    at org.postgresql.core.v3.QueryExecutorImpl.receiveNoticeResponse(QueryExecutorImpl.java:2444)
    at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1091)
    at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1011)
    ... 31 more

EDIT (2020-04-26) :

(2020-04-26)

PostgreSQL Version : 9.6.17 (Host Windows 10 - wal2json compiled and installed)
DataCollector Version : 3.13.0 (Host Docker)

Docker)

After installing PostgreSQL 9.6.17, recompiled wal2json, it seems to work.

I'll make some testing with PostgreSQL 11 and 12 for Windows.

PostgreSQL CDC - UTF-8 Error

Hi ! I am trying to use the Postgres CDC client. Followed the instructions in the documentation. I see the following error - running the pipeline :

DataCollector Version : 3.13.0 (Host Docker)
PostgreSQL Version : 10.12 (Host Windows 10 - wal2json compiled and installed)

org.postgresql.util.PSQLException: Database connection failed when writing to copy
at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1013)
at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:23)
at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:190)
at org.postgresql.core.v3.replication.V3PGReplicationStream.forceUpdateStatus(V3PGReplicationStream.java:109)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.createReplicationStream(PostgresCDCWalReceiver.java:174)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCSource.init(PostgresCDCSource.java:156)
at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48)
at com.streamsets.pipeline.api.base.configurablestage.DStage.init(DStage.java:36)
at com.streamsets.datacollector.runner.StageRuntime.lambda$init$0(StageRuntime.java:220)
at com.streamsets.datacollector.util.LambdaUtil.withClassLoaderInternal(LambdaUtil.java:148)
at com.streamsets.datacollector.util.LambdaUtil.withClassLoader(LambdaUtil.java:44)
at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:218)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:107)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:44)
at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:392)
at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:297)
at com.streamsets.datacollector.runner.preview.PreviewPipeline.run(PreviewPipeline.java:49)
at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:230)
at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$1(AsyncPreviewer.java:98)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Illegal UTF-8 sequence: byte 2 of 3 byte sequence is not 10xxxxxx: 98
    at org.postgresql.core.UTF8Encoding.checkByte(UTF8Encoding.java:28)
    at org.postgresql.core.UTF8Encoding.decode(UTF8Encoding.java:113)
    at org.postgresql.core.PGStream.receiveString(PGStream.java:341)
    at org.postgresql.core.v3.QueryExecutorImpl.receiveNoticeResponse(QueryExecutorImpl.java:2444)
    at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1091)
    at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1011)
    ... 31 more

EDIT (2020-04-26)

PostgreSQL Version : 9.6.17 (Host Windows 10 - wal2json compiled and installed)
DataCollector Version : 3.13.0 (Host Docker)

After installing PostgreSQL 9.6.17, recompiled wal2json, it seems to work.

I'll make some testing with PostgreSQL 11 and 12 for Windows.Windows

EDIT (2020-04-27)

PostgreSQL Version : 9.6.17 (Host Windows 10 - wal2json compiled and installed)
DataCollector Version : 3.13.0 (Host Docker)

It doesn't work ... I've got the same error in sdc logs, but this time the pipeline doesn't stop on error ...

PostgreSQL CDC - UTF-8 Error

Hi ! I am trying to use the Postgres CDC client. Followed the instructions in the documentation. I see the following error - running the pipeline :

DataCollector Version : 3.13.0 (Host Docker)
PostgreSQL Version : 10.12 (Host Windows 10 - wal2json compiled and installed)

org.postgresql.util.PSQLException: Database connection failed when writing to copy
at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1013)
at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:23)
at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:190)
at org.postgresql.core.v3.replication.V3PGReplicationStream.forceUpdateStatus(V3PGReplicationStream.java:109)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCWalReceiver.createReplicationStream(PostgresCDCWalReceiver.java:174)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.postgres.PostgresCDCSource.init(PostgresCDCSource.java:156)
at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48)
at com.streamsets.pipeline.api.base.configurablestage.DStage.init(DStage.java:36)
at com.streamsets.datacollector.runner.StageRuntime.lambda$init$0(StageRuntime.java:220)
at com.streamsets.datacollector.util.LambdaUtil.withClassLoaderInternal(LambdaUtil.java:148)
at com.streamsets.datacollector.util.LambdaUtil.withClassLoader(LambdaUtil.java:44)
at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:218)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:107)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:44)
at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:392)
at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:297)
at com.streamsets.datacollector.runner.preview.PreviewPipeline.run(PreviewPipeline.java:49)
at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:230)
at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$1(AsyncPreviewer.java:98)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Illegal UTF-8 sequence: byte 2 of 3 byte sequence is not 10xxxxxx: 98
    at org.postgresql.core.UTF8Encoding.checkByte(UTF8Encoding.java:28)
    at org.postgresql.core.UTF8Encoding.decode(UTF8Encoding.java:113)
    at org.postgresql.core.PGStream.receiveString(PGStream.java:341)
    at org.postgresql.core.v3.QueryExecutorImpl.receiveNoticeResponse(QueryExecutorImpl.java:2444)
    at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1091)
    at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1011)
    ... 31 more

EDIT (2020-04-26)

PostgreSQL Version : 9.6.17 (Host Windows 10 - wal2json compiled and installed)
DataCollector Version : 3.13.0 (Host Docker)

After installing PostgreSQL 9.6.17, recompiled wal2json, it seems to work.

I'll make some testing with PostgreSQL 11 and 12 for Windows

EDIT (2020-04-27)

PostgreSQL Version : 9.6.17 (Host Windows 10 - wal2json compiled and installed)
DataCollector Version : 3.13.0 (Host Docker)

It doesn't work ... I've got the same error in sdc logs, but this time the pipeline doesn't stop on error ...

EDIT (2020-04-28)

PostgreSQL Version : 9.6.17 (Host Windows 10 - wal2json compiled and installed)
DataCollector Version : 3.13.0 (Host Docker)

I've tested my wal2json.dll on PostgreSQL (compiled especially for this version).

To test the replication slot with wal2json you can do this

-- Create the replication slot with wal2json plugin
SELECT * FROM pg_create_logical_replication_slot('testslot', 'wal2json');

-- Check if the replication slot is created
SELECT slot_name, plugin, slot_type, database, active, restart_lsn FROM pg_replication_slots;

-- Make a change in your database
UPDATE tabletochange SET columntochange = 'yourdata' WHERE id = 1;

-- Check modified data in the slot
SELECT * FROM pg_logical_slot_peek_changes('testslot', NULL, NULL, 'include-timestamp', 'on');

-- Drop the test slot
SELECT pg_drop_replication_slot('testslot');

In my case, it work perfectly ! you can see datas in the slot

image description

I don't really know if it's a encoding problem.

My PostgreSQL server have the "client-encoding" value sets to "UTF-8" for both session and global and all my "LC_*" parameters are sets to "French_France.1252".