PostgreSQL CDC Client doesn't catch changes

asked 2019-10-01 08:55:36 -0500

Quentin gravatar image

Hi everyone,

I'm trying to -more or less- follow this example to create my own CDC pipeline between two local postgres databases: https://streamsets.com/documentation/...

I used the JDBC Multitable Consumer to do the original copy of my data, which worked perfectly.

I then created a new pipeline using the PostgreSQL CDC Client and followed the wall2json installation:

$ git clone https://github.com/streamsets/wal2jso...

$ PATH=/usr/bin/pg_config:$PATH

$ USE_PGXS=1 make

$ USE_PGXS=1 make install

As explained in https://github.com/streamsets/wal2jso..., I modified the postgresql.conf:

wal_level = logical

max_replication_slots = 1

wal_sender_timeout = 2000

And did a restart of both my postgres and my streamsets container.

I then configured my Postgres CDC origin: https://imgur.com/H4dwGyz , https://imgur.com/xoHLN1m

But when running my pipeline I now have the following error:

Error reading PostgreSQL replication stream: Database connection failed when writing to copy.

I checked the postgresql.conf again and it seemed that it should have been:

wal_sender_timeout = 2000s

I changed it, which stopped the error but now nothing happens in streamsets when running the pipeling. No more logs after:

HikariPool-2 - Starting

HikariPool-2 - Start completed.

Any idea to help me? I feel quite stuck at this step. Thank you very much.

Quentin

edit retag flag offensive close merge delete

Comments

Did you create the replication slot at postgres database side?

Torkia Boussada gravatar imageTorkia Boussada ( 2019-10-02 14:48:38 -0500 )edit

I'm experiencing the same issue - SDC automatically creates the slot, it's just not receiving the data. pg_hba.conf is accepting all host connections as well.

jparham gravatar imagejparham ( 2019-10-03 06:14:42 -0500 )edit

I was originally running this on SDC 3.10.1, I reverted back to 3.8.1 and it started working. I believe it's a bug with the 3.10 release. Hopefully this helps.

jparham gravatar imagejparham ( 2019-10-03 09:39:46 -0500 )edit

Hi @jparham, do you mind creating a ticket with all the details so our team can look into the issue http://issues.streamsets.com/? Thanks in advance!

iamontheinet gravatar imageiamontheinet ( 2019-10-04 09:40:24 -0500 )edit

Hi, just discovered these comments. Just to add that wal2json works so this is probably something with SDC (either bug or lack of configuration). I use 3.10.1 too. I'll try to open a ticket if it makes sense with the little knowledge of SDC I have.

Quentin gravatar imageQuentin ( 2019-10-04 09:57:11 -0500 )edit