Ask Your Question
1

PostgreSQL CDC Client doesn't catch changes

asked 2019-10-01 08:55:36 -0500

Quentin gravatar image

Hi everyone,

I'm trying to -more or less- follow this example to create my own CDC pipeline between two local postgres databases: https://streamsets.com/documentation/...

I used the JDBC Multitable Consumer to do the original copy of my data, which worked perfectly.

I then created a new pipeline using the PostgreSQL CDC Client and followed the wall2json installation:

$ git clone https://github.com/streamsets/wal2jso...

$ PATH=/usr/bin/pg_config:$PATH

$ USE_PGXS=1 make

$ USE_PGXS=1 make install

As explained in https://github.com/streamsets/wal2jso..., I modified the postgresql.conf:

wal_level = logical

max_replication_slots = 1

wal_sender_timeout = 2000

And did a restart of both my postgres and my streamsets container.

I then configured my Postgres CDC origin: https://imgur.com/H4dwGyz , https://imgur.com/xoHLN1m

But when running my pipeline I now have the following error:

Error reading PostgreSQL replication stream: Database connection failed when writing to copy.

I checked the postgresql.conf again and it seemed that it should have been:

wal_sender_timeout = 2000s

I changed it, which stopped the error but now nothing happens in streamsets when running the pipeling. No more logs after:

HikariPool-2 - Starting

HikariPool-2 - Start completed.

Any idea to help me? I feel quite stuck at this step. Thank you very much.

Quentin

edit retag flag offensive close merge delete

Comments

Did you create the replication slot at postgres database side?

Torkia Boussada gravatar imageTorkia Boussada ( 2019-10-02 14:48:38 -0500 )edit

I'm experiencing the same issue - SDC automatically creates the slot, it's just not receiving the data. pg_hba.conf is accepting all host connections as well.

jparham gravatar imagejparham ( 2019-10-03 06:14:42 -0500 )edit

I was originally running this on SDC 3.10.1, I reverted back to 3.8.1 and it started working. I believe it's a bug with the 3.10 release. Hopefully this helps.

jparham gravatar imagejparham ( 2019-10-03 09:39:46 -0500 )edit

Hi @jparham, do you mind creating a ticket with all the details so our team can look into the issue http://issues.streamsets.com/? Thanks in advance!

iamontheinet gravatar imageiamontheinet ( 2019-10-04 09:40:24 -0500 )edit

Hi, just discovered these comments. Just to add that wal2json works so this is probably something with SDC (either bug or lack of configuration). I use 3.10.1 too. I'll try to open a ticket if it makes sense with the little knowledge of SDC I have.

Quentin gravatar imageQuentin ( 2019-10-04 09:57:11 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-11-08 19:17:06 -0500

bob gravatar image

Check if there are any errors in the sdc.log which might relate to this issue - such as connection issues, etc.

If there are no issues in the log, please try setting the batch size to 1. The batch size behavior was changed and that might be the problem.

edit flag offensive delete link more

Comments

That was it yes, thank you.

Quentin gravatar imageQuentin ( 2019-11-14 15:27:48 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-10-01 08:55:36 -0500

Seen: 350 times

Last updated: Nov 08 '19