JDBC multitable does not reflect the updated data(CDC) in the destination (JDBC producer).

asked 2017-10-17 02:50:14 -0600

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I am using JDBC multitable consumer origin and JDBC producer destination .I tried Insert operation which gets data written in the destination table.But UPDATE and DELETE is not reflected in the destination but it is visible only in streamsets preview.Kindly help me to fix this.

Thank You

edit retag flag offensive close merge delete

Comments

Are you seeing any errors in sdc.log?

metadaddy ( 2017-10-17 13:22:49 -0600 )edit

No there were no errors. Streamsets shows running successfully,records fetched.But its not visible in the destination table.Only INSERT operation is reflected.Only visible in preview.

Maithri ( 2017-10-19 23:35:58 -0600 )edit

Even with JDBC Query Consumer I am facing the same problem.Only INSERT operation is reflected in the destination table,not UPDATE and DELETE.

Maithri ( 2017-10-19 23:39:58 -0600 )edit

add a comment

1

answered 2018-01-16 08:27:09 -0600

Maithri

368 ●23 ●32 ●48 http://seedolabs.com/

updated 2018-06-27 18:11:13 -0600

metadaddy

2349 ●23 ●31 ●58 https://about.me/patpa...

There is now a better option for CDC on MySQL - the MySQL Binary Log origin.

edit flag offensive delete link

add a comment

1

answered 2017-10-20 11:10:19 -0600

jeff
1967 ●17 ●30 ●53

It sounds like you are trying to do CDC, but that's not going to work from either JDBC query or table origin. Those origins are simply producing records from data in a table, not capturing changes to that underlying table. You need to use a CDC origin for your RDBMS server (if available).

edit flag offensive delete link

Comments

Thank you Jeff,may be that is the problem here. I referred this documentation https://streamsets.com/documentation/datacollector/latest/help/#Pipeline_Design/CDC-Overview.html#concept_lfb_phd_ty ,this is helpful.

Maithri ( 2017-10-22 23:47:22 -0600 )edit

add a comment

JDBC multitable does not reflect the updated data(CDC) in the destination (JDBC producer).

Comments

2 Answers

Comments

Question Tools

Stats

Related questions

JDBC multitable does not reflect the updated data(CDC) in the destination (JDBC producer). edit

Comments

2 Answers

Comments

Question Tools

Stats

Related questions

JDBC multitable does not reflect the updated data(CDC) in the destination (JDBC producer).