Ask Your Question
2

JDBC multitable does not reflect the updated data(CDC) in the destination (JDBC producer).

asked 2017-10-17 02:50:14 -0600

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I am using JDBC multitable consumer origin and JDBC producer destination .I tried Insert operation which gets data written in the destination table.But UPDATE and DELETE is not reflected in the destination but it is visible only in streamsets preview.Kindly help me to fix this.

Thank You

edit retag flag offensive close merge delete

Comments

Are you seeing any errors in sdc.log?

metadaddy gravatar imagemetadaddy ( 2017-10-17 13:22:49 -0600 )edit

No there were no errors. Streamsets shows running successfully,records fetched.But its not visible in the destination table.Only INSERT operation is reflected.Only visible in preview.

Maithri gravatar imageMaithri ( 2017-10-19 23:35:58 -0600 )edit

Even with JDBC Query Consumer I am facing the same problem.Only INSERT operation is reflected in the destination table,not UPDATE and DELETE.

Maithri gravatar imageMaithri ( 2017-10-19 23:39:58 -0600 )edit

2 Answers

Sort by ยป oldest newest most voted
1

answered 2017-10-20 11:10:19 -0600

jeff gravatar image

It sounds like you are trying to do CDC, but that's not going to work from either JDBC query or table origin. Those origins are simply producing records from data in a table, not capturing changes to that underlying table. You need to use a CDC origin for your RDBMS server (if available).

edit flag offensive delete link more

Comments

Thank you Jeff,may be that is the problem here. I referred this documentation https://streamsets.com/documentation/datacollector/latest/help/#Pipeline_Design/CDC-Overview.html#concept_lfb_phd_ty ,this is helpful.

Maithri gravatar imageMaithri ( 2017-10-22 23:47:22 -0600 )edit
0

answered 2018-01-16 08:27:09 -0600

Maithri gravatar image
edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-10-17 02:50:14 -0600

Seen: 363 times

Last updated: Jan 17