Oracle CDC client origin with Redo Logs option not able to capture changes

asked 2018-09-18 02:03:36 -0500

Bijoy
11 ●1 ●2

I am trying to use Oracle CDC client origin to connect to my Oracle database and redirect the changes captured by it to the file system.When I use Online data dictionary option,the pipeline is able to track changes to the database (record insertin,deletion,updation,etc.) and write to filesystem. But when I use the Redo logs option,the pipeline is neither able to detect schema changes nor data insertion/deletion/updation.Last 2 lines of the logs are as below:

Trying to start LogMiner with start date: 18-09-2018 06:00:53 and end date: 18-09-2018 06:00:53

Attempting to generate records

I have followed all the below prerequistes mentioned in the below Streamsets link,but changes are not getting captured.: https://streamsets.com/documentation/datacollector/3.4.0/help/datacollector/UserGuide/Origins/OracleCDC.html

1) Enable LogMiner. 2) Enable supplemental logging for the database or tables. 3) Create a user account with the required roles and privileges. 4) To use the dictionary in redo logs, extract the Log Miner dictionary. 5) Install the Oracle JDBC driver.

answered 2018-09-18 13:12:23 -0500

hshreedharan
96 ●1

Unfortunately, when dictionary from redo logs is used, the origin is much slower. This is not due to any of the code in the origin (there is no difference in logic as far as SDC is concerned), it is entirely within Oracle that it is slow. Oracle just takes much longer to return data when redo log dictionary is chosen. There is not a whole lot we can do.

Also, looking at your logs, you seem to be using an older SDC version. Not that it will make a huge difference, can you try SDC 3.4.3 in which we changed some of the logminer window logic?

edit flag offensive delete link

Comments

Thanks for your response.However,my issue is not with slowness but rather with the Oracle CDC origin tool not able to detect changes in the Oracle table when using Redo logs option.Can we debug why its unable to capture database changes? Also,do you think the version upgrade will resolve this issue?

Bijoy ( 2018-09-19 01:01:07 -0500 )edit

I upgraded Streamsets to 3.5.0. But even after that,the Oracle CDC client(with Redo logs dictionary source option) is unable to detect DDL changes (adding columns,dropping columns).It is able to detect insert/update/delete to the rows,but cannot detect changes to the table schema.Can anyone help?

Bijoy ( 2018-10-03 04:57:59 -0500 )edit

add a comment

Oracle CDC client origin with Redo Logs option not able to capture changes

1 Answer

Comments

Question Tools

Stats

Related questions

Oracle CDC client origin with Redo Logs option not able to capture changes edit

1 Answer

Comments

Question Tools

Stats

Related questions

Oracle CDC client origin with Redo Logs option not able to capture changes