Ask Your Question

Why does a JDBC Multitable origin read the last record repeatedly?

asked 2018-08-06 11:02:23 -0600

Dean Han gravatar image

updated 2018-08-16 10:25:54 -0600

metadaddy gravatar image

I configure a pipeline with JDBC Multitable Consumer, source DB is SQL Server, set 2 override offset columns (stamp,id). Sometimes it is working fine, new records which are inserted/updated in DB can be correctly read into destination. But sometimes it is running unexpectedly, the origin will always read the last record repeatedly and write the same record into destination. Why is this happened? Please help. Thanks.

INFO Executing Query :select * from dbo.test1 WHERE ( ( chstamp > ? ) or ( chstamp = ? and id > ? ) ) ORDER by chstamp, id

DEBUG Parameter Types And Values [(93,1533637810043<n>), (93,1533637810043<n>), (4,5)]

DEBUG Table dbo.test1 exhausted

DEBUG Table dbo.test1 read 1 number of batches from the fetched result set

DEBUG Saving offset {$com.streamsets.pipeline.stage.origin.jdbc.table.TableJdbcSource.offset.version$=2, tableName=dbo.test1;;;partitioned=false;;;partitionSequence=-1;;;partitionStartOffsets=chstamp=1533571200000,id=0;;;partitionMaxOffsets=;;;usingNonIncrementalLoad=false=chstamp=1533637810043<n>::id=5} for pipeline DeanJDBCMultitablecopy1bf366ce-5c0c-4aaa-80df-818c336a1380

DEBUG Closing statement and result set for : dbo.test1

DEBUG Cannot create new partition for (tableName=dbo.test1;;;partitioned=false;;;partitionSequence=-1;;;partitionStartOffsets=chstamp=1533571200000,id=0;;;partitionMaxOffsets=;;;usingNonIncrementalLoad=false) because it is not partitionable, and the underlying table isnot partitionable, or it has been disabled

INFO Executing Query :select * from dbo.test1 WHERE ( ( chstamp > ? ) or ( chstamp = ? and id > ? ) ) ORDER by chstamp, id

I found the reason, but have not found the solution. Even when there are no new records selected, see 'Table dbo.test1 exhausted', but still read the 1 number from the result set, then write into destination. Don't know why. Continue working on it.

edit retag flag offensive close merge delete


I moved the information from your answer to your question. Please add additional info by editing your question rather than adding an answer.

metadaddy gravatar imagemetadaddy ( 2018-08-16 10:25:23 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2018-08-06 23:51:43 -0600

Maithri gravatar image

I suggest you to use a pipeline finisher

edit flag offensive delete link more


Thank you for your suggestion, Maithri, but I need the pipeline running all time to detect the DB, once there is new records inserted/updated, they can be reflected in destination ASAP.

Dean Han gravatar imageDean Han ( 2018-08-07 00:22:01 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-08-06 11:02:23 -0600

Seen: 270 times

Last updated: Aug 16 '18