Ask Your Question

JDBC Producer keeps getting all the data in incremental mode

asked 2017-05-09 13:57:50 -0500

metadaddy gravatar image

The pipeline just keeps looping over the same data, even though it’s set to Incremental Mode. There are 6M records in my table; I stopped it at 10M.

This is my configured query:

SELECT "timestamp", "id" 
FROM pages
WHERE received_at > '${OFFSET}'
ORDER BY received_at DESC
LIMIT 1000
edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted

answered 2017-05-09 14:00:47 -0500

metadaddy gravatar image

SDC saves the value of the specified offset column from the last row retrieved to use in the next query. Since you've requested the results in descending order, the saved offset will be from the FIRST record in the database.

Remove DESC from your query and it should work fine.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-05-09 13:57:50 -0500

Seen: 1,202 times

Last updated: May 09 '17