Ask Your Question

Is there a recommended value for JDBC Consumer "Number of Retries on SQL Error?"

asked 2018-08-15 05:18:08 -0600

neiliox gravatar image

We're losing some database events and I wonder if it's because "Number of Retries on SQL Error" is set to the default of 0. Perhaps a transient timeout situation could be causing events to be missed and not retried.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-08-15 19:01:52 -0600

metadaddy gravatar image

I don't know of a recommended value, but you should try increasing it from the default of 0. There is, of course, a trade-off in that, if the data source is indeed offline, it will take longer to recognize that.

Note that there are other possible causes of lost data with JDBC. You should carefully examine your SQL query. In particular, if the offset column is not unique, you could be missing data. For example, say your SQL query is:

SELECT * FROM mytable WHERE col1 > ${OFFSET} ORDER BY col1

If the pipeline reads all the data up to a given value of col1, then another row is added with a duplicate value of col1, then that row will not be read on the next query. This can happen with timestamps where they are not unique.

edit flag offensive delete link more


That's very useful, thank you!

neiliox gravatar imageneiliox ( 2018-08-16 03:43:19 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-08-15 05:18:08 -0600

Seen: 78 times

Last updated: Aug 15 '18