JDBC Query Consumer can not detect new data of MariaDB

asked 2018-03-13 03:06:36 -0600

Ida gravatar image

updated 2018-03-13 12:54:57 -0600

metadaddy gravatar image

I set the parameters of JDBC Query Consumer as below to consume data from MariaDB:

  • SQL Query : select * from Orders where OrderID > ${OFFSET} order by OrderID;
  • Initial Offset : 0
  • Offset Column : OrderID
  • Transaction isolation : Read committed

it works for the initial query, however, when I insert new data into table Orders, the pipeline doesn't detect that there are new data, and the record count doesn't change after initial query is done.

I try to use the same parameters value for JDBC Multitable Consumer, and the record count does change after I insert new data into table Orders.

Does anyone know what is the reason cause the difference? How do I make JDBC Query Consumer detect new data?

edit retag flag offensive close merge delete


metadaddy gravatar imagemetadaddy ( 2018-03-13 12:56:05 -0600 )edit

The debug log repeat the message "Saving offset {$com.streamsets.datacollector.pollsource.offset$=11077} for pipeline xxxxx" until I restart the pipeline, the actual offset should be 11078. I already set transaction isolation level as READ COMMITTED, do you have any other suggestion? thanks!

Ida gravatar imageIda ( 2018-03-13 22:14:27 -0600 )edit