Can't read from Hive in incremental mode

asked 2019-05-10 02:07:09 -0500

song gravatar image

updated 2019-05-10 20:38:22 -0500

metadaddy gravatar image

The JDBC CONSUMER is working for read hive table data with incrememtal mode.

The code is shown below.

SELECT datetime, timestemp, col1, col2, col3 FROM database.hivetable WHERE timestemp > ${OFFSET} ORDER BY timestemp initial offset : 0 offset column : timestemp

If Preview is executed, there is no error message but data is not moved (to Trash destination) There is no error message, so I can not find the cause.

The code below reads the data well.

SELECT datetime, timestemp, col1, col2, col3 FROM database.hivetable

The timestemp column is double type(also testing bigint type) of hive which converts datetime to unix epoch timestemp. I do not know why the timestemp column is not well working with respect to offset. I want to know if there are related issues, or if there is a reason to guess.

edit retag flag offensive close merge delete


Try increasing the preview timeout - Hive may be running a MapReduce job when you specify the WHERE clause, which is very slow. Have you tried running it with the WHERE clause?

metadaddy gravatar imagemetadaddy ( 2019-05-10 20:41:42 -0500 )edit

you're right. Resolved the issue after setting it to 'x10'. Thank you.

song gravatar imagesong ( 2019-05-13 00:59:10 -0500 )edit