StreamSets with Kafka Unable to Update Kudu with milliseconds precession

asked 2020-04-01 10:14:11 -0600

sumanG gravatar image

updated 2020-04-01 17:56:50 -0600

metadaddy gravatar image

We are using streamSets Opensource edition 3.1.6. We noticed that StreamSets with KAFAK Unable to Update KUDU with milliseconds precession. Our goal is to update the current table by compaing kafkatimestamp , If Kafka_tmestamp of the icoming topi is greater than the existing record in the table then we will update non primary key columns in the table. StreamSets failed to update the current table when kafka sent two records within a millisecond precesion . Please see the example below . Expecting Current Table to be in latest status . Hisotry Table has status in the order of events received A,B,C,D where as current table is still displaying Status as C

Hisotry Table

Status kafka_timestamp streamsets_recieved_ts D 2020-03-29 14:44:29.958000000 2020-03-29 14:44:30.565000000 C 2020-03-29 14:44:29.096000000 2020-03-29 14:44:30.695000000 B 2020-03-28 21:26:38.464000000 2020-03-28 21:26:44.092000000 A 2020-03-28 21:26:38.436000000 2020-03-28 21:26:42.877000000

Current Table

Status kafka_timestamp streamsets_recieved_ts

C 2020-03-29 14:44:29.096000000 2020-03-29 14:44:30.695000000

edit retag flag offensive close merge delete