Ask Your Question

JDBC Multi table consumer Override offset columns

asked 2017-10-31 09:23:23 -0600

Roh gravatar image

updated 2017-10-31 14:05:37 -0600

I'm using JDBC Multitable consumer origin and I don't want to use the default primary key that Stream Sets uses because in my case primary key is not an incremental key.

My deltas identifying column will be TimeStamp, But when I override the offset column with that it is asking me in the long value. Let's suppose my max(Timestamp) in the hive table is "2017-10-31 12:30:00" How do I start from there with the long value? I'm confused a bit here I can make it to long as "20171031123000" but that's not picking the right deltas.

I then realized it will epoch time as "1509453023000" But that's not working as well. I'm bringing in the data from RedShift database. Please help me understand this. image description

image description

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted

answered 2017-10-31 14:36:16 -0600

Roh gravatar image

updated 2017-10-31 21:28:56 -0600

I forget to reset the origin. The time has to be in Milliseconds, not the epoch time

Explained here, used this function to convert the time into milliseconds ${time:dateTimeToMilliseconds(time:extractDateFromString('2017-10-31 12:20:30.915','yyyy-MM-dd HH:mm:ss.SSS'))}

edit flag offensive delete link more

answered 2017-10-31 13:05:26 -0600

jeff gravatar image

1509453023000 appears to be including millis, but you need second-level precision. Can you please try 1509453023?

edit flag offensive delete link more


Second Level precision is the EPOCH time you mentioned right?

Roh gravatar imageRoh ( 2017-10-31 13:22:30 -0600 )edit

Looks like when I use the override offset column and add the epoch time, it's not actually using it in the where clause I guess. I looked at the snapshot data and validate the counts to validate that it's not using in the where clause. I've added the screenshots of what I'm using in the question.

Roh gravatar imageRoh ( 2017-10-31 14:00:06 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-10-31 09:23:23 -0600

Seen: 123 times

Last updated: Oct 31 '17