JDBC Multi table consumer Override offset columns

asked 2017-10-31 09:23:23 -0500

353 ●22 ●25 ●33

updated 2017-10-31 14:05:37 -0500

I'm using JDBC Multitable consumer origin and I don't want to use the default primary key that Stream Sets uses because in my case primary key is not an incremental key.

My deltas identifying column will be TimeStamp, But when I override the offset column with that it is asking me in the long value. Let's suppose my max(Timestamp) in the hive table is "2017-10-31 12:30:00" How do I start from there with the long value? I'm confused a bit here I can make it to long as "20171031123000" but that's not picking the right deltas.

I then realized it will epoch time as "1509453023000" But that's not working as well. I'm bringing in the data from RedShift database. Please help me understand this. image description

image description

edit retag flag offensive close merge delete

add a comment

1

answered 2017-10-31 14:36:16 -0500

Roh

353 ●22 ●25 ●33

updated 2017-10-31 21:28:56 -0500

I forget to reset the origin. The time has to be in Milliseconds, not the epoch time

Explained here, used this function to convert the time into milliseconds ${time:dateTimeToMilliseconds(time:extractDateFromString('2017-10-31 12:20:30.915','yyyy-MM-dd HH:mm:ss.SSS'))}

edit flag offensive delete link

add a comment

1

answered 2017-10-31 13:05:26 -0500

jeff
3097 ●17 ●41 ●70

1509453023000 appears to be including millis, but you need second-level precision. Can you please try 1509453023?

edit flag offensive delete link

Comments

Second Level precision is the EPOCH time you mentioned right?

Roh ( 2017-10-31 13:22:30 -0500 )edit

Looks like when I use the override offset column and add the epoch time, it's not actually using it in the where clause I guess. I looked at the snapshot data and validate the counts to validate that it's not using in the where clause. I've added the screenshots of what I'm using in the question.

Roh ( 2017-10-31 14:00:06 -0500 )edit

add a comment

JDBC Multi table consumer Override offset columns

2 Answers

Comments

Question Tools

Stats

Related questions

JDBC Multi table consumer Override offset columns edit

2 Answers

Comments

Question Tools

Stats

Related questions

JDBC Multi table consumer Override offset columns