About the next query time in JDBC query origin

asked 2018-10-19 00:34:54 -0600

Dean Han
71 ●7 ●10 ●13

Hi, please conside this situation. A piplelien using JDBC query origin, and the query interval is set to 1 hour. Once fetched some data, let's say it will take 40 mins to process all data in this pipeline. After done, when will the next query will happen? For example: At 12:00, I started the pipepile, and origin query the first batch data and process, it spent 40 mins. The next query time wil lbe 13:00 or 13:40? Thanks.

edit retag flag offensive close merge delete

add a comment

1 Answer

Sort by » oldest newest most voted

answered 2018-10-19 16:03:45 -0600

iamontheinet

1579 ●4 ●9 ●31

Hi Dean!

In your example, the next query execution time will be 13:40. Basically it will take into account the time it's taken the current/last query to process data.

You can checkout the source code/logic here -- https://github.com/streamsets/datacol...

Cheers, Dash

edit flag offensive delete link

Comments

Thanks Dash, is there any way to make the query always be happened after certain time, like 1 hours, under the condition of pipeline would spend more or less 40 mins to process.

Dean Han ( 2018-10-23 08:26:31 -0600 )edit

Not that I know of just by looking at the code. But you can try setting the query execution time to maybe 20mins?

iamontheinet ( 2018-10-24 11:39:06 -0600 )edit

add a comment

Question Tools

1 follower

Stats

Asked: 2018-10-19 00:34:54 -0600

Seen: 107 times

Last updated: Oct 19 '18

About the next query time in JDBC query origin edit

1 Answer

Comments

Question Tools

Stats

Related questions

About the next query time in JDBC query origin