Ask Your Question
0

About the next query time in JDBC query origin

asked 2018-10-19 00:34:54 -0600

Dean Han gravatar image

Hi, please conside this situation. A piplelien using JDBC query origin, and the query interval is set to 1 hour. Once fetched some data, let's say it will take 40 mins to process all data in this pipeline. After done, when will the next query will happen? For example: At 12:00, I started the pipepile, and origin query the first batch data and process, it spent 40 mins. The next query time wil lbe 13:00 or 13:40? Thanks.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-10-19 16:03:45 -0600

iamontheinet gravatar image

Hi Dean!

In your example, the next query execution time will be 13:40. Basically it will take into account the time it's taken the current/last query to process data.

You can checkout the source code/logic here -- https://github.com/streamsets/datacol...

Cheers, Dash

edit flag offensive delete link more

Comments

Thanks Dash, is there any way to make the query always be happened after certain time, like 1 hours, under the condition of pipeline would spend more or less 40 mins to process.

Dean Han gravatar imageDean Han ( 2018-10-23 08:26:31 -0600 )edit

Not that I know of just by looking at the code. But you can try setting the query execution time to maybe 20mins?

iamontheinet gravatar imageiamontheinet ( 2018-10-24 11:39:06 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-10-19 00:34:54 -0600

Seen: 20 times

Last updated: Oct 19