Ask Your Question
0

JDBC multitable query interval

asked 2018-07-10 22:49:43 -0500

supahcraig gravatar image

In the single table JDBC query consumer there is a setting for query interval. It's nice. I like it. In the multitable consumer there is no such setting and the result is that Streamsets pounds the database every second or so. I'm getting emails from our DBAs, so I need to figure out how to configure the multitable consumer to only go look for new rows every minute or so.

Surely there is an analog to query interval in the multitable consumer?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2018-07-11 12:07:01 -0500

jeff gravatar image

There is a Queries Per Second setting for the JDBC table consumer you can use to throttle the rate of queries. Because of the complexity of this origin (ex: if multithreaded partitioning is used, and given the fact that M threads can be selecting from N tables), there is no simple "query interval" analog. Instead, this setting can be thought of as a rate limit, and the origin will not issue more than this number of queries per second, regardless of how many tables/partitions end up being used at runtime.

edit flag offensive delete link more

Comments

It's probably because I'm new to the tool, but how it appears to be working in practice doesn't seem to match how it is named. I've set it to 0.1, or 1 query/10 seconds. What I see in the logs is that every 10 seconds, it runs a query against both of my tables. So that's 2 queries/10 sec.

supahcraig gravatar imagesupahcraig ( 2018-07-12 12:26:49 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-07-10 22:49:43 -0500

Seen: 36 times

Last updated: Jul 11