Ask Your Question

JDBC multitable consumer is only reading 3 tables out of 7 tables

asked 2017-10-11 08:12:02 -0600

Roh gravatar image

updated 2017-10-16 08:53:42 -0600

I set up the JDBC Multitable consumer and added the 7 tables by adding in the table config. But only three tables are getting populated and in the summary, i can see it is querying only three tables as thread-0, 1 and 2 but I set up the threads to 7. I confirmed in Redshift that all the tables have data. Not sure if I'm missing something here. Below are the screenshots of my configurations in JDBC MultiTable consumer. Happy to provide some more information if required.

image description

image description

image description

Table Config in the JDBC multi Table consumer: image description

Hive Metadata:

image description

edit retag flag offensive close merge delete


How have you set up the table config?

metadaddy gravatar imagemetadaddy ( 2017-10-11 18:05:51 -0600 )edit

I added the table configs with the schema and table name by doing + in the config. I didn't set anything else in the config other than the schema and table name because I understand that it will look for the primary key in the table by default.

Roh gravatar imageRoh ( 2017-10-16 08:46:37 -0600 )edit

@metdaddy Please see my added part in the question with the JDBC table config and hive table config. Let me know if you need any configurations to help me.

Roh gravatar imageRoh ( 2017-10-16 08:54:58 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2017-10-17 10:43:59 -0600

jeff gravatar image

Can you please try zeroing out the Query Interval parameter? There is an open issue with respect to that parameter and the multi table origin.

edit flag offensive delete link more


If I zero out the query interval, that means there will be a query running in the Redshift right after the batch is completed? If that's not the case then can I know how frequently does it query the source to bring the new data?

Roh gravatar imageRoh ( 2017-10-17 12:12:53 -0600 )edit

It will run immediately after the previous query finishes and generates a batch. However, reading more carefully, if you have 7 threads for 7 tables then this should not be the issue. Can you state exact SDC version, and also capture a thread dump?

jeff gravatar imagejeff ( 2017-10-17 14:49:09 -0600 )edit

Here is my stream sets version StreamSets Data Collector In the version that I'm using doesn't have the threads option at all.

Roh gravatar imageRoh ( 2017-10-21 09:31:34 -0600 )edit

Btw I tried with the zero query interval, still no progress, It is establishing only three threads/connections.

Roh gravatar imageRoh ( 2017-10-21 10:27:51 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-10-11 08:12:02 -0600

Seen: 140 times

Last updated: Oct 17 '17