JDBC Query Consumer not fetching more than 1000 records

asked 2018-06-20 11:37:37 -0600

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

We are trying to fetch data from table in SAP HANA using JDBC Query Consumer in Streamsets, but we are only able to fetch 1000 records (with Maximum Pool Size in Advanced tab in JDBC Query Consumer set to 1) though there are more than 5 million records in source table.

Streamsets version 2.6.0.1

Configuration in Streamsets for max.ProductionBatchSize set to 1000.

But when we increase Maximum Pool Size in Advanced tab in JDBC Query Consumer to 5000, we are able to fetch 5 million records.

But Maximum Pool Size referring to number of database connections using which it will extract data from source table in external system.

Is it always recommended to increase the Maximum Pool Size irrespective of StreamSets version for this scenario, without which we can't fetch more than the records mentioned in max.ProductionBatchSize.

Else Is this already fixed in latest StreamSets version and are we able to fetch more than records mentioned in max.ProductionBatchSize with Maximum Pool Size in Advanced tab in JDBC Query Consumer set with 1 itself.

edit retag flag offensive close merge delete

Comments

What is the SQL query? Are you using full or incremental mode? Why are you using such an old version? The latest is 3.3.0!

metadaddy gravatar imagemetadaddy ( 2018-06-20 12:32:27 -0600 )edit

It is full mode. Tried using both "SELECT * FROM table" and "SELECT column_name1, column_name2,.. FROM table, but didn't help. Whether newer version has the fix that was expected? Or else issue remains until we either change max.ProductionBatchSize / Maximum Pool Size.

Muralikrishna Puvvala gravatar imageMuralikrishna Puvvala ( 2018-06-20 12:42:51 -0600 )edit

Please upgrade to 3.3.0 and see if it works. There have been many, many fixes since 2.6.0.1!

metadaddy gravatar imagemetadaddy ( 2018-06-20 12:46:13 -0600 )edit