Ask Your Question
1

Pipeline load only 1000 records

asked 2018-06-14 10:18:56 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

In pipeline, using JDBC query consumer (source- SAP Hana) and Mapr-FS (destination). it run and load 1000 record to destination (as max batch size is set to 1000) , but then there no next run for other batchs.. and throw error after 2 hours (max connection time is set 120 minuts) "connection not available".

SELECT NAME,CUSTOMER_SK,GENDER,CUSTOMER_SINCE,DOB,MARITAL_SATUS from  CUSTOMER where CUSTOMER_SK > {OFFSET} order by CUSTOMER_SK

Version - StreamSets Data Collector 2.6.0.1

error : com.streamsets.pipeline.api.StageException: JDBC_77 - SQLTransientConnectionException attempting to execute query 'SELECT NAME,CUSTOMER_SK,GENDER,CUSTOMER_SINCE,DOB,MARITAL_SATUS from ENASOLA.CUSTOMER where CUSTOMER_SK > 0 order by CUSTOMER_SK'. Giving up after 1 errors as per stage configuration. First error: SQLState: null Error Code: 0 Message: HikariPool-4 - Connection is not available, request timed out after 7200000ms.
edit retag flag offensive close merge delete

Comments

It sounds like the SAP Hana JDBC driver is not behaving. In any case, 2.6.0.1 is a really old version - you should try with 3.3.0 - there have been many fixes over the past year.

metadaddy gravatar imagemetadaddy ( 2018-06-14 10:25:43 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-06-21 11:16:41 -0500

metadaddy gravatar image

It sounds like the JDBC driver is not properly closing the connection after Data Collector loads the data. A similar issue with the Snowflake JDBC driver was fixed recently (SDC-8653), so just upgrading Data Collector to the current version, 3.3.0, might fix this for you.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-06-14 10:18:56 -0500

Seen: 117 times

Last updated: Jun 21