JDBC Consumer Origin v JDBC Lookup Differences Technically & Thread Starvation

asked 2020-01-14 13:34:48 -0500

leebuckman gravatar image

Hello All,

I am fairly new to streamsets and have only been using it a while. I have read the documentation a lot but have a technical question with regards to what streamsets is actually doing in the background.

I have a JDBC consumer origin reading a table from Snowflake Data Warehouse returning 1 row. Next stage is a JDBC Lookup (not using the output of the consumer just to be clear) which then acts as just a data reader from SQL Server. Long term it will pass the SQL statement dynamically from the consumer but trying to prove something out right now. The SQL query in the JDBC Lookup is reading a subset of data from a table on SQL Server with 4 million rows and about 45 columns. Last stage is just a trash can for testing .........

My question is this - Technically, does a JDBC lookup read data differently from a JDBC Consumer origin. Reason I ask is that if I run the same SQL statement from the JDBC Lookup in a Consumer Origin and go straight to trash it works and starts reading rows no problem. With the original set up described I end up getting thread starvation and it doesn't appear to read data even after a number of minutes .................

What is going on here ? How can I resolve this ? I have now spent 2 days looking at this and cannot come up with an explanation.

Regards

Lee

edit retag flag offensive close merge delete

Comments

What happens with just the JDBC Consumer reading data from Snowflake, and no JDBC Lookup?

metadaddy gravatar imagemetadaddy ( 2020-01-14 13:53:16 -0500 )edit

If you enable DEBUG log level (see https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Administration/Administration_title.html#task_lkv_g2f_wy) can you see the SQL for the lookup query? Does it look correct?

metadaddy gravatar imagemetadaddy ( 2020-01-14 13:53:43 -0500 )edit

Firstly, thanks for responding. JDBC Consumer from Snowflake with no JDBC lookup is fine. Will check out the debugging and report back.

leebuckman gravatar imageleebuckman ( 2020-01-14 14:00:33 -0500 )edit

Further to your question. yes, I can see the SQL for the lookup query and it looks fine. I can copy it into PL/SQL Developer from the log file and it runs fine.

leebuckman gravatar imageleebuckman ( 2020-01-14 15:10:59 -0500 )edit