JDBC Multitable- Pipeline stall

asked 2019-09-27 12:16:24 -0500

Torkia Boussada gravatar image

Hello, I'm using JDBC Multitable origin in a pipeline to load tables from Postgres database to Hive in a dev environment. I had to restart the pipeline 4 times before Streamsets finished and didn’t stall during the process (pipeline stay in running status, input/output records not changing after loading 3 125 696 records). I checked the log, there is no error.

Does anyone see this issue before?

We need to figure out this issue, as this could really impact the production transfer. If it happens in production where we think the pipeline will take days to finish we could be attempting to run it for weeks before it finishes. This would greatly impact our timelines to have to keep restarting the full load pipeline.

streamsets version: 3.7.1 Regards, Torkia

edit retag flag offensive close merge delete

Comments

When it is stuck, what are the stage metrics showing? It should show something like "Table Metrics for Thread - N" along with a state like WAITING_FOR_RATE_LIMIT_PERMIT, ACQUIRED_RATE_LIMIT_PERMIT, QUERYING_TABLE, etc.

jeff gravatar imagejeff ( 2019-09-27 13:08:31 -0500 )edit