Pipelines going in "Hang State" after a period of running

asked 2018-10-15 08:47:48 -0500

MarcoM gravatar image

Hello everyone,

I am experiencing a sort of slowness, or worst a hang/stuck status, of the pipelines after they are running for a long time. I have not exactly the clue why is this happening. Is related to the java memory size? Is related to other things that i'm not considering? The strange thing is that is not an error, simply the pipeline doesn't process data that comes in until i stopped and restart, or in case of Kafka origins also i have to change the consumer group. I hope someone can help me.

Thank you.

Greetings, Marco

edit retag flag offensive close merge delete

Comments

1

Hi Marco, I have recently started using SDC and have seen similar behaviour with the Kafka Consumer. Using the Kafka Multitopic Consumer seems to work a lot better and you can have multiple threads running, each reading from different topic partitions. Hope this is of some use if tardy. Tim

tmotters gravatar imagetmotters ( 2019-12-15 09:18:53 -0500 )edit