Pipeline with Spark executor is not working in standalone mode

asked 2020-08-24 10:31:38 -0600

yg gravatar image

Hi All,

We are using StreamSets Data Collector 3.10.0 and trying to use spark executor . Pipeline contains directory-->spark executor, we configured spark executor with below arguments but when i run this pipeline this is bringing down one node with maxingout the memory and we configured this pipeline as standalone mode. Configuration: Driver Memory: 1G Executor Memory: 1G Minimum Number of Worker Nodes :2 Maximum Number of Worker Nodes :2

After triggering pipeline starts and bringing down the node where sdc is installed by using the memory more than 30 gb.

could you please help me with thoughts/suggestions on the same. Thanks in Advance

edit retag flag offensive close merge delete