Streamsets Pipeline gets stuck at phase STARTING

asked 2021-08-02 20:55:37 -0500

Adi gravatar image

Occasionally Streamsets Pipeline get stuck at Phase STARTING. We can not stop/force stop the pipeline through RESTAPI as well.

Please suggest a workaround to reset the status of the given Pipeline without restarting the SDC service

Version: StreamSets Data Collector 3.15.0

sdc.properties

Max number of concurrent REST calls allowed for the /rest/v1/admin/log endpoint

max.logtail.concurrent.requests=200

Max number of concurrent WebSocket calls allowed

max.webSockets.concurrent.requests=200

Pipeline runner pool

Default value is sufficient to run 22 pipelines. One pipeline requires 5 Threads and pipelines share

threads using thread pool. Approximate runner thread pool size = (Number of Running Pipelines) * 2.2.

Increasing this value will not increase parallelisation of individual pipelines.

runner.thread.pool.size=8000

Maximal number of source-less pipeline instances (=runners) that are allowed for a single multi-threaded

pipeline. The default is 50.

pipeline.max.runners.count=1000

edit retag flag offensive close merge delete

Comments

Can anyone help on this ??

Adi gravatar imageAdi ( 2021-08-11 23:40:41 -0500 )edit