Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

SDC version 2.7 introduced a very important change, via SDC-6865. Specifically, when stopping a pipeline, we no longer interrupt the origin thread, which can potentially lead to loss of data. With the SDC-6865 change, SDC will no longer forcibly interrupt the pipeline, but will rather wait for the current batch to finish before stopping, in the interest of preventing data loss. You can still use the Force Stop operation if you want to stop the pipeline immediately, and aren't concerned about the loss of potential data (ex: you know nothing is currently in-process, or don't care if messages are lost, which can depend on the type of origin).

If you have a pipeline that is taking a very long time to stop gracefully, check the source configuration carefully. There are options to control the maximum amount of time it waits before flushing a partial batch (ex: _Batch Wait Time (secs)_ for the Directory origin). Adjust this value downwards as desired.

SDC version 2.7 introduced a very important change, via SDC-6865. Specifically, when stopping a pipeline, we no longer interrupt the origin thread, which can potentially lead to loss of data. With the SDC-6865 change, SDC will no longer forcibly interrupt the pipeline, but will rather wait for the current batch to finish before stopping, in the interest of preventing data loss. You can still use the Force Stop operation if you want to stop the pipeline immediately, and aren't concerned about the loss of potential data (ex: you know nothing is currently in-process, or don't care if messages are lost, which can depend on the type of origin).

If you have a pipeline that is taking a very long time to stop gracefully, check the source configuration carefully. There are options to control the maximum amount of time it waits before flushing a partial batch (ex: _Batch Batch Wait Time (secs)_ (secs) for the Directory origin). Adjust this value downwards as desired.

click to hide/show revision 3
No.3 Revision

SDC version 2.7 introduced a very important change, via SDC-6865. Specifically, when stopping a pipeline, we no longer interrupt the origin thread, which can potentially lead to loss of data.

With the SDC-6865 change, SDC will no longer forcibly interrupt the pipeline, but will rather wait for the current batch to finish all processing before stopping, in the interest of preventing data loss. You can still use the Force Stop operation if you want to stop the pipeline immediately, and aren't concerned about the loss of potential data (ex: (e.g. you know nothing is currently in-process, or don't care if messages are lost, lost - which can depend on the type of origin).

If you have a pipeline that is taking a very long time to stop gracefully, check the source configuration carefully. There are options to control the maximum amount of time it waits before flushing a partial batch (ex: (e.g. Batch Wait Time (secs) for the Directory origin). Adjust this value downwards as desired.