Ask Your Question
2

Why does my pipeline take a very long time to stop, as of SDC 2.7?

asked 2017-08-31 15:23:41 -0600

jeff gravatar image

I noticed that as of SDC version 2.7, my pipelines take a very long time to stop, particularly when using certain origins (ex: Directory). What is causing this, and what is the solution?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
4

answered 2017-08-31 15:28:06 -0600

jeff gravatar image

updated 2017-08-31 15:57:03 -0600

LC gravatar image

SDC version 2.7 introduced a very important change, via SDC-6865. Specifically, when stopping a pipeline, we no longer interrupt the origin thread, which can potentially lead to loss of data.

With the SDC-6865 change, SDC will no longer forcibly interrupt the pipeline, but will rather wait for the current batch to finish all processing before stopping, in the interest of preventing data loss. You can still use the Force Stop operation if you want to stop the pipeline immediately, and aren't concerned about the loss of potential data (e.g. you know nothing is currently in-process, or don't care if messages are lost - which can depend on the type of origin).

If you have a pipeline that is taking a very long time to stop gracefully, check the source configuration carefully. There are options to control the maximum amount of time it waits before flushing a partial batch (e.g. Batch Wait Time (secs) for the Directory origin). Adjust this value downwards as desired.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-08-31 15:23:41 -0600

Seen: 1,623 times

Last updated: Aug 31 '17