Ask Your Question

Custom script run at service stop?

asked 2018-07-11 11:15:35 -0500

BenKentzer gravatar image

updated 2018-07-11 12:13:06 -0500

jeff gravatar image

We're running Streamsets from within Cloudera's CDH (version 5.9.0). Currently when someone restarts the services on the cluster, it means that the pipelines all end up in the "START_ERROR" state; I assume this is down to us trying to get it to do too much at once.

As part of our batch processing, we have a script which cycles round the pipelines on a host and stops them gracefully. This then prevents the error, and we can use our "start" scripts to bring the pipelines up one by one.

What I'd really like is to be able to run our "stop" custom script immediately before the services are stopped which would send a stop request to each pipeline.

Is this possible? I've asked the same question on the Cloudera forum, as we always stop and start the services from there - rather than direct against the service itself.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-07-11 12:19:10 -0500

jeff gravatar image

You can certainly dig into the underlying exceptions that lead to the particular START_ERROR states and work to resolve them. It's certainly possible that something can be done to alleviate that, depending on what the root causes are.

I'm not aware of any particular mechanism to execute customs scripts based on the SDC lifecycle. There are pipeline lifecycle events, which don't seem relevant to what you're trying to do.

Does Cloudera Manager provide a mechanism to execute a custom shell script upon stopping the service, which is configured on the CM side of things? I'm guessing the answer is "no", otherwise you wouldn't be asking this question, but just thought I'd double check.

In the meantime, I suggest watching SDC-9256, which would likely help the situation you're describing once implemented.

edit flag offensive delete link more


SDC-9256 (and SDC-7153 as linked in the comments on there) look to help quite a bit - thank you. I'll investigate the logs more deeply. If we can get version then SDC-7153 will solve the problem. Unfortunately, the current CDH we have only provides version

BenKentzer gravatar imageBenKentzer ( 2018-07-12 00:58:57 -0500 )edit

On the custom script front, the answer so far on the CDH forums is "sorry no, ask Streamsets".

BenKentzer gravatar imageBenKentzer ( 2018-07-12 01:03:34 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-07-11 11:15:35 -0500

Seen: 195 times

Last updated: Jul 11 '18