Stop pipeline on RUN_ERROR does not retry

asked 2019-10-21 04:27:34 -0600

LaurentS gravatar image

Hello,

I am using the docker version of datacollector, datacollector:3.12.0 I tried also with version 10.

I have a simple pipeline that read from salesforce platform event, and push to kafka.

I want my pipeline to: - Retry every failed event on error (reading from salesforce, pushing to kafka). - Failing after 10 failed event

General configuration of my pipeline:

  • shouldRetry: True
  • retryAttempts: 20

Stage configuraiton:

  • onError: Stop Pipeline

Expectation:

  • Pipeline stop on RUNNING_ERROR, then retry after 15 sec.

What happen:

  • On running error (any stage), the pipeline stop correctly (RUNNING_ERROR, then RUN_ERROR) but never go to the RETRY state afterwards.

I think this is due to: Pipeline.shouldStopOnStageError() returning true int:

        ./datacollector/execution/runner/common/ProductionPipeline.java



        if (shouldRetry && !pipeline.shouldStopOnStageError() &&
              !isExecutingInSlave && isRecoverable
              && !wasStopped()) {
          stateChanged(PipelineStatus.RETRY,
        runningErrorMsg, null);    }

I am probably not configuring my pipeline correctly and this can be fixed easily. Roughly, I only want my pipeline to not save the offset on error, but just retry.

According to this: https://ask.streamsets.com/question/4... Stopping the pipeline on error should have been the proper way, but it seems it does not work.

edit retag flag offensive close merge delete