Ask Your Question

Workaround when HTTP Client Request Failed

asked 2017-10-19 05:34:58 -0600

Vivian Y gravatar image

updated 2017-10-19 14:18:09 -0600

metadaddy gravatar image

Aware that this is an issue that when HTTP Client failed , it will cause the whole pipeline to keep on retry.

Is there any workaround for this? When HTTP Client Return Error, we able to treat the files as error records and send to that particular files to another places instead of letting the files stay at the particular folder forever and let the pipeline keep retry.

My use case will be even there is one records failed, instead of keep on retrying the pipeline, we can move the error files to another folder and raise alert on this error records, and let the rest of the records to continue run.

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted

answered 2019-06-29 11:53:42 -0600

I'm seeing something similar. I have an installation of StreamSets Data Collector 3.9.0 on ubuntu. I've got an Amazon S3 source, a custom whole file processor, and an Amazon S3 destination. It generally works, but I frequently get this in the logs:

2019-06-29 16:50:01,261 [user:*admin] [pipeline:S3-to-S3 Avro Augmentation/S3toS3AvroAugmentation2234363b-fb5b-4560-bf27-f8e2207bdaa0] [runner:] [thread:ProductionPipelineRunnable-S3toS3AvroAugmentation2234363b-fb5b-4560-bf27-f8e2207bdaa0-S3-to-S3 Avro Augmentation] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.amazonaws.SdkClientException: Unable to execute HTTP request: The target server failed to respond

When this happens, the pipeline stops complete, waits 5 minutes or so ("Retrying to start Pipeline in 200 seconds"), and restarts automatically. So it does gradually work its way through that data, but these frequent 5-minute pauses really slow it down. Is this expected behavior? I would prefer that it keep running the pipeline and retry just that file later, or even that it indicate an error and not process that file at all.

edit flag offensive delete link more

answered 2017-10-19 13:46:19 -0600

jeff gravatar image

Are you talking about the HTTP Client Processor? That already has support for what you describe. Specifically, if any non success status (outside of 2xx) is seen in the response, it generates an error record. What specific stage are you talking about?

edit flag offensive delete link more


Which version does this function support? My version is and i does not see this in the the HTTP client processor. Is it in the latest version of release?

Vivian Y gravatar imageVivian Y ( 2017-10-19 22:07:42 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-10-19 05:34:58 -0600

Seen: 5,992 times

Last updated: Oct 19 '17