How to avoid HTTP client origin fetching same data repeatedly?

asked 2019-02-24 22:54:06 -0500

Maithri gravatar image

updated 2019-02-25 22:54:01 -0500

I am using HTTP client origin to fetch the SDC log data and store the required details in a table using JDBC Producer destination.

The pipeline looks like this : image description

Configuration looks like this:

image description

image description

image description

image description

 url: http://localhost:18630/rest/v1/pipeline/checkm2b5978dc-c6e1-4202-adf3-0c4e1281af0a/history/

I tried pagination by page number,none etc. All worked in the same way. The pipeline fetches data successfully,but if the same pipeline is ran and there are no new data it fetches same old data repeatedly.In this case what kind of pagination should be applied so that the http request pulls only new data and does not pull old data if there are no new logs.

I am filtering contents of the metrics such as timeOfLastReceivedRecord,stageStarttime,stageOutputrecordCount...status,etc

edit retag flag offensive close merge delete

Comments

1

Please paste the full configuration. Right now I am not seeing the full Resource URL and pagination configs in the screenshot.

jeff gravatar imagejeff ( 2019-02-25 12:17:08 -0500 )edit

Also, are you restarting the pipeline, or leaving it running?

metadaddy gravatar imagemetadaddy ( 2019-02-25 12:26:00 -0500 )edit

i have used batch mode.This makes pipeline fech data and goes to finished state.

Maithri gravatar imageMaithri ( 2019-02-25 22:34:11 -0500 )edit

I have pasted full config .Thanks for the response

Maithri gravatar imageMaithri ( 2019-02-25 22:34:46 -0500 )edit

@jeff@metadaddy I have added full configuration in the question

Maithri gravatar imageMaithri ( 2019-02-26 22:34:22 -0500 )edit