Ask Your Question

Revision history [back]

How to avoid HTTP client origin fetching same data repeatedly?

I am using HTTP client origin to fetch the SDC log data and store the required details in a table using JDBC Producer destination.

The pipeline looks like this : image description

Configuration looks like this:

image description

I tried pagination by page number,none etc. All worked in the same way. The pipeline fetches data successfully,but if the same pipeline is ran and there are no new data it fetches same old data repeatedly.In this case what kind of pagination should be applied dso the the http request pulls only new data and does not pull old data if there are no new logs.

How to avoid HTTP client origin fetching same data repeatedly?

I am using HTTP client origin to fetch the SDC log data and store the required details in a table using JDBC Producer destination.

The pipeline looks like this : image description

Configuration looks like this:

image description

I tried pagination by page number,none etc. All worked in the same way. The pipeline fetches data successfully,but if the same pipeline is ran and there are no new data it fetches same old data repeatedly.In this case what kind of pagination should be applied dso the so that the http request pulls only new data and does not pull old data if there are no new logs.

How to avoid HTTP client origin fetching same data repeatedly?

I am using HTTP client origin to fetch the SDC log data and store the required details in a table using JDBC Producer destination.

The pipeline looks like this : image description

Configuration looks like this:

image descriptionimage description

image descriptionimage description(/upfiles/15511563774773577.png) I tried pagination by page number,none etc. All worked in the same way. The pipeline fetches data successfully,but if the same pipeline is ran and there are no new data it fetches same old data repeatedly.In this case what kind of pagination should be applied so that the http request pulls only new data and does not pull old data if there are no new logs.

How to avoid HTTP client origin fetching same data repeatedly?

I am using HTTP client origin to fetch the SDC log data and store the required details in a table using JDBC Producer destination.

The pipeline looks like this : image description

Configuration looks like this:

image description

image descriptionimage description(/upfiles/15511563774773577.png) (/upfiles/15511563774773577.png)

url: http://localhost:18630/rest/v1/pipeline/checkm2b5978dc-c6e1-4202-adf3-0c4e1281af0a/history/

I tried pagination by page number,none etc. All worked in the same way. The pipeline fetches data successfully,but if the same pipeline is ran and there are no new data it fetches same old data repeatedly.In this case what kind of pagination should be applied so that the http request pulls only new data and does not pull old data if there are no new logs.

How to avoid HTTP client origin fetching same data repeatedly?

I am using HTTP client origin to fetch the SDC log data and store the required details in a table using JDBC Producer destination.

The pipeline looks like this : image description

Configuration looks like this:

image description

image descriptionimage description(/upfiles/15511563774773577.png)

url: http://localhost:18630/rest/v1/pipeline/checkm2b5978dc-c6e1-4202-adf3-0c4e1281af0a/history/

I tried pagination by page number,none etc. All worked in the same way. The pipeline fetches data successfully,but if the same pipeline is ran and there are no new data it fetches same old data repeatedly.In this case what kind of pagination should be applied so that the http request pulls only new data and does not pull old data if there are no new logs.

I am filtering contents of the metrics such as timeOfLastReceivedRecord,stageStarttime,stageOutputrecordCount...status,etc

How to avoid HTTP client origin fetching same data repeatedly?

I am using HTTP client origin to fetch the SDC log data and store the required details in a table using JDBC Producer destination.

The pipeline looks like this : image description

Configuration looks like this:

image description

image descriptionimage description(/upfiles/15511563774773577.png)

 url: http://localhost:18630/rest/v1/pipeline/checkm2b5978dc-c6e1-4202-adf3-0c4e1281af0a/history/

http://localhost:18630/rest/v1/pipeline/checkm2b5978dc-c6e1-4202-adf3-0c4e1281af0a/history/

I tried pagination by page number,none etc. All worked in the same way. The pipeline fetches data successfully,but if the same pipeline is ran and there are no new data it fetches same old data repeatedly.In this case what kind of pagination should be applied so that the http request pulls only new data and does not pull old data if there are no new logs.

I am filtering contents of the metrics such as timeOfLastReceivedRecord,stageStarttime,stageOutputrecordCount...status,etc

How to avoid HTTP client origin fetching same data repeatedly?

I am using HTTP client origin to fetch the SDC log data and store the required details in a table using JDBC Producer destination.

The pipeline looks like this : image description

Configuration looks like this:

image description

image description

image description

image description(/upfiles/15511563774773577.png)

  url: http://localhost:18630/rest/v1/pipeline/checkm2b5978dc-c6e1-4202-adf3-0c4e1281af0a/history/

I tried pagination by page number,none etc. All worked in the same way. The pipeline fetches data successfully,but if the same pipeline is ran and there are no new data it fetches same old data repeatedly.In this case what kind of pagination should be applied so that the http request pulls only new data and does not pull old data if there are no new logs.

I am filtering contents of the metrics such as timeOfLastReceivedRecord,stageStarttime,stageOutputrecordCount...status,etc