Ask Your Question

How to avoid HTTP client origin fetching same data repeatedly?

asked 2019-02-24 22:54:06 -0500

Maithri gravatar image

updated 2019-02-25 22:54:01 -0500

I am using HTTP client origin to fetch the SDC log data and store the required details in a table using JDBC Producer destination.

The pipeline looks like this : image description

Configuration looks like this:

image description

image description

image description

image description

 url: http://localhost:18630/rest/v1/pipeline/checkm2b5978dc-c6e1-4202-adf3-0c4e1281af0a/history/

I tried pagination by page number,none etc. All worked in the same way. The pipeline fetches data successfully,but if the same pipeline is ran and there are no new data it fetches same old data repeatedly.In this case what kind of pagination should be applied so that the http request pulls only new data and does not pull old data if there are no new logs.

I am filtering contents of the metrics such as timeOfLastReceivedRecord,stageStarttime,stageOutputrecordCount...status,etc

edit retag flag offensive close merge delete



Please paste the full configuration. Right now I am not seeing the full Resource URL and pagination configs in the screenshot.

jeff gravatar imagejeff ( 2019-02-25 12:17:08 -0500 )edit

Also, are you restarting the pipeline, or leaving it running?

metadaddy gravatar imagemetadaddy ( 2019-02-25 12:26:00 -0500 )edit

i have used batch mode.This makes pipeline fech data and goes to finished state.

Maithri gravatar imageMaithri ( 2019-02-25 22:34:11 -0500 )edit

I have pasted full config .Thanks for the response

Maithri gravatar imageMaithri ( 2019-02-25 22:34:46 -0500 )edit

@jeff@metadaddy I have added full configuration in the question

Maithri gravatar imageMaithri ( 2019-02-26 22:34:22 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2019-12-02 07:45:05 -0500

Carol gravatar image

Any response on this?

I am experiencing same type of issue -- the Pagination Mode "By Page Number' fetches data, but seems to be fetching the same data over and over again. This is running on version 3.8.0 of the SDC. The HTTP Mode is configured as 'Batch' so it should eventually stop on its own, but it does not. I'm using the HTTP_CLIENT to use a REST API to read a Salesforce Marketing Cloud data extension.

edit flag offensive delete link more


My apologies.... i made an error typing my response to the ANSWER box... i meant to add a comment to the main thread.

Carol gravatar imageCarol ( 2019-12-02 07:49:07 -0500 )edit
Login/Signup to Answer

Question Tools



Asked: 2019-02-24 22:54:06 -0500

Seen: 210 times

Last updated: Dec 02 '19