HTTP Client Polling Mode is extracting same data again and again

asked 2019-02-17 22:13:28 -0500

suman gravatar image

Hi All,

I am extracting data from REST API using HTTP Client source in Polling Mode it was expected to stop after reading last bit of data and poll again for new data after specified interval But,it is extracting same data again and again and getting duplicate records.

I have verified in batch mode and I get around 1K record. Polling mode should also stop after 1K record and query API once as there is no new data being added in API

Please see below my configurations.

Resouce URL - ${xxxxxURL}/${table_name} Mode - Polling Polling interval - 5000 Pagination Mode - Link in response field. Next page link - /links[1]/href stop condition - ${record:value('/links[1]/href') == NULL}

edit retag flag offensive close merge delete

Comments

Can you pl confirm if the Resource URL, Link in Response Field and Next Page Link Field are setup as specified in the docs https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Origins/HTTPClient.html#concept_wk4_bjz_5r?

iamontheinet gravatar imageiamontheinet ( 2019-02-19 15:16:24 -0500 )edit

yes it is configured as per above documentation

suman gravatar imagesuman ( 2019-02-20 02:35:02 -0500 )edit