Ask Your Question
0

How to pull full and incremental data from HTTP Client Origin using API call?

asked 2020-07-08 01:02:30 -0500

anilkumardvs gravatar image

Hi , I have a requirement to pull the data from HTTP client through API. I am able to pull the data in the form of json and able to load to csv file. My API allows only 10 records per API call. I have 300+ records in the HTTP Client source. If I rerun the pipeline same data is getting pulled. Reset origin also doesn't pull entire data. Please help me how to pull full data and incremental data and how to pass parameters from external file

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2020-07-15 18:04:04 -0500

Hi,

There might be a pagination on the API that allows only 10 records. Did you try this to pull all the records using single call?

<api endpoint="" url="">/submission?limit=100000

Check the processing mode and also make it to streaming (it looks like that applies for your requirement)

Below documentation is helpful for me on this: https://streamsets.com/documentation/...

Best, Meher Bezawada

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-07-08 01:02:30 -0500

Seen: 43 times

Last updated: Jul 08