Ask Your Question

How to get(offset) data from HTTP Client origin generating bearer token?

asked 2020-07-13 10:27:53 -0500

Manimars gravatar image

Hi , I have a requirement to pull the data from HTTP client through API. I'm able to pull the data in the form of json. My API allows only 10 records per API call. I have 1k+ records in the HTTP Client source. If I rerun the pipeline same data is getting pulled. Reset origin also doesn't pull entire data. I have tried pagination option in http client origin but it generates the bearer token multiple times..I have the main API call defined in http client processor but I m getting only 10 records per request. Please help me here... Below is my pipeline flow. Http client origin(bearer token)-> http client processor(fetching bearer token and getting the 10records per call) ---> other processors(applying transformations)---> destination(snowflake dB)

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-07-14 07:01:27 -0500

Jmbertoncelli gravatar image

Do you know how many record? because if yes then you might be able to issue a first http call to get authentication information then you could generate many records that would be used to propagate the token and other type of information needed to get the data per batch of 10 to the next http stage... for instance after the first http call then have a stage using jython to create has many record as needed then the next stage is an http call using token and other data created byt the jython stage to retrieve the data from the origin.

edit flag offensive delete link more


Thanks for the solution @Jmbertoncelli . Without adding any stage it did worked for me in http client processor by passing the parameters along with the url. As I know the record count, I have passed the larger number in case if there is an update from API.?skip=0&take=1500 worked for me in paramtrs

Manimars gravatar imageManimars ( 2020-07-14 22:45:52 -0500 )edit

Thanks for taking time and giving us an another idea to approach this scenario.

Manimars gravatar imageManimars ( 2020-07-14 22:48:27 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-07-12 01:06:21 -0500

Seen: 82 times

Last updated: Jul 14