Ask Your Question

Calling an API where the access token expires in 3600 seconds

asked 2017-12-14 05:17:44 -0500

Ayush gravatar image

updated 2017-12-19 11:59:42 -0500

metadaddy gravatar image

My application obtains an access token, and the token is used to make calls to the API. But the token gets expired after 3600 seconds. Its a bearer token.

How can I schedule this job in StreamSets to ingest data in hdfs.

edit retag flag offensive close merge delete


Do you know what time does the tocken start to expire after 3600 seconds ?

Roh gravatar imageRoh ( 2017-12-18 23:49:20 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2017-12-19 11:59:12 -0500

metadaddy gravatar image

There are a couple of options here:

  • Use an HTTP Client Origin in polling mode to 'drive' the pipeline, obtaining the access token for each batch, then an HTTP Client Processor later in the pipeline to use the access token to call the API. Use this option if it's ok to obtain a token repeatedly, as it will get a new token after every polling interval.
  • Use the Dev Random Origin to drive the pipeline, and a script evaluator (Groovy/JavaScript/Jython) to obtain the access token. The script would save the token in the state variable and only get it when the pipeline started, or if it is more than, say, 3000 seconds old. Again, you would use the HTTP Client Processor to make the actual API call. Use this option if getting the access token carries a 'cost' and you only want to do it when necessary.
edit flag offensive delete link more
Login/Signup to Answer

Question Tools



Asked: 2017-12-14 05:17:44 -0500

Seen: 628 times

Last updated: Dec 19 '17