Ask Your Question
1

Calling an API where the access token expires in 3600 seconds

asked 2017-12-14 05:17:44 -0600

Ayush gravatar image

updated 2017-12-19 11:59:42 -0600

metadaddy gravatar image

My application obtains an access token, and the token is used to make calls to the API. But the token gets expired after 3600 seconds. Its a bearer token.

How can I schedule this job in StreamSets to ingest data in hdfs.

edit retag flag offensive close merge delete

Comments

Do you know what time does the tocken start to expire after 3600 seconds ?

Roh gravatar imageRoh ( 2017-12-18 23:49:20 -0600 )edit

1 Answer

Sort by » oldest newest most voted
1

answered 2017-12-19 11:59:12 -0600

metadaddy gravatar image

There are a couple of options here:

  • Use an HTTP Client Origin in polling mode to 'drive' the pipeline, obtaining the access token for each batch, then an HTTP Client Processor later in the pipeline to use the access token to call the API. Use this option if it's ok to obtain a token repeatedly, as it will get a new token after every polling interval.
  • Use the Dev Random Origin to drive the pipeline, and a script evaluator (Groovy/JavaScript/Jython) to obtain the access token. The script would save the token in the state variable and only get it when the pipeline started, or if it is more than, say, 3000 seconds old. Again, you would use the HTTP Client Processor to make the actual API call. Use this option if getting the access token carries a 'cost' and you only want to do it when necessary.
edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-12-14 05:17:44 -0600

Seen: 21 times

Last updated: Dec 19 '17