Ask Your Question

How to configure HTTP Client (destination) OAuth2 headers for Control Hub

asked 2018-08-01 11:41:56 -0500

sheraz Khan gravatar image

updated 2018-08-01 14:30:45 -0500

metadaddy gravatar image

The github curl commands works well to get access token from SCH, however, if I want to implement the same on SDC using oAuth2 option using following curl command, I'm unable to define headers and cookie location on oAuth2 configuration tab. It seems HTTP headers doesn't work for oAuth2 link. The pipeline starts with oAuth2 authorization link and receive an error as <title>"Error 415 Unsupported Media Type</title>, its picking up media type text/html by default, however as per curl command we should be able to define header at oAuth2 level as application/json.

curl command:

curl -X POST -d "{\"userName\":\"${USERNAME}\", \"password\": \"${PASSWORD}\"}" ${URL}/security/public-rest/v1/authentication/login --header "Content-Type:application/json" --header "X-Requested-By:DPMClient" -c /tmp/cookie.txt
edit retag flag offensive close merge delete


Just to clarify - are you trying to use OAuth2 to authenticate to Control Hub?

metadaddy gravatar imagemetadaddy ( 2018-08-01 13:27:53 -0500 )edit

yes, I wanted to kick-off one pipeline to another using HTTP Client and I'm able to do it by using local SDC instance however, when i tried the same pipelines under SCH instance I need authorization token, here I'm getting issues to configure oAuth2 headers as there is no place holder

sheraz Khan gravatar imagesheraz Khan ( 2018-08-01 13:54:10 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2018-08-01 14:30:10 -0500

metadaddy gravatar image

updated 2018-09-06 15:58:41 -0500

StreamSets Control Hub does not currently support OAuth 2 as an authentication mechanism. You will need to use one HTTP Client to authenticate, then another to actually send a request using the returned token, or do this in a script evaluator.

Note - the token comes back from the authentication request to ${URL}/security/public-rest/v1/authentication/login as the SS-SSO-LOGIN cookie - you will need to extract it from the Set-Cookie HTTP header. You will see this in the record as a header attribute.

If you go to Help/RESTful API in the Data Collector web UI and scroll to the bottom, you'll see how to do this from the command line:

Sample script for calling SDC REST API when Control Hub is enabled:

# login to Control Hub security app
curl -X POST -d '{"userName":"DPMUserID", "password": "DPMUserPassword"}' --header "Content-Type:application/json" --header "X-Requested-By:SDC" -c cookie.txt

# generate auth token from security app
sessionToken=$(cat cookie.txt | grep SSO | rev | grep -o '^\S*' | rev)
echo "Generated session token : $sessionToken"

# Call SDC REST APIs using auth token
curl -X GET http://localhost:18630/rest/v1/pipelines --header "Content-Type:application/json" --header "X-Requested-By:SDC" --header "X-SS-REST-CALL:true" --header "X-SS-User-Auth-Token:$sessionToken" -i
edit flag offensive delete link more


Thanks for the advice, I tried the same with 2 HTTP clients. I used first HTTP client for authentication URL to get access token, however, was unable to read the response header. Could you please guide me how to read response header variable "X-SS-User-Auth-Token" value using HTTP Client component?

sheraz Khan gravatar imagesheraz Khan ( 2018-08-07 10:42:46 -0500 )edit

Added a note to my answer on obtaining the token from the authentication response.

metadaddy gravatar imagemetadaddy ( 2018-08-07 11:09:16 -0500 )edit

Can you elaborate on how to extract the SS-SSO-LOGIN cookie from 'Set-Cookie' header on the http processor? And this goes in the X-SS-User-Auth-Token header? Something like this? -> { "key": "X-SS-User-Auth-Token", "value": "record:dValue(Set-Cookie)" }

MT gravatar imageMT ( 2018-09-06 15:45:19 -0500 )edit

Yes - try it from the command line (I edited the details into my answer) and you should be able to see what's happening.

metadaddy gravatar imagemetadaddy ( 2018-09-06 15:59:17 -0500 )edit

Thanks. The curl makes sense. my question is more around syntax of doing it within a local sdc pipeline. In the HEADERS section of the HTTP processor, I'm trying to set below but my VALUE syntax is wrong "key": "X-SS-User-Auth-Token", "value": "$(str:substring(record:dValue(Set-Cookie),13,51))"

MT gravatar imageMT ( 2018-09-07 10:06:52 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-08-01 11:41:56 -0500

Seen: 963 times

Last updated: Sep 06 '18