Error importing a pipeline using Python requests api.

asked 2019-09-16 06:10:34 -0500

hcbharath
33 ●10 ●12 ●16

updated 2019-09-16 06:32:19 -0500

I'm trying to import a pipeline to StreamSets Data Collector. Since the Control hub is enabled I'm getting a cookie token from Control hub for user Authentication.

cookie = token got from control hub.

target_url = "https://Data_Collector_host_name:port_no/rest/v1/pipelines/import"

files = {'file': open("pipelines.zip", 'rb')}

import_response = requests.post(target_url, cookies=cookie, files=files, verify=False)

But it gives an error saying 405: Method not allowed. <title>Error 405 Method Not Allowed</title> </head> <body>

HTTP ERROR 405

Problem accessing /rest/v1/pipelines/import. Reason:

 Method Not Allowed

edit retag flag offensive close merge delete

add a comment

1

answered 2019-09-19 23:34:53 -0500

metadaddy

5464 ●26 ●41 ●81 https://about.me/patpa...

I'm not sure why it returns 405 specifically, but I do know that you can't just supply the SCH cookie in the SDC API call. You have to use the cookie value in the X-SS-User-Auth-Token header, like this:

cookie = cookies from Control Hub

files = {'file': open("pipelines.zip", 'rb')}

headers = {
    "X-Requested-By":"SDC",
    "X-SS-REST-CALL":"true",
    "X-SS-User-Auth-Token":cookies['SS-SSO-LOGIN']
}

import_response = requests.post(target_url, files=files, headers=headers, verify=False)

edit flag offensive delete link

Comments

Hi, Thanks a lot, this works. I was facing this issue for quite some time. And now finally this works. The issue was I was not passing the authentication token in the header. Weirdly this works for GET requests. Anyways the issue I was having is resolved. Thanks again.

hcbharath ( 2019-09-19 23:57:24 -0500 )edit

And just another question, how long will this authentication token be valid?

hcbharath ( 2019-09-20 00:09:15 -0500 )edit

86400 seconds i.e. 24 hrs

ak47 ( 2019-09-20 00:38:23 -0500 )edit

Thank you.

hcbharath ( 2019-09-20 00:39:21 -0500 )edit

instead of token, can we use, username and password to post? if so, then what would be the header? target url would be DC not SDH?

Prometheous2019 ( 2020-01-15 15:54:29 -0500 )edit

add a comment

1

answered 2019-09-16 17:22:03 -0500

iamontheinet

1966 ●4 ●11 ●33

updated 2019-09-16 17:23:00 -0500

Hi!

I'd recommend using StreamSets SDK for Python instead for more concise and readable code. For instance:

Importing into SDC:

from streamsets.sdk import DataCollector
data_collector = DataCollector('http://$SDC_HOST:$SDC_PORT')

with open('/path/to/your_pipeline.json', 'r') as input_file:
      pipeline_json = json.load(input_file)

pipeline = data_collector.import_pipeline(pipeline=pipeline_json)

Importing into SCH:

from streamsets.sdk import ControlHub
control_hub = ControlHub('https://$SCH_HOST', username="SCH_USER", password="SCH_PWD")

with open('/path/to/your_pipeline.json', 'r') as input_file:
       pipeline_json = json.load(input_file)

pipeline = control_hub.import_pipeline(pipeline=pipeline_json, name='Imported via SDK')

Cheers, Dash

edit flag offensive delete link

Comments

Hi, This can be done, but why does the POST request give "405 METHOD NOT ALLOWED" error?

hcbharath ( 2019-09-19 22:46:48 -0500 )edit

My guess is that it really should be 401, since you're not passing the token in the correct way. See my answer.

metadaddy ( 2019-09-19 23:35:51 -0500 )edit

can we pass " ssl verify off " for DataCollector part? if so, what is the argument?

Prometheous2019 ( 2020-01-15 15:26:15 -0500 )edit

@Prometheous2019 can you ask a new question? It's not good practice to ask and answer new questions in the comments.

metadaddy ( 2020-01-15 16:10:41 -0500 )edit

thanks. understood will do...

Prometheous2019 ( 2020-01-15 16:11:36 -0500 )edit

add a comment

Error importing a pipeline using Python requests api.

HTTP ERROR 405

2 Answers

Comments

Comments

Question Tools

Stats

Related questions

Error importing a pipeline using Python requests api. edit

HTTP ERROR 405

2 Answers

Comments

Comments

Question Tools

Stats

Related questions

Error importing a pipeline using Python requests api.