Ask Your Question
1

Error importing a pipeline using Python requests api.

asked 2019-09-16 06:10:34 -0500

hcbharath gravatar image

updated 2019-09-16 06:32:19 -0500

I'm trying to import a pipeline to StreamSets Data Collector. Since the Control hub is enabled I'm getting a cookie token from Control hub for user Authentication.

cookie = token got from control hub.

target_url = "https://Data_Collector_host_name:port_no/rest/v1/pipelines/import"

files = {'file': open("pipelines.zip", 'rb')}

import_response = requests.post(target_url, cookies=cookie, files=files, verify=False)

But it gives an error saying 405: Method not allowed. <title>Error 405 Method Not Allowed</title> </head> <body>

HTTP ERROR 405

Problem accessing /rest/v1/pipelines/import. Reason:

 Method Not Allowed



edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
1

answered 2019-09-19 23:34:53 -0500

metadaddy gravatar image

I'm not sure why it returns 405 specifically, but I do know that you can't just supply the SCH cookie in the SDC API call. You have to use the cookie value in the X-SS-User-Auth-Token header, like this:

cookie = cookies from Control Hub

files = {'file': open("pipelines.zip", 'rb')}

headers = {
    "X-Requested-By":"SDC",
    "X-SS-REST-CALL":"true",
    "X-SS-User-Auth-Token":cookies['SS-SSO-LOGIN']
}

import_response = requests.post(target_url, files=files, headers=headers, verify=False)
edit flag offensive delete link more

Comments

Hi, Thanks a lot, this works. I was facing this issue for quite some time. And now finally this works. The issue was I was not passing the authentication token in the header. Weirdly this works for GET requests. Anyways the issue I was having is resolved. Thanks again.

hcbharath gravatar imagehcbharath ( 2019-09-19 23:57:24 -0500 )edit

And just another question, how long will this authentication token be valid?

hcbharath gravatar imagehcbharath ( 2019-09-20 00:09:15 -0500 )edit

86400 seconds i.e. 24 hrs

ak47 gravatar imageak47 ( 2019-09-20 00:38:23 -0500 )edit

Thank you.

hcbharath gravatar imagehcbharath ( 2019-09-20 00:39:21 -0500 )edit
1

answered 2019-09-16 17:22:03 -0500

iamontheinet gravatar image

updated 2019-09-16 17:23:00 -0500

Hi!

I'd recommend using StreamSets SDK for Python instead for more concise and readable code. For instance:

Importing into SDC:

from streamsets.sdk import DataCollector
data_collector = DataCollector('http://$SDC_HOST:$SDC_PORT')

with open('/path/to/your_pipeline.json', 'r') as input_file:
      pipeline_json = json.load(input_file)

pipeline = data_collector.import_pipeline(pipeline=pipeline_json)

Importing into SCH:

from streamsets.sdk import ControlHub
control_hub = ControlHub('https://$SCH_HOST', username="SCH_USER", password="SCH_PWD")

with open('/path/to/your_pipeline.json', 'r') as input_file:
       pipeline_json = json.load(input_file)

pipeline = control_hub.import_pipeline(pipeline=pipeline_json, name='Imported via SDK')

Cheers, Dash

edit flag offensive delete link more

Comments

Hi, This can be done, but why does the POST request give "405 METHOD NOT ALLOWED" error?

hcbharath gravatar imagehcbharath ( 2019-09-19 22:46:48 -0500 )edit

My guess is that it really should be 401, since you're not passing the token in the correct way. See my answer.

metadaddy gravatar imagemetadaddy ( 2019-09-19 23:35:51 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-09-16 06:10:34 -0500

Seen: 174 times

Last updated: Sep 19