Ask Your Question
1

How to import a pipeline that already exists using curl

asked 2020-02-07 13:28:00 -0500

subie gravatar image

Hi there, This relates to version control. We are trying to update an existing pipeline to a new version by basically overwriting the json with an import curl command from the SDC APIs.

The following works for creating the pipeline the first time:

curl -XPOST -u testuser:testpw -v -H 'Content-Type: application/json' -H 'X-Requested-By: sdc-user' -d "@/path/to/our/pipelineFile.json" https://localhost:1863/rest/v1/pipeline/dummy_id/import?autoGeneratePipelineId=true -o sdc_test_import.log

And then say it assigns the pipelineID: sdc-1234-456-345 ... and I want to overwrite the same pipeline with an updated version of pipelineFile.json...

This fails: curl -XPOST -u testuser:testpw -v -H 'Content-Type: application/json' -H 'X-Requested-By: sdc-user' -d "@/path/to/our/pipelineFile.json" https://localhost:1863/rest/v1/pipeline/sdc-1234-456-345/import -o sdc_test_import.log

This fails: curl -XPOST -u testuser:testpw -v -H 'Content-Type: application/json' -H 'X-Requested-By: sdc-user' -d "@/path/to/our/pipelineFile.json" https://localhost:1863/rest/v1/pipeline/sdc-1234-456-345 -o sdc_test_import.log

This fails: curl -XPUT -u testuser:testpw -v -H 'Content-Type: application/json' -H 'X-Requested-By: sdc-user' -d "@/path/to/our/pipelineFile.json" https://localhost:1863/rest/v1/pipeline/sdc-1234-456-345/import -o sdc_test_import.log

They all say the pipeline already exists:

"RemoteException" : { "message" : "com.streamsets.datacollector.store.PipelineStoreException: CONTAINER_0201 - Pipeline 'sdc-1234-456-345' already exists", "errorCode" : "CONTAINER_0201",

So how should I overwrite the same pipeline (by ID) with a new version of the json file??

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
2

answered 2020-02-07 14:36:16 -0500

subie gravatar image

updated 2020-02-07 14:37:40 -0500

metadaddy gravatar image

It turns out there is an optional overwrite parameter for the endpoint that I am using. (I didn't realize that).

Therefore, I solve my problem by using this: curl -XPOST -u testuser:testpw -H 'Content-Type: application/json' -H 'X-Requested-By: sdc' -d "@/path/to/our/pipelineFile.json" https://remotehost:1863/rest/v1/pipeline/sdc-1234-456-345/import?overwrite=true -o sdc_test_import.log

edit flag offensive delete link more
0

answered 2020-02-07 14:22:11 -0500

metadaddy gravatar image

The key here is to GET the pipeline via the /v1/pipeline/{pipelineId} URL (labelled 'Find Pipeline Configuration by name and revision' in the REST UI) rather than exporting it:

curl -u testuser:testpw -X GET http://localhost:18630/rest/v1/pipeline/sdc-1234-456-345 -H "X-Requested-By:sdc" -o /path/to/pipeline.json

Now you can modify it and POST it back to the same URL:

curl -v -u testuser:testpw -X POST http://localhost:18630/rest/v1/pipeline/sdc-1234-456-345 -H "X-Requested-By:sdc" -H 'Content-Type: application/json' -d @/tmp/pipeline.json -o sdc_test_import.log

Note that if you try to modify the file again, and POST it back, you will get an error:

{
  "RemoteException" : {
    "message" : "com.streamsets.datacollector.store.PipelineStoreException: CONTAINER_0205 - The pipeline 'qqqb1b5259d-ae3e-435e-af56-595d5c7b71ca' has been changed. Reload the page to view or edit the latest version of the pipeline.",
    "errorCode" : "CONTAINER_0205",
    "localizedMessage" : "CONTAINER_0205 - The pipeline 'qqqb1b5259d-ae3e-435e-af56-595d5c7b71ca' has been changed. Reload the page to view or edit the latest version of the pipeline.",
    "exception" : "PipelineStoreException",
    "javaClassName" : "com.streamsets.datacollector.store.PipelineStoreException",
    "antennaDoctorMessages" : [ ],
    "stackTrace" : "com.streamsets.datacollector.store.PipelineStoreException: CONTAINER_0205 - The pipeline 'qqqb1b5259d-ae3e-435e-af56-595d5c7b71ca' has been changed. Reload the page to view or edit the latest version of the pipeline....lots of stack trace..."
}

To modify it a second time, you will first need to get it a second time. This behavior is to avoid API clients modifying the pipeline concurrently, overwriting one of their updates.

edit flag offensive delete link more

Comments

1

This isn't really what I was asking, but I appreciate the effort in the response. My use case is SDC instance 1 >>>> version control >>>> SDC instance 2. I need to overwrite on the second leg. I found a solution and will post in a few minutes.

subie gravatar imagesubie ( 2020-02-07 14:30:35 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-02-07 13:28:00 -0500

Seen: 109 times

Last updated: Feb 07