Ask Your Question

Import a pipeline as part of the container

asked 2020-02-26 23:11:53 -0600

Sakthi gravatar image

After going the SDC Docker Documentation, I was able to spin-up an SDC instance in and connect to the instance both via browser and exec -it.

I was able to import the pipeline via UI and using the curl manually. But I'm unable to import a pipeline part of the provisioning.

Here is my docker file,

FROM streamsets/datacollector:3.13.0

COPY pipeline.json pipelinestaginglocation/

EXPOSE 18630


CMD ["dc", "-exec" ; "/bin/bash", "-c", "curl -XPOST -u user:password -v -H 'Content-Type: application/json' -H 'X-Requested-By: My Import Process' -d "@/pipelinestaginglocation/pipeline.json" http://localhost:18630/rest/v1/pipeline/sample_id/import?autoGeneratePipelineId=true"]

edit retag flag offensive close merge delete


Do you see any errors?

metadaddy gravatar imagemetadaddy ( 2020-02-27 09:24:15 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2020-02-27 10:56:40 -0600

kirti gravatar image

updated 2020-02-27 11:17:24 -0600

Just a guess here, since we do not know the exact nature of problem/error: So as I understand, you are saying, that command ran for you when you ran it from inside container. Right?

Here are my comments: 1) Like @metadaddy asked, what is the error you are facing? Or simply the file is not imported.

2) I would say log into the container and see the exact location of the file is .e.g let's say it is at /pipelinestaginglocation/pipeline.json See if it exists. For some reason, if it did not, that might be a problem to look into.

3) Have you looked into StreamSets SDK For Python? That helps to do all the import/export, creation of a pipeline etc. easily rather than you dealing with CURL etc. If not, I would highly recommend you to take a look at it. This is a good starting point

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-02-26 23:11:53 -0600

Seen: 147 times

Last updated: Feb 27 '20