Import a pipeline as part of the container

asked 2020-02-26 23:11:53 -0600

Sakthi
1 ●1 ●1 ●1

After going the SDC Docker Documentation, I was able to spin-up an SDC instance in and connect to the instance both via browser and exec -it.

I was able to import the pipeline via UI and using the curl manually. But I'm unable to import a pipeline part of the provisioning.

Here is my docker file,

FROM streamsets/datacollector:3.13.0

COPY pipeline.json pipelinestaginglocation/

EXPOSE 18630

ENTRYPOINT ["/docker-entrypoint.sh"]

CMD ["dc", "-exec" ; "/bin/bash", "-c", "curl -XPOST -u user:password -v -H 'Content-Type: application/json' -H 'X-Requested-By: My Import Process' -d "@/pipelinestaginglocation/pipeline.json" http://localhost:18630/rest/v1/pipeline/sample_id/import?autoGeneratePipelineId=true"]

edit retag flag offensive close merge delete

Comments

Do you see any errors?

metadaddy ( 2020-02-27 09:24:15 -0600 )edit

add a comment

answered 2020-02-27 10:56:40 -0600

kirti
306 ●1 ●3 ●9

updated 2020-02-27 11:17:24 -0600

Just a guess here, since we do not know the exact nature of problem/error: So as I understand, you are saying, that command ran for you when you ran it from inside container. Right?

Here are my comments: 1) Like @metadaddy asked, what is the error you are facing? Or simply the file is not imported.

2) I would say log into the container and see the exact location of the file is .e.g let's say it is at /pipelinestaginglocation/pipeline.json See if it exists. For some reason, if it did not, that might be a problem to look into.

3) Have you looked into StreamSets SDK For Python? That helps to do all the import/export, creation of a pipeline etc. easily rather than you dealing with CURL etc. If not, I would highly recommend you to take a look at it. This is a good starting point https://streamsets.com/documentation/...

edit flag offensive delete link

add a comment

Import a pipeline as part of the container

Comments

1 Answer

Question Tools

Stats

Related questions

Import a pipeline as part of the container edit

Comments

1 Answer

Question Tools

Stats

Related questions

Import a pipeline as part of the container