Ask Your Question

How to post a JSON pipeline file using curl in Dockerfile

asked 2019-12-30 04:24:39 -0500

Prometheous2019 gravatar image

updated 2020-01-15 16:14:17 -0500

metadaddy gravatar image

I have created a base StreamSets ([]) Dockerfile and pushed it into ECR.

Now I'm creating another child Dockerfile which will refer to this base ECR image and then post the pipeline.json to upload the pipeline. It sounds easy, but I still can't figure it out how to post the pipeline json in dockerfile.

# COPY --chown=sdc:sdc pipelines/pipeline.json /data/ 
COPY --chown=sdc:sdc pipelines/pipeline.json /tmp/

This throws an error, but without the copy command if I simply build a child docker image and upload this pipeline.json in GUI it works fine.

So can anyone suggest me how to post the pipeline using REST API via docker.

curl --noproxy localhost:18631 --insecure --silent --request POST http://localhost:18631/rest/v1/pipeline/test/import" --data-binary @/tmp/pipeline.json" --header "Content-Type: application/json" --header  "X-Requested-By:sdc" --write-out "%{http_code}" --output /tmp/sdc-pipeline-response.json
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2019-12-31 06:29:50 -0500

Prometheous2019 gravatar image

updated 2020-01-15 16:45:25 -0500

never mind, fixed it.

edit: instead of using curl I used

> streamsets cli -U
> http://localhost:<port> -u <username> 
> -p <password> <rest of command to upload the pipeline>
edit flag offensive delete link more


Please explain the fix - this doesn't help anyone else with the same problem.

metadaddy gravatar imagemetadaddy ( 2020-01-15 16:11:45 -0500 )edit

Is the curl command in the question the solution?

metadaddy gravatar imagemetadaddy ( 2020-01-15 16:13:57 -0500 )edit

@metadaddy, edited my answer/fix. But still don't know, why curl didn't work ...

Prometheous2019 gravatar imagePrometheous2019 ( 2020-01-15 16:17:11 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-12-30 04:24:39 -0500

Seen: 162 times

Last updated: Jan 15