Ask Your Question
1

How to post a JSON pipeline file using curl in Dockerfile

asked 2019-12-30 04:24:39 -0600

Prometheous2019 gravatar image

updated 2020-01-15 16:14:17 -0600

metadaddy gravatar image

I have created a base StreamSets ([https://github.com/streamsets/datacol...]) Dockerfile and pushed it into ECR.

Now I'm creating another child Dockerfile which will refer to this base ECR image and then post the pipeline.json to upload the pipeline. It sounds easy, but I still can't figure it out how to post the pipeline json in dockerfile.

FROM XXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/basestreamset:latest
# COPY --chown=sdc:sdc pipelines/pipeline.json /data/ 
COPY --chown=sdc:sdc pipelines/pipeline.json /tmp/

This throws an error, but without the copy command if I simply build a child docker image and upload this pipeline.json in GUI it works fine.

So can anyone suggest me how to post the pipeline using REST API via docker.

curl --noproxy localhost:18631 --insecure --silent --request POST http://localhost:18631/rest/v1/pipeline/test/import" --data-binary @/tmp/pipeline.json" --header "Content-Type: application/json" --header  "X-Requested-By:sdc" --write-out "%{http_code}" --output /tmp/sdc-pipeline-response.json
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-12-31 06:29:50 -0600

Prometheous2019 gravatar image

updated 2020-01-15 16:45:25 -0600

never mind, fixed it.

edit: instead of using curl I used

> streamsets cli -U
> http://localhost:<port> -u <username> 
> -p <password> <rest of command to upload the pipeline>
edit flag offensive delete link more

Comments

Please explain the fix - this doesn't help anyone else with the same problem.

metadaddy gravatar imagemetadaddy ( 2020-01-15 16:11:45 -0600 )edit

Is the curl command in the question the solution?

metadaddy gravatar imagemetadaddy ( 2020-01-15 16:13:57 -0600 )edit

@metadaddy, edited my answer/fix. But still don't know, why curl didn't work ...

Prometheous2019 gravatar imagePrometheous2019 ( 2020-01-15 16:17:11 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-12-30 04:24:39 -0600

Seen: 36 times

Last updated: Jan 15