Ask Your Question

Is it possible to process an output file from another pipeline?

asked 2020-10-29 02:44:37 -0600

carlo.newbie gravatar image

updated 2020-10-29 22:47:28 -0600

The first pipeline will store the files in S3 (origin is S3) depending on the the column value (date) so it will be partioned by Year, month, day. Then the second pipeline (origin is S3) will perform the ETL and store it in the DW.

I have managed to do the first pipeline, only problem I got is that the output of the pipeline changed the filename and the second pipeline cannot read or ingest the data to the 2nd pipeline

Update: I was able to read the output file of the another pipeline, only thing needed is to check the common prefix and prefix pattern of the S3 origin of the 2nd pipeline.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-10-30 05:02:20 -0600

Gaja gravatar image

There is a option in Streamsets called "topologies" which allows you to interlink two pipelines together and work as one

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-10-29 02:44:37 -0600

Seen: 607 times

Last updated: Oct 30 '20