Ask Your Question

How to read the files from different folders with same pipeline

asked 2020-08-24 07:13:51 -0500

Venkat gravatar image

I have multiple folders and need to fetch the files with same pipeline. can use parameter with multiple location in SDC/SCH ex: /source1/a.txt /source2/a.txt /source3/a.txt /source4/a.txt

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-09-16 02:56:30 -0500

satender gravatar image

I believe you can do it many ways and two of them are as below. 1) Parameterize the location - Create a pipeline and generate a job template from it. Using job template you can create & configure multiple jobs and each of those jobs will be having one file location.

2) Use Metadata Table- Create a metadata tale and configure all the locations along with other related information in it and iteratively use location details using the records attributes.

But, I will prefer the option#1 to you as it has more flexibility and reusability in terms of using the same job template in future for any additional location.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-08-24 07:13:51 -0500

Seen: 139 times

Last updated: Sep 16