Ask Your Question

Process and save multiple files in same folder

asked 2018-05-11 01:49:17 -0500

Kailegh gravatar image

updated 2018-05-11 10:28:53 -0500

jeff gravatar image

I want to process all txt in a folder, i can write origin like this and they get read:


The problem comes when I want to save them. If I want to process them I have to read in a delimited file format, but doing it like that I am not able to save them with the same origin name but in other folder ( or at least I do not know how). The only way I have been able to do that is reading them like whole files, but by doing that I process them in the in the right way, for example I cannot put them in Hbase or Hive tables.

Do you know what could I do to solve this?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-05-11 10:30:31 -0500

jeff gravatar image

If I understand the question correctly, you want to process the data from the CSV files, then move the files to a new directory when finished? If so, use the Post Processing Actions configuration for the origin, specifically the Archive action. See the docs for more details.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-05-11 01:49:17 -0500

Seen: 380 times

Last updated: May 11 '18