Ask Your Question
0

Google Cloud Storage - Archive processed files

asked 2020-12-16 10:53:51 -0600

jacoleon gravatar image

Hello! I have a pipeline that reads files from a Google Cloud Storage Bucket, but in order to avoid the file to be continuosly being processed, I would like either to archive it to a different folder inside the bucket or at least rename.

The problem is that I don't see post-processing options on Google Cloud Storage origin. Does anyone have an idea how could I achieve that?

Thanks a million

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2020-12-16 19:00:15 -0600

iamontheinet gravatar image

Hi!

Use events framework to capture no-more-data event along with Shell executor to rename or move the file.

Also check out this short snackable video I created around using Shell executor to rename files.

Cheers, Dash

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-12-16 10:53:51 -0600

Seen: 19 times

Last updated: Dec 16 '20