Ask Your Question

How to setup a S3 source so that it ignores the historical files

asked 2017-12-04 12:08:31 -0500

Alfredo Gomez gravatar image

Hello there,

I am trying to put in production some processes but I have found something that is making it complicated. How can I setup the S3 sources so that they ignore historical files on their locations but start feeding only for new files? or since certain modified file date?


edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2017-12-04 12:35:07 -0500

metadaddy gravatar image

updated 2017-12-04 12:36:40 -0500

By default they're read in a lexicographic ascending sort order but can also be done via modified time. Lexicographic sorting is useful when your files conform to a pattern such as 2017-12-01-xyz, 2017-12-02-xyz, and so on.

Please choose one of the two options in the following setting:

image description

See the documentation on read order for more details.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-12-04 12:08:31 -0500

Seen: 217 times

Last updated: Dec 04 '17