Is there a way to do a reset origin for a pipeline automatically for every run?

asked 2019-05-24 09:33:35 -0600

adarshpanda gravatar image

I have a case wherein I need to load data from a file to the target table every day. The catch is the file might or might not be replaced every day. So, my requirement is even if a file is not replaced on a particular day, I should go ahead and reload the same data in the target table. Can anyone suggest me a way to run a pipeline with reset origin enabled by default for every single run?

edit retag flag offensive close merge delete

Comments

I too am interested in how to do this. It may be a good feature request. My use case is similar, I get the same files dumped on an SFTP site each week. Timestamp changes don't seem to mean that the file gets re-read, although that'd be my preferred record of origin for the Directory Origin.

Challenge Addict gravatar imageChallenge Addict ( 2019-09-08 23:05:59 -0600 )edit