Stop pipeline run after n errors during file processing

asked 2021-05-10 14:37:07 -0500

Kunjutti gravatar image

updated 2021-05-10 14:39:01 -0500

I have a pipeline with 'Directory' as origin. csv files are uploaded to a directory and the records are written to database. There could be approximately 2 million records in a file. When all the records are incorrect (happens sometimes due to incorrect timestamp format given in all records), it goes ahead to process all the records and thus fill up the logs. My requirement is that the file processing should stop when there is an error in at least one record. How can I achieve the same?

edit retag flag offensive close merge delete