Ask Your Question

Make sure StreamSets process file when the file's .ready file is generated?

asked 2018-01-18 00:04:15 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I have a source (SFTP) which can be populated by different type of files like csv, txt, gz etc. But I want StreamSets to process the files only when they are ready. So when any file is completed, the client (of mine) generates a file with the same name but with .ready extension. So if the file with 'abc.txt' is complete then they generate 'abc.txt.ready'. I want StreamSets to pick abc.txt only when abc.txt.ready is created. How would I do this in StreamSets?

PS: There are many clients and all of them generate .ready files so I cannot ask the client to change the logic, like if the file is ready then rename it to some other name as is suggested Here

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-01-19 11:55:06 -0500

jeff gravatar image

This is currently not supported, but it makes sense. Please comment on SDC-6827 with your requirements and we will add it to the roadmap as time and priorities permit.

edit flag offensive delete link more


Are there any workaround for this, as this issue is major for us and can't wait till SDC adds them in their future releases?

Avishek Gurung gravatar imageAvishek Gurung ( 2018-01-19 22:16:39 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-01-18 00:04:15 -0500

Seen: 507 times

Last updated: Jan 19 '18