Ask Your Question
1

Make sure StreamSets process file when the file's .ready file is generated?

asked 2018-01-18 00:04:15 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I have a source (SFTP) which can be populated by different type of files like csv, txt, gz etc. But I want StreamSets to process the files only when they are ready. So when any file is completed, the client (of mine) generates a file with the same name but with .ready extension. So if the file with 'abc.txt' is complete then they generate 'abc.txt.ready'. I want StreamSets to pick abc.txt only when abc.txt.ready is created. How would I do this in StreamSets?

PS: There are many clients and all of them generate .ready files so I cannot ask the client to change the logic, like if the file is ready then rename it to some other name as is suggested Here

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2018-01-19 11:55:06 -0500

jeff gravatar image

This is currently not supported, but it makes sense. Please comment on SDC-6827 with your requirements and we will add it to the roadmap as time and priorities permit.

edit flag offensive delete link more

Comments

Are there any workaround for this, as this issue is major for us and can't wait till SDC adds them in their future releases?

Avishek Gurung gravatar imageAvishek Gurung ( 2018-01-19 22:16:39 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-01-18 00:04:15 -0500

Seen: 143 times

Last updated: Jan 19