Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

SFTP/FTP can't get all the files when there are many files at the same time

I get files from ftp, and then use expression evlauator to get the file name, and finally write the data to kafka. The detailed steps are as follows: 1. Uploads multiple files to the ftp server directory A at the same time. 2. To ensure the files are uploaded, after a few minutes, I move all of these files to directory B, and touch them. 3. StreamSets SFTP/FTP client reads files from directory B. 4. Expression Evlauator gets the file name. 5. The data is writed to kafka. Use the above process, when there are many files at the same timeļ¼Œ StreamSets SFTP/FTP client can't get all the files. Is there any way to solve this problem?