SFTP/FTP can't get all the files when there are many files at the same time

asked 2019-02-28 20:11:35 -0600

anonymous user

Anonymous

I get files from ftp, and then use expression evlauator to get the file name, and finally write the data to kafka. The detailed steps are as follows: 1. Uploads multiple files to the ftp server directory A at the same time. 2. To ensure the files are uploaded, after a few minutes, I move all of these files to directory B, and touch them. 3. StreamSets SFTP/FTP client reads files from directory B. 4. Expression Evlauator gets the file name. 5. The data is writed to kafka. Use the above process, when there are many files at the same time, StreamSets SFTP/FTP client can't get all the files. Is there any way to solve this problem?

edit retag flag offensive close merge delete

Comments

Can you post the errors or the logs for confirming if all the files aren't copied?

nishparadox gravatar imagenishparadox ( 2019-03-03 07:49:50 -0600 )edit