SFTP FTP client origin configuration issue for charset "utf-8".

asked 2019-01-21 05:10:57 -0500

Abhishek gravatar image

I am using SFTP FTP client origin to process txt files present at edge node, while working i found out that with my configuration of origin where Data Format is Text and Charset is set to UTF-8, my streamsets pipeline is not picking source files (from edge node) which are greater than 1 MB in size. I don't have much idea on charset configurations butI tried changing charset from UTF-8 to UTF-16, my pipeline picked up the files (greater than 1 MB) but change the text from plain English to Chinese (Chinese if i am not wrong).

Any workaround here?

Thanks in advance.

edit retag flag offensive close merge delete