I think we can setup a separate pipeline to read data from JDBC or a directory with specific file names and setup variables. Join this pipeline to the main pipeline of readying csv files and populate parameters from variables setup earlier. Have not tried this but might work.
The StreamSets Tutorial covers ingest of CSV format data.
@metadaddy Thank you for answer. But here are my senario clearly, read all the files from sftp/ftp/https and load into Hadoop. So i have to configure the parameters for sftp like host address, usrname,pwd etc. Those parameters have to take from external .txt/.csv file not hardcoded into processor.
Asked: 2017-12-08 18:33:41 -0500
Seen: 502 times
Last updated: Dec 11 '17
What file format do you want to read?
@metadaddy: .txt or csv file format.