Ask Your Question

Reading the properties/values from a external file in streamsets origin

asked 2017-12-08 18:33:41 -0500

praneeth gravatar image

updated 2017-12-11 13:24:42 -0500

metadaddy gravatar image

Is any use case available for reading the parameters/values from external file/table in a StreamSets origin?

edit retag flag offensive close merge delete


What file format do you want to read?

metadaddy gravatar imagemetadaddy ( 2017-12-08 18:56:48 -0500 )edit

@metadaddy: .txt or csv file format.

praneeth gravatar imagepraneeth ( 2017-12-08 21:13:49 -0500 )edit

2 Answers

Sort by ยป oldest newest most voted

answered 2020-01-27 10:07:42 -0500

ghanashyaml gravatar image

I think we can setup a separate pipeline to read data from JDBC or a directory with specific file names and setup variables. Join this pipeline to the main pipeline of readying csv files and populate parameters from variables setup earlier. Have not tried this but might work.

edit flag offensive delete link more

answered 2017-12-11 13:24:03 -0500

metadaddy gravatar image

The StreamSets Tutorial covers ingest of CSV format data.

edit flag offensive delete link more


@metadaddy Thank you for answer. But here are my senario clearly, read all the files from sftp/ftp/https and load into Hadoop. So i have to configure the parameters for sftp like host address, usrname,pwd etc. Those parameters have to take from external .txt/.csv file not hardcoded into processor.

praneeth gravatar imagepraneeth ( 2017-12-11 15:07:07 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-12-08 18:33:41 -0500

Seen: 619 times

Last updated: Dec 11 '17