Properties file in streamsets pipeline

asked 2019-05-12 23:44:10 -0500

rickymart gravatar image

Hi Team,

Is there a way i can read a properties file from the cloud like Azure / Google cloud /AWS and use it in the evaluator? The properties file should be read once and stored in cache for the lookups from the evaluator like javascript eval.

Based on the incoming data, the properties file keys are matched to get the respective value.

For example: I am storing a file (map.properties) in Blob storage or S3. The contents of the map.properties are as follows:

key1=value1
key2=value2
key3=value3

Incoming records :

{"keyfield":"key1", "value":"2"}
{"keyfield":"key2", "value":"4"}

So the keyfield should be read and based on the keyfield value , the properties file has to be read and the corresponding value should be appended with the record.

Output:

 {"keyfield":"key1", "value":"2","propvalue":"value1"}
 {"keyfield":"key2", "value":"4","propvalue":"value2"}
edit retag flag offensive close merge delete