Ask Your Question
0

Read file in Processor Stage (Not destination or Origin)

asked 2020-07-16 14:51:20 -0500

evanmg7 gravatar image

Hi,

I am streaming messages from Eventhub. Part of the message body will contain the local directory path where to read in the file from. Therefore, my origin is eventhub streaming messages and a subsequent processor would need to be reading in a file from the path specified in the message. Is this possible? If so, how so?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2020-07-17 17:26:58 -0500

zac gravatar image

I don't know if this is the best way, but you could use a Groovy Evaluator Processor to read the file. You'll need to grant the Groovy script permissions to read from disk.

Update sdc-security.policy with:

// groovy source code - allow to read a file from disk
grant codebase "file:///groovy/script" {
   permission java.util.PropertyPermission "*", "read";
   permission java.io.FilePermission "/YOUR_PATH_HERE/*", "read";
};

You could also write your own Stage/Processor if you wanted. https://github.com/streamsets/tutorials

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-07-16 14:51:20 -0500

Seen: 169 times

Last updated: Jul 17