Ask Your Question

Revision history [back]

Yes, this is possible. Use an Expression Evaluator between each destination and the executor to set a record field or attribute. Here's an example:

image description

Here, the raw data source has some delimited test data:

test
1
-1

The Stream Selector simply compares the test field to 0. If the value is greater than zero, then the record is written to a file in JSON format via the upper Local FS destination, otherwise it's sent to the lower one. The two Expression Evaluators set a field in the record: /flag; the upper one sets it to 1. the lower to 2. The Shell Executor copies the values of the flag field and event type into the script's environment:

image description

and its script writes their values to a file:

echo $FLAG, $EVENT_TYPE >> /tmp/out/flags

Running the pipeline results in the expected output:

file1

{"test":"1"}

file2

{"test":"-1"}

flags

1, file-closed
2, file-closed

Yes, this is possible. Use an Expression Evaluator between each destination and the executor to set a record field or attribute. Here's an example:

image description

Here, the raw data source has some delimited test data:

test
1
-1

The Stream Selector simply compares the test field to 0. If the value is greater than zero, then the record is written to a file in JSON format via the upper Local FS destination, otherwise it's sent to the lower one. one.

The two Expression Evaluators set a field in the record: /flag; the upper one sets it to 1. the lower to 2. They also set a /timestamp field to ${time:now()}. The Shell Executor copies the values of the flag field and timestamp fields and event type into the script's environment:

image descriptionimage description

and its script writes their values to a file:

echo $FLAG, $EVENT_TYPE $EVENT_TYPE, $TIME_NOW >> /tmp/out/flags

Running the pipeline results in the expected output:

file1

{"test":"1"}

file2

{"test":"-1"}

flags

1, file-closed
file-closed, Wed Jan 17 08:16:01 PST 2018
2, file-closed
file-closed, Wed Jan 17 08:16:01 PST 2018