Ask Your Question
1

Can two destinations trigger the same executor?

asked 2018-01-17 04:11:38 -0600

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I want to make a pipeline that triggers an event after the whole process finishes. The pipeline at some point breaks away with a stream selector where two sub-pipelines are created ending at their destination respectively. I want these two destinations to produce an event that goes to the same executor. Is there a way so that the executor can know which destination has produce what event?

For example, destination 1 finishes at time x and produces event file-closed and then destination 2 finishes at time y>x and produces event file-closed. i want the executor to know that when destination 1 is file-closed and destination 2 is file closed then trigger an action.

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
0

answered 2018-01-18 01:30:50 -0600

kapa gravatar image

This is great, thanks a lot!

edit flag offensive delete link more
0

answered 2018-01-17 10:09:43 -0600

metadaddy gravatar image

updated 2018-01-17 10:18:49 -0600

Yes, this is possible. Use an Expression Evaluator between each destination and the executor to set a record field or attribute. Here's an example:

image description

Here, the raw data source has some delimited test data:

test
1
-1

The Stream Selector simply compares the test field to 0. If the value is greater than zero, then the record is written to a file in JSON format via the upper Local FS destination, otherwise it's sent to the lower one.

The two Expression Evaluators set a field in the record: /flag; the upper one sets it to 1. the lower to 2. They also set a /timestamp field to ${time:now()}. The Shell Executor copies the values of the flag and timestamp fields and event type into the script's environment:

image description

and its script writes their values to a file:

echo $FLAG, $EVENT_TYPE, $TIME_NOW >> /tmp/out/flags

Running the pipeline results in the expected output:

file1

{"test":"1"}

file2

{"test":"-1"}

flags

1, file-closed, Wed Jan 17 08:16:01 PST 2018
2, file-closed, Wed Jan 17 08:16:01 PST 2018
edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-01-17 04:11:38 -0600

Seen: 55 times

Last updated: Jan 18