Stream Selection anomaly

asked 2019-08-09 09:29:16 -0500

daveh
113 ●13 ●16 ●22

updated 2019-08-12 17:50:39 -0500

metadaddy

4173 ●26 ●40 ●72 https://about.me/patpa...

I have a Stream Selector stage with the following condition:

${record:value('/create-user_response/data[0]/request_results/this_does_not_exist') == 'failure'}

The field at the end of the path DOES NOT EXIST, yet the pipeline doesn't error out completely and actually passes the record to a continuing processor stage instead of an error stage that its supposed to send to.

Any insight would be greatly appreciated.

Thanks.

Per Jeff's update request...

For some reason now the behavior has changed and the Selector is sending to the proper stream (don't know why it changed as before is was definitely sending it to the default) However, I'm still not sure why it allowed the records to be sent to Stream 1 when the field in the the condition does not exists.

Screen shot (redacted a bit for privacy sake):

$C:\fakepath\2019-08-12_11-40-16.jpg$

Comments

As a sanity check, can you trigger this in a preview? In other words, you run a preview that shows the record entering the stream selector, and shows the output going down the wrong lane?

jeff ( 2019-08-09 15:21:46 -0500 )edit

Jeff, yes when running in preview it sends to the wrong selection branch. Thx.

daveh ( 2019-08-12 10:09:24 -0500 )edit

Please update your question showing screenshots of the configuration, and incoming record into the stream selector, and outgoing record down the incorrect path

jeff ( 2019-08-12 10:10:21 -0500 )edit

Please paste the stage configuration for your stream selector too. I am noticing from the first line of your preview screenshot (the "Condition" string) that it's different from what you pasted in the question.

jeff ( 2019-08-13 09:27:13 -0500 )edit

add a comment

answered 2019-08-13 19:21:25 -0500

metadaddy

4173 ●26 ●40 ●72 https://about.me/patpa...

updated 2019-08-13 19:26:05 -0500

This pipeline is behaving as expected. We can use Expression Evaluator to see a bit more detail of what's going on.

Here's my test pipeline:

image description

Here's my test data:

image description

My Expression Evaluator will show me the value of the field /f4 and the value of the condition ${record:value('/f4') == 'success'}

image description

My Stream Selector will send records that match ${record:value('/f4') == 'success'} to stream 1, and all others to stream 2

image description

Now, in preview, this is what I see in the Expression Evaluator:

image description

For the first record, the value of the expression ${record:value('/f4')} is null. There's no error here - this EL function is defined to return null if the field does not exist. Consequently, since null != 'success', the value of the condition is false. For the second record, the values are 'success' and 'true', as expected.

Now, looking at the Stream Selector:

image description

The first record does not match the condition ${record:value('/f4') == 'success'}, so it goes to stream 2. The second record does match, so it goes to stream 1.

Note that, if the existence of the field is important, you can use record:exists('/path/to/field') either on its own or in an expression with record:value('/path/to/field'). If records that are missing a field should be sent to the error stream, then you can simply list that field in the stage's required fields configuration:

image description