Ask Your Question
0

JSON - Wildcard Use for Arrays and Maps

asked 2018-07-10 13:47:54 -0600

KRT gravatar image

updated 2018-07-10 14:02:54 -0600

I have to access JSON array and validate the field is present in the record. From the below sample JSON, I need to access "Listing" and validate ListId field is available or not? If any records without ListId field it should sent to error.

VALID record { "Id":2296, "Department":"StreamSet", "Status":"Open",
"CreatedDate":"2017-04-18T11:55:23.087" "Listing":[ { "ListId":111, "Description":"aaaaaaaaaaaa", "Status":"Closed", "CreatedDate":"2018-01-18T05:21:18.48" }, { "ListId":222, "Description":"bbbbbbbbbb", "Status":"Open", "CreatedDate":"2018-06-19T09:27:54.343" } ] }

INVALID record { "Id":2297, "Department":"StreamSet", "Status":"Open",
"CreatedDate":"2017-04-18T11:55:23.087" "Listing":[ { "Description":"ccccccc", "Status":"Closed", "CreatedDate":"2018-01-18T05:21:18.48" }, { "ListId":333, "Description":"dddddd", "Status":"Open", "CreatedDate":"2018-06-19T09:27:54.343" } ] }

What I tried: Used Static Lookup processor and added the below condition in the precondition. But even the valid records are failing to pass the processor stage. ${record:value('/Listing[*]/ListId') != ''}

How to achieve this validation?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-07-10 14:08:33 -0600

jeff gravatar image

I think this ends up being a fairly complex validation, and you will likely need to use a scripting process (ex: the Jython one) to evaluate all of the fields in the /Listing LIST field to see if they have the ListId subfield. You can then set an attribute (in the script) like INVALID_LIST or something similar, then attach a stream selector processor afterward (or whatever you need to do) in order take differential action on them.

edit flag offensive delete link more

Comments

Hi Jeff, In support doc they mentioned some wildcards in array index - https://streamsets.com/documentation/datacollector/latest/help/#datacollector/UserGuide/Pipeline_Configuration/Expressions.html#concept_vqr_sqc_wr Can we use something like this? or what is the usage of wildcard is for?

KRT gravatar imageKRT ( 2018-07-10 14:26:24 -0600 )edit

Not exactly. Wildcards on field paths are supported, and really only make sense, when you have an operation that is to be performed on fields individually. For example: hashing, removing, type converting, etc. It's not clear how such a semantic would translate into a record-level operation.

jeff gravatar imagejeff ( 2018-07-11 14:07:03 -0600 )edit

Ex: in your particular use case, I assume that you want ALL records whose Listing values comport with the "required field". So you probably want the condition where the set of fields matched by an expression matches the size of the list.

jeff gravatar imagejeff ( 2018-07-11 14:08:38 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-07-10 13:47:54 -0600

Seen: 87 times

Last updated: Jul 10