Ask Your Question

Storing complex data structures in record attributes?

asked 2018-03-09 16:35:46 -0600

mstang gravatar image

I may be asking the wrong question... is there a way to store a list of maps from a pipeline stage into a variable or record attribute for use later in the pipeline? The reason is I have an http origin that produces JSON containing a list of a list of maps. I have a second http lookup that produces a list of maps. After running the lookup I want to pivot the record coming from the origin breaking the first level of lists into separate records. Then I want to zip that with my lookup list. If I pivot (and don't keep fields so I get separate records) I lose my lookup. If I wait and do the lookup after the pivot I would have to do hundreds of lookups instead of 1. If I could store the list of maps from the lookup I could reinsert it into the records after the pivot. When I tried using expression evaluator to store in a record attribute the list of maps was stored as a string that I haven't figured out reformat later into a list of maps... any ideas?

( I thought about running the pipelines separately and using an hbase type lookup after the pivot, but I'm on maprdb and not running hbase and it didn't look like that processor would work).

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-03-12 14:01:51 -0600

jeff gravatar image

I would try to approach it this way:

  1. Use the JSON Generator processor to serialize your /list_maps field to a new STRING field, let's suppose it's called /list_maps_serialized
  2. Use the Expression Evaluator processor to capture the value of /list_maps_serialized into a record attribute, suppose it's called list_maps_serialized_attr. As you noted, this will have the type String as all record attributes do.
  3. After your pivoting, add another Expression Evaluator, which will simply transfer the header attribute (via record:attribute EL function) back to a new field, called /list_maps_from_header
  4. Finally, put a Data Parser processor (with JSON format) after to parse the /list_maps_from_header field to a new field, which will contain the parsed structure.
edit flag offensive delete link more


That sounds like a good approach. I had tried using the Data Parser on the record attribute string that was automatically created when I saved the /list_maps to an attribute but that didn't work. I ended up using a javascript processor to loop through all the nesting and create individual attribs

mstang gravatar imagemstang ( 2018-03-22 14:32:11 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-03-09 16:35:46 -0600

Seen: 323 times

Last updated: Mar 12 '18