Ask Your Question

How to keep null xml fields when using Field Flattener?

asked 2019-03-05 09:24:51 -0600

fede gravatar image

I'm Using streamset and have a nested XML which I need to parse into a .csv. When I apply the Field Flattener component, it flattens the xml, but it flattens stripping out the field with NULL values. example :

<xml>  <a>null</a>    <b>
       <d>NULL</d>    </b>    <b>
      <d>2</d>    </b> </xml>

output wanted in my excel :

|NULL   |   hi   |NULL |
|NULL   |  salut     |2 |

and not

| hi |<br>
| salut | 2 |
edit retag flag offensive close merge delete


Can you export your pipeline to JSON, remove any passwords etc, and add it (or a link to it) to your question?

metadaddy gravatar imagemetadaddy ( 2019-03-05 10:42:34 -0600 )edit

Sorry, but for GDPR laws I cannot post the json of pipeline even without any password or links etc.etc. in a public site

fede gravatar imagefede ( 2019-03-05 10:48:49 -0600 )edit

OK... Use preview to see where the "NULL" string is being converted to a null value. I think that's the problem here.

metadaddy gravatar imagemetadaddy ( 2019-03-05 11:28:27 -0600 )edit

Checking on preview, the field is listed as d : MAP >0 where it does not show nothing. checking with a Jython Compiler,the length of d field is 0 and is the only way to work on field that has NULL or are empty. I've some field that are null and most that are empty.

fede gravatar imagefede ( 2019-03-06 02:25:18 -0600 )edit

With field Flattener component, tag/fields with NULL or Empy value like "" are taken out. I want to keep fields with NULL or empty values, this is my goal. How can I tell streamsets Condition : all fields that you receive which each element has length == 0 then do ?

fede gravatar imagefede ( 2019-03-06 02:28:33 -0600 )edit

1 Answer

Sort by » oldest newest most voted

answered 2019-03-07 04:37:38 -0600

fede gravatar image

Since Pat only answer on comments, if someone is stuck on how to convert an xml into a csv, here is the explanation:

an xml is like a map, so create a Directory Stage component to read the files, explode them with a Pivot Field, if you have empty or Null values, use a jython and check every field with length == 0, replace it with a costant. Add a Flattener field and then a Field order component to convert the map record into Map - List record, et voilà, you have your record ready to be written on your local filesystem

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-03-05 08:38:07 -0600

Seen: 547 times

Last updated: Mar 07 '19