Ask Your Question

xml to hive processing

asked 2018-07-09 15:13:22 -0500

anonymous user


updated 2018-07-09 17:36:59 -0500

Hi, I am new to streamsets and trying to process xml file to hive table. The pipeline is failing because of list data types. I cannot convert list to string using field type converter. is there any way i can process xml.

Source : xml file (configuration - reading in xml data format) Target: Hive table <root version="3.0"> <header> <company>TEST COMPANY</company> <sendername>TEST SENDER</sendername> <senderphone>222-333-4444</senderphone> </header> <a1 action="A" id="1234567"> <account id="1234"/> <subaccount id="123"/> <qty>20</qty> <type id="10456"/> <post id="1"/> <num> </a1> <a1 action="A" id="23456789"> <account id="141004"/> <subaccount id="20"/> <qty>30</qty> <type id="10456"/> <pos id="1"/> <num> </a1> <footer> <recordcount>2</recordcount> </footer> </root>

edit retag flag offensive close merge delete


Hi! Can you provide sample XML input *after* removing any personal and/or confidential info as well as your origin settings?

iamontheinet gravatar imageiamontheinet ( 2018-07-09 15:55:37 -0500 )edit

sample xml is added above. Configuration options: origin - hdfs standalone Data Format: xml delimiter: A1

Robot gravatar imageRobot ( 2018-07-09 17:38:44 -0500 )edit

What are you trying to convert -- which list to string? What's your desired output? What's the error you're getting? (Note that <num> doesn't have closing tag--prob just a typo.) I'd also recommend starting with a basic destination (like 'Trash', for e.g.) till you have everything else working.

iamontheinet gravatar imageiamontheinet ( 2018-07-09 18:04:44 -0500 )edit

num closing tag is typo <num></num> I need all fields under A1 element and write them to hive account id, subaccount id, qty, type id, post id and num fields are showing as list in preview and hive is throwing an error HIVE_19 unsupported data type LIST

Robot gravatar imageRobot ( 2018-07-09 18:14:39 -0500 )edit

PREVIEW INPUT DATA: Input Data Record1 : MAP attr|action : STRING A attr|id : STRING 1234567 Account : LIST [ 1 ] 0 : MAP attr|id : STRING 1234 Subaccount : LIST [ 1 ] Qty : LIST [ 1 ] Type : LIST [ 1 ] Post : LIST [ 1 ] Num : LIST [ 1 ]

Robot gravatar imageRobot ( 2018-07-09 18:17:27 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2018-07-09 18:57:07 -0500

iamontheinet gravatar image


Use Field Flattener to flatten entire record and then you may use Field Renamer to rename qualified names.

Here are some screenshots for reference purposes:

image description

image description

image description

Hope this helps!

Cheers, Dash

edit flag offensive delete link more


Thank you!

Robot gravatar imageRobot ( 2018-07-09 20:13:00 -0500 )edit

You're welcome! :)

iamontheinet gravatar imageiamontheinet ( 2018-07-09 20:42:11 -0500 )edit
Login/Signup to Answer

Question Tools



Asked: 2018-07-09 15:13:22 -0500

Seen: 148 times

Last updated: Jul 09 '18