Ask Your Question
1

JDBC Query Consumer - Mapping to Hive Metadata

asked 2017-11-01 14:04:05 -0500

Roh gravatar image

updated 2017-11-01 14:19:30 -0500

jeff gravatar image

I have 3 JDBC origins writing to the SDC RPC destination(same destination). Two of them are JDBC consumers and another one is JDBC multi-table consumer.

On the receiving side, i have an SDC RPC origin connected to the hive/hdfs and so on. My question is i'm using this ${record:attribute('jdbc.tables')} in the hive metadata at table name and its working fine for the data coming from Multitable consumer origin but its failing to feed the data coming from JDBC consumer. What should I add to the JDBC consumer so that hive can accept the data in this format ${record:attribute('jdbc.tables')} in the receiving side.

Here is the snapshot of the advanced table for the jdbc consumer. image description

ERROR SC : image description

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
2

answered 2017-11-01 14:18:35 -0500

jeff gravatar image

updated 2017-11-01 14:55:40 -0500

When using the query consumer, there is the option for Create JDBC Header Attributes, which can, in some cases, populate jdbc.tables similarly to in JDBC multi-table origin. There are various reasons why this might not work (ex: your query itself doesn't query a single table, the driver doesn't support population of this metadata, etc.). You can do a preview, with Show Record/Field Header option checked, to see if this is being populated.

If this isn't working as expected, and you know the table name that you want to appear in the attribute, you can simply add a separate expression evaluator processor after your origin to populate this. Specifically, use an entry in the Header Attribute Expressions section. You can simply reuse the jdbc.tables name and set it yourself, or choose a different attribute name. Either way, this should be specified under the Header Attribute column, and you can then use that attribute name in your destination.

edit flag offensive delete link more

Comments

I don't want to change the expression in the hive. Can I know what exactly I have to do in the expression evaluator? and can i leave the JDBC header prefix as `jdbc.` and by ticking the header attributes?

Roh gravatar imageRoh ( 2017-11-01 14:42:26 -0500 )edit

I was mistaken about the query origin. I have updated the answer with more info.

jeff gravatar imagejeff ( 2017-11-01 14:54:58 -0500 )edit

Working like charm :) thank you so much @jeff

Roh gravatar imageRoh ( 2017-11-01 15:52:25 -0500 )edit
Login/Signup to Answer

Question Tools

2 followers

Stats

Asked: 2017-11-01 14:04:05 -0500

Seen: 2,116 times

Last updated: Nov 01 '17