Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Hive Metadata table name can take multiple expressions ?

I have multiple tables data coming from SDC RPC. My requirement is I need the partition for few tables and not for every table because some table data doesn't satisfy our minimum partition requirement of 256 MB. For the once I don't want to do partitions I kept the regular expression as ${record:attribute('jdbc.tables1')} in hive table name config and for the once I need partition looks like this ${record:attribute('jdbc.tables')} i don't see an option of adding multiple regular expression in one hive metadata and making the partitions accordingly.

So i made a pipeline as the first picture and it works fine, but the challenge is because both the hive_metdata processors receive the same data even though it can make the tables and feed the data we will see the error in the second screenshot. how to get around this?

image description

image description

Side Note: I can get around it by broadcasting the data to other port and reading it, but I don't want to do that.

What I Tried so far? Broadcasting to same port but changing the sdc-rpc ID, which didn't work :(

Hive Metadata table name can take multiple expressions ?

I have multiple tables data coming from SDC RPC. My requirement is I need the partition for few tables and not for every table because some table data doesn't satisfy our minimum partition requirement of 256 MB. For the once I don't want to do partitions I kept the regular expression as ${record:attribute('jdbc.tables1')} in hive table name config and for the once I need partition looks like this ${record:attribute('jdbc.tables')} i don't see an option of adding multiple regular expression in one hive metadata and making the partitions accordingly.

So i made a pipeline as the first picture and it works fine, but the challenge is because both the hive_metdata processors receive the same data even though it can make the tables and feed the data we will see the error in the second screenshot. how to get around this?

image descriptionimage description

image description

Side Note: I can get around it by broadcasting the data to other port and reading it, but I don't want to do that.

What I Tried so far? Broadcasting to same port but changing the sdc-rpc ID, which didn't work :(