Ask Your Question

HIVE_25 - Trying to create partition for non existing table

asked 2018-02-27 05:01:47 -0600

dhalfageme gravatar image

updated 2018-02-28 02:15:17 -0600

Hello, I'm facing the following issue. My streamsets pipeline is as follows:

Hive workflow

The flows starts from scratch, which means no table is created before running the pipeline

Hive Metastore recives 2 records as input:

  1. Table creation
  2. Partition creation

I'm receiving the error: HIVE_25 - Trying to create partition for non existing table

If I look at the Hive error, I found the following near to the create table statement that was tried to execute: FAILED: SemanticException [Error 10035]: Column repeated in partitioning columns

The column I'm ussing to partition is specified in both list of columns and partitioned by clauses.

Thank you in advance.

edit retag flag offensive close merge delete


How did you configure hive and what is the version you are using?

Shruthi gravatar imageShruthi ( 2018-02-27 06:40:50 -0600 )edit

This is my configuration: JDBC URL: jdbc:hive2:// JDBC DriverName: org.apache.hive.jdbc.HiveDriver Haddop conbfiguration directory: /etc/hive/conf

dhalfageme gravatar imagedhalfageme ( 2018-02-27 07:11:55 -0600 )edit

I'm running against a Cloudera distribuction (CDH 5.14) but with Stage library CDH 5.12 which is the higher version that our current streamsets support (Be can't update it but the moment because of another issue related with kafka). (We are not experiencing issues wich CDH 5.12 in other components)

dhalfageme gravatar imagedhalfageme ( 2018-02-27 07:13:15 -0600 )edit

(I edited the question with more error log info that I found)

dhalfageme gravatar imagedhalfageme ( 2018-02-27 07:19:22 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2018-03-07 04:14:13 -0600

dhalfageme gravatar image

updated 2018-05-24 12:11:58 -0600

metadaddy gravatar image

I finally figured out what was the error. The partition name I was giving was the same name as an existing field, which is invalid. I created a header attribute by copying the field value, removed the field from the record before the HiveMetadata processor, and configure the partition name with the removed field name and getting the field value from the header attribute.

edit flag offensive delete link more


I am facing the same problem but I don't think I doing it right I mean your fix, Is there any way you can share just the part with hive metadata and fix you did in json ?

taher843 gravatar imagetaher843 ( 2018-05-24 12:01:14 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-02-27 05:01:47 -0600

Seen: 817 times

Last updated: May 24 '18