Ask Your Question

HIVE_25 - Trying to create partition for non existing table

asked 2018-02-27 05:01:47 -0500

dhalfageme gravatar image

updated 2018-02-28 02:15:17 -0500

Hello, I'm facing the following issue. My streamsets pipeline is as follows:

Hive workflow

The flows starts from scratch, which means no table is created before running the pipeline

Hive Metastore recives 2 records as input:

  1. Table creation
  2. Partition creation

I'm receiving the error: HIVE_25 - Trying to create partition for non existing table

If I look at the Hive error, I found the following near to the create table statement that was tried to execute: FAILED: SemanticException [Error 10035]: Column repeated in partitioning columns

The column I'm ussing to partition is specified in both list of columns and partitioned by clauses.

Thank you in advance.

edit retag flag offensive close merge delete


How did you configure hive and what is the version you are using?

Shruthi gravatar imageShruthi ( 2018-02-27 06:40:50 -0500 )edit

This is my configuration: JDBC URL: jdbc:hive2:// JDBC DriverName: org.apache.hive.jdbc.HiveDriver Haddop conbfiguration directory: /etc/hive/conf

dhalfageme gravatar imagedhalfageme ( 2018-02-27 07:11:55 -0500 )edit

I'm running against a Cloudera distribuction (CDH 5.14) but with Stage library CDH 5.12 which is the higher version that our current streamsets support (Be can't update it but the moment because of another issue related with kafka). (We are not experiencing issues wich CDH 5.12 in other components)

dhalfageme gravatar imagedhalfageme ( 2018-02-27 07:13:15 -0500 )edit

(I edited the question with more error log info that I found)

dhalfageme gravatar imagedhalfageme ( 2018-02-27 07:19:22 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2018-03-07 04:14:13 -0500

dhalfageme gravatar image

I finally fidured out what was the error. The partition name I was giving was the same name than an existing field, which is invalid. I created a header attribute by copying the field value, removed the field from the record before the HiveMetadata procesor, and configure the partition name with the removed field name and getting the field value from the header attribute.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-02-27 05:01:47 -0500

Seen: 24 times

Last updated: Mar 07