Ask Your Question
0

Hive_40 Column 'xxx' is used for partition column and at the same time appears in input data

asked 2018-04-17 23:28:57 -0600

casel.chen gravatar image

image description

image description

com.streamsets.pipeline.api.base.OnRecordErrorException: HIVE_40 - Column 'tenantid' is used for partition column and at the same time appears in input data at com.streamsets.pipeline.stage.processor.hive.HiveMetadataProcessor.process(HiveMetadataProcessor.java:595) at com.streamsets.pipeline.api.base.RecordProcessor.process(RecordProcessor.java:52) at com.streamsets.pipeline.configurablestage.DProcessor.process(DProcessor.java:35) at com.streamsets.datacollector.runner.StageRuntime$2.call(StageRuntime.java:246) at com.streamsets.datacollector.runner.StageRuntime$2.call(StageRuntime.java:236) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:194) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:263) at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:225) at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.lambda$runSourceLessBatch$0(PreviewPipelineRunner.java:343) at com.streamsets.datacollector.runner.PipeRunner.executeBatch(PipeRunner.java:132) at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.runSourceLessBatch(PreviewPipelineRunner.java:339) at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.runPollSource(PreviewPipelineRunner.java:321) at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.run(PreviewPipelineRunner.java:215) at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:514) at com.streamsets.datacollector.runner.preview.PreviewPipeline.run(PreviewPipeline.java:51) at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:214) at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$0(AsyncPreviewer.java:94) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:227) at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:223) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:227) at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:223) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-07-13 08:55:50 -0600

bob gravatar image

HIVE_40 occurs when the name of a data column matches the name of a partitioning column.

The easiest way to fix this is to rename the data column and update the code that accesses the table.

Although it seems like it would be less impact to rename the partitioning column, I have not found a way to rename it since the partitioning column name is "exposed" as part of the file name. The alternative seems to be to drop and recreate the partition using a different column name.

edit flag offensive delete link more

Comments

Thanks for the answer. My solution is put the partition column name in record header attributes and then remove it in record fields.

casel.chen gravatar imagecasel.chen ( 2018-07-13 19:36:46 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-04-17 23:28:57 -0600

Seen: 187 times

Last updated: Jul 13