Ask Your Question
1

HIVE_32 - Table test is created using Storage Format Type org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, but Avro requested instead

asked 2018-06-14 02:05:40 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

When I use Hive Metadata Processor, the output shows this error. The input file type is datatime, and the output file type is string.

com.streamsets.pipeline.api.base.OnRecordErrorException: HIVE_32 - Table test is created using Storage Format Type org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, but Avro requested instead 
    at com.streamsets.pipeline.stage.processor.hive.HiveMetadataProcessor.process(HiveMetadataProcessor.java:595)
    at com.streamsets.pipeline.api.base.RecordProcessor.process(RecordProcessor.java:52)
    at com.streamsets.pipeline.api.base.configurablestage.DProcessor.process(DProcessor.java:35)
    at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:245)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:195)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:257)
    at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:219)
    at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.lambda$runSourceLessBatch$0(PreviewPipelineRunner.java:337)
    at com.streamsets.datacollector.runner.PipeRunner.executeBatch(PipeRunner.java:136)
    at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.runSourceLessBatch(PreviewPipelineRunner.java:333)
    at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.runPollSource(PreviewPipelineRunner.java:315)
    at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.run(PreviewPipelineRunner.java:209)
    at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:522)
    at com.streamsets.datacollector.runner.preview.PreviewPipeline.run(PreviewPipeline.java:51)
    at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:214)
    at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$0(AsyncPreviewer.java:94)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-06-14 09:02:52 -0500

metadaddy gravatar image

It looks like you created the test table manually, using LazySimpleSerDe, before running the pipeline, and you're getting this error because Data Collector is trying to write Avro data. Delete the test table and the Hive Metadata processor / Hive Metastore destination will create it automatically when the first record is processed.

edit flag offensive delete link more

Comments

yes,you are right.after i drop the test table,it can do normal. But now i have another question,when orgin is mongodb collection and it have not the same filed like rdbms,it will discard the different document between the hive table filed.

supersujj gravatar imagesupersujj ( 2018-06-14 20:32:58 -0500 )edit

Please create a new question for this, rather than asking in a comment

metadaddy gravatar imagemetadaddy ( 2018-06-14 21:34:05 -0500 )edit

Also, vote up the answer if it worked for you.

metadaddy gravatar imagemetadaddy ( 2018-06-14 21:34:22 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-06-14 02:05:40 -0500

Seen: 9 times

Last updated: Jun 14