HIVE_32 - Table test is created using Storage Format Type org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, but Avro requested instead

MongoDB
hive

asked 2018-06-14 02:05:40 -0500

This post is a wiki. Anyone with karma >75 is welcome to improve it.

When I use Hive Metadata Processor, the output shows this error. The input file type is datatime, and the output file type is string.

com.streamsets.pipeline.api.base.OnRecordErrorException: HIVE_32 - Table test is created using Storage Format Type org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, but Avro requested instead 
    at com.streamsets.pipeline.stage.processor.hive.HiveMetadataProcessor.process(HiveMetadataProcessor.java:595)
    at com.streamsets.pipeline.api.base.RecordProcessor.process(RecordProcessor.java:52)
    at com.streamsets.pipeline.api.base.configurablestage.DProcessor.process(DProcessor.java:35)
    at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:245)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:195)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:257)
    at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:219)
    at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.lambda$runSourceLessBatch$0(PreviewPipelineRunner.java:337)
    at com.streamsets.datacollector.runner.PipeRunner.executeBatch(PipeRunner.java:136)
    at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.runSourceLessBatch(PreviewPipelineRunner.java:333)
    at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.runPollSource(PreviewPipelineRunner.java:315)
    at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.run(PreviewPipelineRunner.java:209)
    at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:522)
    at com.streamsets.datacollector.runner.preview.PreviewPipeline.run(PreviewPipeline.java:51)
    at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:214)
    at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$0(AsyncPreviewer.java:94)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

edit retag flag offensive close merge delete

add a comment

answered 2018-06-14 09:02:52 -0500

metadaddy

4173 ●26 ●40 ●72 https://about.me/patpa...

It looks like you created the test table manually, using LazySimpleSerDe, before running the pipeline, and you're getting this error because Data Collector is trying to write Avro data. Delete the test table and the Hive Metadata processor / Hive Metastore destination will create it automatically when the first record is processed.

edit flag offensive delete link

Comments

yes,you are right.after i drop the test table,it can do normal. But now i have another question,when orgin is mongodb collection and it have not the same filed like rdbms,it will discard the different document between the hive table filed.

supersujj ( 2018-06-14 20:32:58 -0500 )edit

Please create a new question for this, rather than asking in a comment

metadaddy ( 2018-06-14 21:34:05 -0500 )edit

Also, vote up the answer if it worked for you.

metadaddy ( 2018-06-14 21:34:22 -0500 )edit

add a comment

HIVE_32 - Table test is created using Storage Format Type org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, but Avro requested instead

1 Answer

Comments

Question Tools

Stats

Related questions

HIVE_32 - Table test is created using Storage Format Type org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, but Avro requested instead edit

1 Answer

Comments

Question Tools

Stats

Related questions

HIVE_32 - Table test is created using Storage Format Type org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, but Avro requested instead