Ask Your Question
1

Unable to execute Hive Query Start Event in Pipeline

asked 2017-09-28 10:55:51 -0500

jshivers gravatar image

updated 2017-10-02 11:29:37 -0500

metadaddy gravatar image

I have a pipeline which connects to a MS SQL server using the JDBC query consumer origin and writes data to hive. This portion of the pipeline works fine.

I have created a Hive Query start event to truncate the target hive table at the beginning of each run. This step receives the following error when the pipeline is executed. Any ideas?

2017-09-28 10:40:41,915 [user:*admin] [pipeline:TDM - Well Header/TDM020885e7-4735-451b-bf61-c25d28df3f7b] [runner:] [thread:ProductionPipelineRunnable-TDM020885e7-4735-451b-bf61-c25d28df3f7b-TDM - Well Header] ERROR HiveQueryExecutor - Error when processing record: com.streamsets.pipeline.api.base.OnRecordErrorException: QUERY_EXECUTOR_001 - Failed to execute queries. Details : Failed to execute query 'truncate table tdm.well_header'. Reason: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Exception while processing
com.streamsets.pipeline.api.base.OnRecordErrorException: QUERY_EXECUTOR_001 - Failed to execute queries. Details : Failed to execute query 'truncate table tdm.well_header'. Reason: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Exception while processing
    at com.streamsets.pipeline.stage.destination.hive.queryexecutor.HiveQueryExecutor.executeQueries(HiveQueryExecutor.java:152)
    at com.streamsets.pipeline.stage.destination.hive.queryexecutor.HiveQueryExecutor.write(HiveQueryExecutor.java:91)
    at com.streamsets.pipeline.configurablestage.DExecutor.write(DExecutor.java:34)
    at com.streamsets.datacollector.runner.StageRuntime$2.call(StageRuntime.java:238)
    at com.streamsets.datacollector.runner.StageRuntime$2.call(StageRuntime.java:222)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:180)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:249)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runLifecycleEvent(ProductionPipelineRunner.java:329)
    at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:273)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:94)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:74)
    at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:740)
    at com.streamsets.datacollector.execution.AbstractRunner.lambda$scheduleForRetries$0(AbstractRunner.java:173)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:249)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:245)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
edit retag flag offensive close merge delete

Comments

Can you see anything in the Hive logs?

metadaddy gravatar imagemetadaddy ( 2017-10-02 11:30:27 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2017-10-02 12:19:46 -0500

jshivers gravatar image

This is no longer an issue. I needed to add a hive proxy user to the Hive JDBC URL in the Start Event - Hive Query.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-09-27 15:47:15 -0500

Seen: 99 times

Last updated: Oct 02 '17