Ask Your Question
1

Connecting to Apache Drill via JDBC

asked 2017-09-30 10:45:21 -0500

mstang gravatar image

updated 2017-09-30 16:12:48 -0500

metadaddy gravatar image

Has anyone successfully connected to Drill via the JDBC origin? I'm getting a strange error when I try...

I can connect from the same server using the same driver with sqlline (both direct and with zookeeper) though it always forces me to do an interactive login (I can't seem to pass UID and PWD in my connect string.

With the JDBC Origin I get this error when I try a JDBC connection direct to a drillbit:

2017-09-29 23:06:56,664 [user:*eusmist] [pipeline:drilltest/drilltest982f5d29-cf85-454d-a843-ae6935e0c2a1] [thread:preview-pool-1-thread-4] ERROR JdbcSource - Cannot connect to specified database: com.streamsets.pipeline.api.StageException: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.PoolInitializationException: Exception during pool initialization: org/apache/hadoop/conf/Configuration
com.streamsets.pipeline.api.StageException: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.PoolInitializationException: Exception during pool initialization: org/apache/hadoop/conf/Configuration
        at com.streamsets.pipeline.lib.jdbc.JdbcUtil.createDataSourceForRead(JdbcUtil.java:763)
        at com.streamsets.pipeline.stage.origin.jdbc.JdbcSource.init(JdbcSource.java:216)
        at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48)
        at com.streamsets.pipeline.configurablestage.DStage.init(DStage.java:36)
        at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:159)
        at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:101)
        at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:49)
        at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:382)
        at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:295)
        at com.streamsets.datacollector.runner.Pipeline.validateConfigs(Pipeline.java:205)
        at com.streamsets.datacollector.runner.preview.PreviewPipeline.validateConfigs(PreviewPipeline.java:60)
        at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.validateConfigs(SyncPreviewer.java:133)
        at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer$1.call(AsyncPreviewer.java:70)
        at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:249)
        at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
        at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:245)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: com.zaxxer.hikari.pool.PoolInitializationException: Exception during pool initialization: org/apache/hadoop/conf/Configuration
        at com.zaxxer.hikari.pool.HikariPool.initializeConnections(HikariPool.java:581)
        at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:152)
        at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:73)
        at com.streamsets.pipeline.lib.jdbc.JdbcUtil.createDataSourceForRead(JdbcUtil.java:756)
        ... 21 more
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
        at oadd.org.apache.drill.exec.rpc.security.plain.PlainFactory.createAndLoginUser(PlainFactory.java:61)
        at oadd.org.apache.drill.exec.rpc.user.UserClient.authenticate(UserClient.java:219)
        at oadd.org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:155)
        at oadd.org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
        at oadd.org.apache.drill.exec ...
(more)
edit retag flag offensive close merge delete

Comments

1

What jars did you install into your

streamsets-datacollector-jdbc-lib/lib
directory? It looks like this driver requires a class that is part of the `hadoop-common` artifact. Was that packaged with your driver? If so, it will need to be included along with the driver jar.

jeff gravatar imagejeff ( 2017-10-04 14:39:04 -0500 )edit

The only jar I've included was drill-jdbc-all-1.10.0.jar. This is the only jar I'm using with sqlline to connect from the same server. I agree it seems to be looking for hadoop-common, I'm just not sure why...

mstang gravatar imagemstang ( 2017-10-05 07:42:01 -0500 )edit

Perhaps it has to do with the authentication mechanism, purely based on the stack trace? When you use the command line tool, are you using the "plain" mechanism as SDC is apparently using?

jeff gravatar imagejeff ( 2017-10-05 12:50:46 -0500 )edit

Also, I notice that this driver is expecting drill-java-exec as a provided dependency, which is itself what includes hadoop-common. See https://mvnrepository.com/artifact/org.apache.drill.exec/drill-jdbc-all/1.11.0

jeff gravatar imagejeff ( 2017-10-05 13:16:51 -0500 )edit

I also think it has something to do with the way it is authenticating... as I mentioned originally from the cli tool if I specify auth=PLAIN and try to pass a username and password it seems to ignore it and force an interactive login.

mstang gravatar imagemstang ( 2017-10-06 12:45:15 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-03-25 14:17:10 -0500

mstang gravatar image

It took me a while to get back and test this, but the suggestions from Rupal and Jeff were correct. Streamsets needed additional libraries (I'm not exactly sure which ones, I just copied all the Drill JDBC jars and it did work).

edit flag offensive delete link more

Comments

oddly, this does not behave the same at all version levels. In 3.1.3 I only needed the single driver file but I had to change my legacy drivers jdbc class field to org.apache.drill.jdbc.Driver and add a validation query of "SELECT 1 FROM 1" and select auto commit under advanced tab.

mstang gravatar imagemstang ( 2018-04-22 13:13:58 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-09-30 10:45:21 -0500

Seen: 240 times

Last updated: Mar 25