Ask Your Question
0

How to use other JDBC driver for JDBC Producer?

asked 2019-07-18 03:11:17 -0500

foxgarden gravatar image

I know Mysql/Oracle/SQLServer/PG driver can be used in JDBC Producer.

Now I want to use Snappydata(something like Memsql) driver in this destination.

So I:

  1. put snappydata-jdbc_2.11-1.1.0.jar in /opt/sdc-extras/streamsets-datacollector-jdbc-lib/lib
  2. set sdc-env.sh and sdc-security.policy
  3. restart

When I preview the pipeline, I got below message:

2019-07-18 15:07:57,773 snappydata-test/snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e INFO Processing lifecycle start event with stage Pipeline *admin preview-pool-1-thread-1 2019-07-18 15:07:57,837 snappydata-test/snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e INFO Registered JDBC drivers: JdbcUtil *admin preview-pool-1-thread-1 2019-07-18 15:07:57,873 snappydata-test/snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e INFO Destroying pipeline with reason=FINISHED Pipeline *admin preview-pool-1-thread-1 2019-07-18 15:07:57,899 snappydata-test/snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e INFO Processing lifecycle stop event Pipeline *admin preview-pool-1-thread-1 2019-07-18 15:07:57,900 snappydata-test/snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e INFO Pipeline finished destroying with final reason=FINISHED Pipeline *admin preview-pool-1-thread-1 2019-07-18 15:07:57,906 snappydata-test/snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e ERROR Uncaught throwable from com.streamsets.datacollector.execution.preview.async.AsyncPreviewer$$Lambda$111/942442767@116a3e9e: com.streamsets.datacollector.util.PipelineException: PREVIEW_0003 - Encountered error while previewing : java.lang.NoClassDefFoundError: Could not initialize class io.snappydata.jdbc.ClientDriver SafeScheduledExecutorService *admin preview-pool-1-thread-1 com.streamsets.datacollector.util.PipelineException: PREVIEW_0003 - Encountered error while previewing : java.lang.NoClassDefFoundError: Could not initialize class io.snappydata.jdbc.ClientDriver at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:248) at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$0(AsyncPreviewer.java:95) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226) at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226) at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoClassDefFoundError: Could not initialize class io.snappydata.jdbc.ClientDriver at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at java.sql.DriverManager.isDriverAllowed(DriverManager.java:556) at java.sql.DriverManager.isDriverAllowed(DriverManager.java:548) at java.sql.DriverManager.getDrivers(DriverManager.java:446) at com.streamsets.pipeline.lib.jdbc.JdbcUtil.createDataSourceConfig(JdbcUtil.java:708) at com.streamsets.pipeline.lib.jdbc.JdbcUtil.createDataSourceForRead(JdbcUtil.java:793) at com.streamsets.pipeline.stage.origin.jdbc.JdbcSource.init(JdbcSource.java:220) at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48) at com.streamsets.pipeline.api.base.configurablestage.DStage.init(DStage.java:36) at com.streamsets.datacollector.runner.StageRuntime.lambda$init$0(StageRuntime.java:211) at com.streamsets.datacollector.util.LambdaUtil.withClassLoaderInternal(LambdaUtil.java:148) at ... (more)

edit retag flag offensive close merge delete

Comments

Can you try using the package manager to install the driver instead? https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Configuration/ExternalLibs.html#concept_amy_pzs_gz See if the behavior is any different when using that approach.

jeff gravatar imagejeff ( 2019-07-18 15:31:00 -0500 )edit

Same exception. I'm a little confused, it seems that SDC cannot find that jar file. But if I use mysql-connector-java-8.0.13.jar for Mysql, or sqljdbc42.jar for SqlServer, it works well. So my environment should be no problem.

foxgarden gravatar imagefoxgarden ( 2019-07-18 20:22:26 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-07-18 22:35:36 -0500

foxgarden gravatar image

@jeff

Finally I resolved this issue with an ugly method:

  1. Download source code api-classloader.properties, add a line:

    io.snappydata.jdbc.

  2. Upload this file to server, under bootstrap jar folder. I use CDH, so default path is /opt/cloudera/parcels/STREAMSETS_DATACOLLECTOR-3.5.2/libexec/bootstrap-libs/main/

  3. Replace old file:

    jar -uvf streamsets-datacollector-bootstrap-3.5.2.jar api-classloader.properties

  4. cp snappydata-jdbc_2.11-1.1.0.jar /opt/cloudera/parcels/STREAMSETS_DATACOLLECTOR-3.5.2/root-lib/

  5. Restart SDC

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-07-18 03:11:17 -0500

Seen: 107 times

Last updated: Jul 18