Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

How to use other JDBC driver for JDBC Producer?

I know Mysql/Oracle/SQLServer/PG driver can be used in JDBC Producer.

Now I want to use Snappydata(something like Memsql) driver in this destination.

So I:

  1. put snappydata-jdbc_2.11-1.1.0.jar in /opt/sdc-extras/streamsets-datacollector-jdbc-lib/lib
  2. set sdc-env.sh and sdc-security.policy
  3. restart

When I preview the pipeline, I got below message:

2019-07-18 15:07:57,773 snappydata-test/snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e INFO Processing lifecycle start event with stage Pipeline *admin preview-pool-1-thread-1 2019-07-18 15:07:57,837 snappydata-test/snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e INFO Registered JDBC drivers: JdbcUtil *admin preview-pool-1-thread-1 2019-07-18 15:07:57,873 snappydata-test/snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e INFO Destroying pipeline with reason=FINISHED Pipeline *admin preview-pool-1-thread-1 2019-07-18 15:07:57,899 snappydata-test/snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e INFO Processing lifecycle stop event Pipeline *admin preview-pool-1-thread-1 2019-07-18 15:07:57,900 snappydata-test/snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e INFO Pipeline finished destroying with final reason=FINISHED Pipeline *admin preview-pool-1-thread-1 2019-07-18 15:07:57,906 snappydata-test/snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e ERROR Uncaught throwable from com.streamsets.datacollector.execution.preview.async.AsyncPreviewer$$Lambda$111/942442767@116a3e9e: com.streamsets.datacollector.util.PipelineException: PREVIEW_0003 - Encountered error while previewing : java.lang.NoClassDefFoundError: Could not initialize class io.snappydata.jdbc.ClientDriver SafeScheduledExecutorService *admin preview-pool-1-thread-1 com.streamsets.datacollector.util.PipelineException: PREVIEW_0003 - Encountered error while previewing : java.lang.NoClassDefFoundError: Could not initialize class io.snappydata.jdbc.ClientDriver at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:248) at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$0(AsyncPreviewer.java:95) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226) at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226) at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoClassDefFoundError: Could not initialize class io.snappydata.jdbc.ClientDriver at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at java.sql.DriverManager.isDriverAllowed(DriverManager.java:556) at java.sql.DriverManager.isDriverAllowed(DriverManager.java:548) at java.sql.DriverManager.getDrivers(DriverManager.java:446) at com.streamsets.pipeline.lib.jdbc.JdbcUtil.createDataSourceConfig(JdbcUtil.java:708) at com.streamsets.pipeline.lib.jdbc.JdbcUtil.createDataSourceForRead(JdbcUtil.java:793) at com.streamsets.pipeline.stage.origin.jdbc.JdbcSource.init(JdbcSource.java:220) at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48) at com.streamsets.pipeline.api.base.configurablestage.DStage.init(DStage.java:36) at com.streamsets.datacollector.runner.StageRuntime.lambda$init$0(StageRuntime.java:211) at com.streamsets.datacollector.util.LambdaUtil.withClassLoaderInternal(LambdaUtil.java:148) at com.streamsets.datacollector.util.LambdaUtil.withClassLoader(LambdaUtil.java:44) at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:209) at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:123) at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:47) at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:403) at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:312) at com.streamsets.datacollector.runner.preview.PreviewPipeline.run(PreviewPipeline.java:49) at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:217) ... 14 more 2019-07-18 15:07:58,831 snappydata-test/snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e WARN Evicting idle previewer 'snappydatatest35afcd31-9b37-46fe-8790-aa70c036c81e::0'::'5b12bfa3-7985-4f45-adc6-0e45ed9155bd' in status 'RUN_ERROR' StandaloneAndClusterPipelineManager admin webserver-20

I have tried:

  1. add "-cp /opt/sdc-extras/streamsets-datacollector-jdbc-lib/lib/snappydata-jdbc_2.11-1.1.0.jar" in java.opts, no use.
  2. add "com.streamsets.pipeline.stage.jdbc.drivers.load=io.snappydata.jdbc.ClientDriver" in file sdc.properties, no use.