Custom Stage load class from $STREAMSETS_LIBRARIES_EXTRA_DIR ?

asked 2020-06-30 19:08:56 -0500

zac gravatar image

I have created a custom stage and I am trying to load a JDBC driver, but it is failing with a ClassNotFoundException from within the custom stage. (Exception is below) Can I configure a custom stage to use a JDBC driver that is available in the normal JDBC stages? The code I'm using to load the driver class is: Class.forName( "oracle.jdbc.driver.OracleDriver" ).newInstance();

Streamsets is installed via RPM

I believe I have the jdbc driver installed in the correct location because I can successfully use the JDBC Lookup and the JDBC Destination stages.

I have updated the /usr/lib/systemd/system/sdc.service to define the variables: USER_LIBRARIES_DIR and STREAMSETS_LIBRARIES_EXTRA_DIR.

I have placed the custom stage and all its jars in the USER_LIBRARIES_DIR directory.

I have placed the jdbc driver in the STREAMSETS_LIBRARIES_EXTRA_DIR.

I have updated the /etc/sdc/sdc-security.policy to grant permissions for the custom stage:

// custom stage library directory
grant codebase "file:///opt/streamsets-user-libs/-" {
    permission java.security.AllPermission;
};

When I place the jdbc driver in the custom stage directory the ClassNotFoundException does not occur.

Exception:

com.streamsets.pipeline.api.StageException: LRDT_DATE_LOOKUP_ERROR - Failed to lookup the last replication date time parameter.  Exception message: oracle.jdbc.driver.OracleDriver
    at com.sfr.ss.getupdated.UpdatedRecordsRetriever.lookupObjectLastReplicationDateTime(UpdatedRecordsRetriever.java:176)
    at com.sfr.ss.getupdated.UpdatedRecordsRetriever.retrieveRecords(UpdatedRecordsRetriever.java:81)
    at com.sfr.ss.getupdated.origin.GetUpdatedRecordsSource.produce(GetUpdatedRecordsSource.java:66)
    at com.streamsets.pipeline.api.base.configurablestage.DSource.produce(DSource.java:38)
    at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:296)
    at com.streamsets.pipeline.api.impl.CreateByRef.call(CreateByRef.java:40)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:244)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:311)
    at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:221)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:855)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:585)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:391)
    at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:520)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:112)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75)
    at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:726)
    at com.streamsets.datacollector.execution.runner.common.AsyncRunner.lambda$start$3(AsyncRunner.java:151)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:34)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:34)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at com.streamsets ...
(more)
edit retag flag offensive close merge delete