Ask Your Question
1

Null pointer exception when processing subdirectories

asked 2018-07-31 10:24:23 -0500

mstang gravatar image

updated 2018-07-31 13:44:28 -0500

How does "process subdirectories" work with a directory origin?

If I specify a directory my pipeline works, If I specify the parent directory and check "process subdirectories" I get a null pointer exception. The files and directories are all owned by the user that sdc is running as...

Any ideas on what could cause this?

I'm on version 3.1.3.0

2018-07-31 13:49:23,848 [user:*mike] [pipeline:PROD_AMCS_LOAD_TRICAPITALPROJECT_Contains/PRODAMCSLOADTRICAPITALPROJECTContains72e1d495-f22f-4278-b671-425b4f58f120] [runner:] [thread:preview-pool-1-thread-3] WARN  Pipeline - Stage 'Directory_01' initialization error: java.lang.NullPointerException
java.lang.NullPointerException
        at com.streamsets.pipeline.lib.dirspooler.DirectorySpooler.startSpooling(DirectorySpooler.java:421)
        at com.streamsets.pipeline.lib.dirspooler.DirectorySpooler.init(DirectorySpooler.java:400)
        at com.streamsets.pipeline.stage.origin.spooldir.SpoolDirSource.init(SpoolDirSource.java:215)
        at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48)
        at com.streamsets.pipeline.configurablestage.DStage.init(DStage.java:36)
        at com.streamsets.datacollector.runner.StageRuntime.lambda$init$0(StageRuntime.java:176)
        at com.streamsets.datacollector.util.LambdaUtil.withClassLoaderInternal(LambdaUtil.java:148)
        at com.streamsets.datacollector.util.LambdaUtil.withClassLoader(LambdaUtil.java:44)
        at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:174)
        at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:100)
        at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:48)
        at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:386)
        at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:299)
        at com.streamsets.datacollector.runner.preview.PreviewPipeline.run(PreviewPipeline.java:49)
        at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:214)
        at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$0(AsyncPreviewer.java:94)
        at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:227)
        at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
        at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:223)
        at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:227)
        at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
        at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:223)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-07-31 13:49:24,691 [user:mike] [pipeline:PROD_AMCS_LOAD_TRICAPITALPROJECT_Contains/PRODAMCSLOADTRICAPITALPROJECTContains72e1d495-f22f-4278-b671-425b4f58f120] [runner:] [thread:webserver-115435] WARN  StandaloneAndClusterPipelineManager - Evicting idle previewer 'PRODAMCSLOADTRICAPITALPROJECTContains72e1d495-f22f-4278-b671-425b4f58f120::0'::'4e1abe97-8816-4c54-9455-f7fefbfdd7ef' in status 'INVALID'
edit retag flag offensive close merge delete

Comments

Could you find the full stack trace in sdc.log (including any 'caused by' traces following the initial one) and edit it into your question?

metadaddy gravatar imagemetadaddy ( 2018-07-31 10:44:56 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-07-31 22:38:50 -0500

metadaddy gravatar image

It looks like you're running into something very similar to SDC-8882 - preview is timing out while the origin is compiling a list of files to process. If I'm correct, you should be able to work around this by increasing the preview timeout.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-07-31 10:24:23 -0500

Seen: 20 times

Last updated: Jul 31