Ask Your Question
1

NPE While writing data to ES

asked 2020-01-07 09:22:38 -0600

Gk gravatar image

updated 2020-01-07 10:34:25 -0600

metadaddy gravatar image

My pipelines are failing once in a while due to null pointer exception while reading data from MySQL using JDBC consumer and getting restarted automatically in 2 to 5 minutes. Not able to identify the issue causing this based on the log, here's the complete log for your reference. Please let me know if you have any answers or suggestions.

java.lang.NullPointerException
    at com.streamsets.pipeline.stage.destination.elasticsearch.ElasticsearchTarget.write(ElasticsearchTarget.java:344)
    at com.streamsets.pipeline.api.base.configurablestage.DTarget.write(DTarget.java:34)
    at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:290)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:235)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:298)
    at com.streamsets.datacollector.runner.production.BadRecordsHandler.handle(BadRecordsHandler.java:75)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.executeRunner(ProductionPipelineRunner.java:858)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runSourceLessBatch(ProductionPipelineRunner.java:831)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:569)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:383)
    at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:527)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:109)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75)
    at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:703)
    at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.retryOrStart(StandaloneRunner.java:363)
    at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.onDataCollectorStart(StandaloneRunner.java:349)
    at com.streamsets.datacollector.execution.runner.common.AsyncRunner.lambda$onDataCollectorStart$0(AsyncRunner.java:110)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2020-01-07 10:39:48 -0600

metadaddy gravatar image

It's hard to see what the root cause is from the stack trace, but it looks like this is happening when error records are sent to Elasticsearch. If you look a little earlier in the log, you should see what the actual error was.

Please file an issue at https://issues.streamsets.com and include the stack trace above, any more context (other errors that occurred in the same timeframe), and the exported pipeline.

edit flag offensive delete link more

Comments

1

I configured my pipelines to write any error records to a separate index where I am able to index all the other related error without any issues. I am totally out of clue where this NPE is coming from with the amount of log level access I have. I will open an issue using the given link, Thank you.

Gk gravatar imageGk ( 2020-01-07 12:03:48 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-01-07 09:22:38 -0600

Seen: 16 times

Last updated: Jan 07