Ask Your Question
1

NPE While writing data to ES

asked 2020-01-07 09:22:38 -0500

Gk gravatar image

updated 2020-01-28 10:06:46 -0500

metadaddy gravatar image

My pipelines are failing once in a while due to null pointer exception while reading data from MySQL using JDBC consumer and getting restarted automatically in 2 to 5 minutes. Not able to identify the issue causing this based on the log, here's the complete log for your reference. Please let me know if you have any answers or suggestions.

java.lang.NullPointerException
    at com.streamsets.pipeline.stage.destination.elasticsearch.ElasticsearchTarget.write(ElasticsearchTarget.java:344)
    at com.streamsets.pipeline.api.base.configurablestage.DTarget.write(DTarget.java:34)
    at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:290)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:235)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:298)
    at com.streamsets.datacollector.runner.production.BadRecordsHandler.handle(BadRecordsHandler.java:75)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.executeRunner(ProductionPipelineRunner.java:858)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runSourceLessBatch(ProductionPipelineRunner.java:831)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:569)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:383)
    at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:527)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:109)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75)
    at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:703)
    at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.retryOrStart(StandaloneRunner.java:363)
    at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.onDataCollectorStart(StandaloneRunner.java:349)
    at com.streamsets.datacollector.execution.runner.common.AsyncRunner.lambda$onDataCollectorStart$0(AsyncRunner.java:110)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

If I look earlier in the log using the debug level mode logging, this is what I am seeing.

2020-01-24 15:41:12,696 Orders_Modified_v2/OrdersModifiedv2ef7ef2a2-c718-45f5-9fe5-e9ddb50e590c DEBUG   ResourceBundle 'com.streamsets.pipeline.stage.config.elasticsearch.Errors-bundle' not found via ClassLoader 'SDCClassLoader[type=streamsets-libs name=streamsets-datacollector-elasticsearch_5-lib private=false]'  LocalizableMessage  admin   0   ProductionPipelineRunnable-OrdersModifiedv2ef7ef2a2-c718-45f5-9fe5-e9ddb50e590c-Orders_Modified_v2
2020-01-24 15:41:23,321 Orders_Modified_v2/OrdersModifiedv2ef7ef2a2-c718-45f5-9fe5-e9ddb50e590c ERROR   Pipeline execution failed   ProductionPipelineRunner    admin       ProductionPipelineRunnable-OrdersModifiedv2ef7ef2a2-c718-45f5-9fe5-e9ddb50e590c-Orders_Modified_v2
java.lang.NullPointerException
    at com.streamsets.pipeline.stage.destination.elasticsearch.ElasticsearchTarget.write(ElasticsearchTarget.java:344)
    at com.streamsets.pipeline.api.base.configurablestage.DTarget.write(DTarget.java:34)
    at com.streamsets.datacollector.runner.StageRuntime ...
(more)
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2020-01-07 10:39:48 -0500

metadaddy gravatar image

It's hard to see what the root cause is from the stack trace, but it looks like this is happening when error records are sent to Elasticsearch. If you look a little earlier in the log, you should see what the actual error was.

Please file an issue at https://issues.streamsets.com and include the stack trace above, any more context (other errors that occurred in the same timeframe), and the exported pipeline.

edit flag offensive delete link more

Comments

1

I configured my pipelines to write any error records to a separate index where I am able to index all the other related error without any issues. I am totally out of clue where this NPE is coming from with the amount of log level access I have. I will open an issue using the given link, Thank you.

Gk gravatar imageGk ( 2020-01-07 12:03:48 -0500 )edit

I added your other stack trace to your question. I can't see what the issue might be. If you're a StreamSets customer, please open a support ticket and we can collect more information and get to the bottom of it.

metadaddy gravatar imagemetadaddy ( 2020-01-28 10:08:14 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-01-07 09:22:38 -0500

Seen: 78 times

Last updated: Jan 28