Ask Your Question
1

Appear oom with MongoDB oplog as origin

asked 2019-04-27 23:52:28 -0500

hanruochen1 gravatar image

updated 2019-04-29 10:17:47 -0500

metadaddy gravatar image

I use the mongo oplog as the origin, the mongo cluster is the replica set mode, all the properties in the origin I set is default except the connection string, but when I run the pipeline, for a while the pipeline will be dead 。when I check the jvm dump file , find that there are a large amount of instances of
com.streamsets.pipeline.api.impl.ErrorMessage indicating that

com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 12003995670 not found on server *******:27017'
 at com.mongodb.operation.QueryHelper.translateCommandException(QueryHelper.java:27)
 at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:215)
 at com.mongodb.operation.QueryBatchCursor.tryHasNext(QueryBatchCursor.java:180)
 at com.mongodb.operation.QueryBatchCursor.tryNext(QueryBatchCursor.java:164)
 at com.mongodb.MongoBatchCursorAdapter.tryNext(MongoBatchCursorAdapter.java:65)
 at com.streamsets.pipeline.stage.origin.mongodb.oplog.MongoDBOplogSource.getOplogRecord(MongoDBOplogSource.java:241)
 at com.streamsets.pipeline.stage.origin.mongodb.oplog.MongoDBOplogSource.produce(MongoDBOplogSource.java:109)
 at com.streamsets.pipeline.api.base.configurablestage.DSource.produce(DSource.java:38)
 at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:283)
 at com.streamsets.pipeline.api.impl.CreateByRef.call(CreateByRef.java:40)
 at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:235)
 at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:298)
 at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:219)
 at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:810)
 at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:554)
 at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:383)
 at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:527)
 at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:109)
 at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75)
 at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:703)
 at com.streamsets.datacollector.execution.runner.common.AsyncRunner.lambda$start$3(AsyncRunner.java:151)
 at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
 at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
 at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
 at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
 at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
 at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

Is this the problem of our mongo cluster or my pipeline config ?

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted
0

answered 2019-04-29 10:21:44 -0500

metadaddy gravatar image

This can be caused by the cursor timing out in MongoDB. You can increase the cursor timeout interval at MongoDB, but a better solution is probably to reduce the batch size in your pipeline.

edit flag offensive delete link more

Comments

I tried to change batch size from the default value of 1000 to 30, still reporting this error.

hanruochen1 gravatar imagehanruochen1 ( 2019-04-30 03:30:06 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-04-27 23:52:28 -0500

Seen: 63 times

Last updated: Apr 29