Kafka Multi Topic consumer : KUDU_03 - Errors while interacting with Kudu can not complete before timeout

asked 2020-01-06 09:58:46 -0500

Suman gravatar image

updated 2020-01-07 13:35:21 -0500

metadaddy gravatar image

Hi, One of our pipeline with stage Kafka Multitopic Consumer subscribe to 18 Kafka topics schema Protrobuf from Kafka Cluster. We process data using Stage JavaScriptEvaluator_02 along side with customized java script, a Kudu Lookup to lookup for existing values by lastmodified timestamp and update if the data is older than the incoming message. finally the Kudu Target to write the data into tables. We are facing timeout as in below . Please let us know whats wrong here

KUDU Properties:

Stage Library: Apache Kudu 1.7.0

Pipeline Status: RUNNING_ERROR: com.streamsets.pipeline.api.StageException: KUDU_03 - Errors while interacting with Kudu: org.apache.kudu.client.NonRecoverableException: can not complete before timeout: ScanRequest(scannerId=null, tablet=54dd4e1fec7043c4a623b2ea67521cca, attempt=1, KuduRpc(method=Scan, tablet=54dd4e1fec7043c4a623b2ea67521cca, attempt=1, DeadlineTracker(timeout=10000, elapsed=20890), Traces: [0ms] sending RPC to server server1[20891ms] received from server server1 response Network error: [peer servcer2:7050)] encountered a read timeout; closing the channel, [20891ms] delaying RPC due to Network error: [peer servcer2:7050)] encountered a read timeout; closing the channel))



com.google.common.util.concurrent.UncheckedExecutionException: com.streamsets.pipeline.api.StageException: KUDU_03 - Errors while interacting with Kudu: org.apache.kudu.client.NonRecoverableException: can not complete before timeout: ScanRequest(scannerId=null, tablet=54dd4e1fec7043c4a623b2ea67521cca, attempt=1, KuduRpc(method=Scan, tablet=54dd4e1fec7043c4a623b2ea67521cca, attempt=1, DeadlineTracker(timeout=10000, elapsed=20890), Traces: [0ms] sending RPC to server server1,[20891ms] received from server server2, response Network error: [peer server2,(xxxx:7050)] encountered a read timeout; closing the channel, [20891ms] delaying RPC due to Network error: [peer server2,(xxxx:7050)] encountered a read timeout; closing the channel))
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.streamsets.pipeline.stage.processor.kudulookup.KuduLookupProcessor.process(KuduLookupProcessor.java:243)
    at com.streamsets.pipeline.api.base.SingleLaneRecordProcessor.process(SingleLaneRecordProcessor.java:53)
    at com.streamsets.pipeline.stage.processor.kudulookup.KuduLookupProcessor.process(KuduLookupProcessor.java:227)
    at com.streamsets.pipeline.api.base.SingleLaneProcessor.process(SingleLaneProcessor.java:95)
    at com.streamsets.pipeline.api.base.configurablestage.DProcessor.process(DProcessor.java:35)
    at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:299)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:244)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:311)
    at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:220)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:851)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.lambda$executeRunner$3(ProductionPipelineRunner.java:895)
    at com.streamsets.datacollector.runner.PipeRunner.acceptConsumer(PipeRunner.java:221)
    at com.streamsets.datacollector.runner.PipeRunner.executeBatch(PipeRunner.java:142)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.executeRunner(ProductionPipelineRunner.java:894)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runSourceLessBatch(ProductionPipelineRunner.java:872)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processBatch(ProductionPipelineRunner.java:507)
    at com.streamsets.datacollector.runner.StageRuntime$3.run(StageRuntime.java:383)
    at java.security ...
(more)
edit retag flag offensive close merge delete