kafka producer is failing to write message to kafka

asked 2019-07-25 12:50:50 -0500

spider8686 gravatar image

updated 2019-07-25 12:54:06 -0500

metadaddy gravatar image

am getting error meesage in pipleine when writing data to kafka , before this used to work.

com.streamsets.pipeline.api.StageException: KAFKA_50 - Error writing data to the Kafka broker: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request
    at com.streamsets.pipeline.kafka.impl.KafkaProducer09.createWriteException(KafkaProducer09.java:79)
    at com.streamsets.pipeline.kafka.impl.BaseKafkaProducer09.write(BaseKafkaProducer09.java:94)
    at com.streamsets.pipeline.stage.destination.kafka.KafkaTarget.writeOneMessagePerRecord(KafkaTarget.java:262)
    at com.streamsets.pipeline.stage.destination.kafka.KafkaTarget.write(KafkaTarget.java:83)
    at com.streamsets.pipeline.api.base.configurablestage.DTarget.write(DTarget.java:34)
    at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:290)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:235)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:298)
    at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:244)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:824)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.lambda$executeRunner$3(ProductionPipelineRunner.java:869)
    at com.streamsets.datacollector.runner.PipeRunner.executeBatch(PipeRunner.java:136)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.executeRunner(ProductionPipelineRunner.java:868)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runSourceLessBatch(ProductionPipelineRunner.java:846)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:583)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:397)
    at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:529)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:109)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75)
    at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:724)
    at com.streamsets.datacollector.execution.AbstractRunner.lambda$scheduleForRetries$0(AbstractRunner.java:357)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
edit retag flag offensive close merge delete

Comments

What version is the Kafka broker?

metadaddy gravatar imagemetadaddy ( 2019-07-25 12:54:42 -0500 )edit

confluent kafka 0.11

spider8686 gravatar imagespider8686 ( 2019-07-25 13:35:02 -0500 )edit

What have you selected as the stage library in the Kafka producer? You should install the Kafka 0.11 stage library if you have not already done so. Note - it is classed as a 'legacy' stage library - you will need to install it from package manager even if you did a full install of SDC.

metadaddy gravatar imagemetadaddy ( 2019-07-25 14:20:50 -0500 )edit

these are libraries i have kafka. Apache Kafka 0.11.0.0 streamsets-datacollector-apache-kafka_0_11-lib Apache Kafka 1.0.0 streamsets-datacollector-apache-kafka_1_0-lib Apache Kafka 1.1.0 streamsets-datacollector-apache-kafka_1_1-lib Apache Kafka 2.0.0 streamsets-datacollector-apache-ka

spider8686 gravatar imagespider8686 ( 2019-07-26 12:47:17 -0500 )edit