Ask Your Question

Kafka Not able to Consume Data

asked 2018-01-05 04:42:19 -0600

Vivian Yang gravatar image

Currently i having kafka cluster and streamsets managed by cloudera manager version 5.13.

Initially Kafka is Kafka 3.0, however, when configuring in streamsets and validate, the validation is successful. However, when start sending in data into topic, kafka consumer does not able to received any data. Row count remain as 0.

I suspected is library not supported due to the Kafka version and streamsets version is latest version. When running pipeline, no error message , but not able to consume data. When preview, this error message occur

com.streamsets.pipeline.api.StageException: KAFKA_30 - Error fetching offset from Kafka: java.nio.channels.ClosedChannelException

We try to downgrade our kafka version to 2.1.0 in CDH, however, still having the same issues like Kafka3.

  1. Cloudera manager Version - CDH 5.13.0
  2. Kafka Version - KAFKA-2.1.0-
  3. Streamsets Version - StreamSets Data Collector

How to further debug this?

Which library is more suitable to use for kafka cluster? CDH Kafka 2.1.1 or Apache Kafka

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-01-05 10:40:02 -0600

jeff gravatar image

Based on this matrix, it seems that Cloudera Kafka 3.0 corresponds to Apache Kafka 0.11. Support for that was added in SDC 3.0. Did you try using the Kafka 0.11 stage library?

edit flag offensive delete link more


For Kafka 3.0, we did try on Kafka 0.11 stage library as well, however , it does not succeed.

Vivian Yang gravatar imageVivian Yang ( 2018-01-08 01:57:45 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-01-05 04:42:19 -0600

Seen: 50 times

Last updated: Jan 05