Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Kafka Not able to Consume Data

Currently i having kafka cluster and streamsets managed by cloudera manager version 5.13.

Initially Kafka is Kafka 3.0, however, when configuring in streamsets and validate, the validation is successful. However, when start sending in data into topic, kafka consumer does not able to received any data. Row count remain as 0.

I suspected is library not supported due to the Kafka version and streamsets version is latest version. When running pipeline, no error message , but not able to consume data. When preview, this error message occur

com.streamsets.pipeline.api.StageException: KAFKA_30 - Error fetching offset from Kafka: java.nio.channels.ClosedChannelException

We try to downgrade our kafka version to 2.1.0 in CDH, however, still having the same issues like Kafka3.

  1. Cloudera manager Version - CDH 5.13.0
  2. Kafka Version - KAFKA-2.1.0-1.2.1.0.p0.115
  3. Streamsets Version - StreamSets Data Collector 3.0.0.0

How to further debug this?

Which library is more suitable to use for kafka cluster? CDH Kafka 2.1.1 or Apache Kafka 0.10.0.0