Ask Your Question
0

Kafka Not able to Consume Data

asked 2018-01-05 04:42:19 -0600

Vivian Y gravatar image

Currently i having kafka cluster and streamsets managed by cloudera manager version 5.13.

Initially Kafka is Kafka 3.0, however, when configuring in streamsets and validate, the validation is successful. However, when start sending in data into topic, kafka consumer does not able to received any data. Row count remain as 0.

I suspected is library not supported due to the Kafka version and streamsets version is latest version. When running pipeline, no error message , but not able to consume data. When preview, this error message occur

com.streamsets.pipeline.api.StageException: KAFKA_30 - Error fetching offset from Kafka: java.nio.channels.ClosedChannelException

We try to downgrade our kafka version to 2.1.0 in CDH, however, still having the same issues like Kafka3.

  1. Cloudera manager Version - CDH 5.13.0
  2. Kafka Version - KAFKA-2.1.0-1.2.1.0.p0.115
  3. Streamsets Version - StreamSets Data Collector 3.0.0.0

How to further debug this?

Which library is more suitable to use for kafka cluster? CDH Kafka 2.1.1 or Apache Kafka 0.10.0.0

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-01-05 10:40:02 -0600

jeff gravatar image

Based on this matrix, it seems that Cloudera Kafka 3.0 corresponds to Apache Kafka 0.11. Support for that was added in SDC 3.0. Did you try using the Kafka 0.11 stage library?

edit flag offensive delete link more

Comments

For Kafka 3.0, we did try on Kafka 0.11 stage library as well, however , it does not succeed.

Vivian Y gravatar imageVivian Y ( 2018-01-08 01:57:45 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-01-05 04:42:19 -0600

Seen: 174 times

Last updated: Jan 05