Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Kafka Consumer

We are using streamsets pipeline by adding a Kafka Consumer as origin. The execution mode has been configure as Cluster Yarn Streaming and Kafka Consumer is using CDH 5.10.0 Kafka Cluster 2.1.0 Lib as the Stage Library. A kafka topic has been create with 7 partitions and 3 replicates.

Here are the issue we when start the pipeline:

  1. Pipiline's Kafka Consumer able to consume message at the beginning.
  2. After leaving the pipeline idle for some times (5 minutes), the kafka consumer not able to consume any new message.
  3. With Kafka-consumer-console, it show all the new message can be consume.

We checked the logs and found there is no errors returned. Just simply kafka is not consuming any of the data.

How we can troubleshoot on this issues?

Kafka Consumer

We are using streamsets pipeline by adding a Kafka Consumer as origin. The execution mode has been configure as Cluster Yarn Streaming and Kafka Consumer is using CDH 5.10.0 Kafka Cluster 2.1.0 Lib as the Stage Library. A kafka topic has been create with 7 partitions and 3 replicates.

Here are the issue we when start the pipeline:

  1. Pipiline's Kafka Consumer able to consume message at the beginning.
  2. After leaving the pipeline idle for some times (5 minutes), the kafka consumer not able to consume any new message.
  3. With Kafka-consumer-console, it show all the new message can be consume.

We checked the logs and found there is no errors returned. Just simply kafka is not consuming any of the data.

How we can troubleshoot on this issues?