Ask Your Question

Kafka consumer kerberos cluster mode HDP 2.5 does not work

asked 2018-01-13 14:03:59 -0600

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I'm designing a pipeline consuming data from a Kerberized Kafka cluster installed in an HDP cluster 2.5.3. The designed pipeline works fine in local mode. But if fails when it is deployed in cluster mode. I can't even validate and the error seems to be related to security. The error is:

KAFKA_41 - Could not get partition count for topic 'streamsets5' : com.streamsets.pipeline.api.StageException: KAFKA_41 - Could not get caseition count for topic 'streamsets5' : org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

What am I missing in my configuration ?

Thanks in advance for your help

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-05-22 15:39:58 -0600

junko_urata gravatar image

updated 2018-05-22 15:54:37 -0600

metadaddy gravatar image

Kerberized Kafka is not currently supported in cluster mode, but we will add the support in StreamSets Data Collector 3.3.0, which will be available soon (SDC-6857).

Please note that we will only support Spark 2.x from 3.3.0 and drop Spark 1.x and Kafka 0.9 or earlier. I believe Spark 2.x cannot be installed in HDP cluster 2.5.x, so you will need to switch to HDP 2.6 and install Spark 2.x.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-01-13 14:03:59 -0600

Seen: 836 times

Last updated: May 22 '18