Ask Your Question
1

Client ran out of brokers to talk to

asked 2019-01-27 10:37:36 -0500

LArchinal gravatar image

updated 2019-01-28 09:51:40 -0500

metadaddy gravatar image

I have set up Zookeeper, Kafka, and Streamsets to all run locally on one machine. I am doing this because it is a test environment and I am limited in hardware. I have everything up and running on different ports: when I run: netstat -tlnp this is the output:

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN -
tcp6 0 0 :::35180 :::* LISTEN -
tcp6 0 0 :::111 :::* LISTEN -
tcp6 0 0 :::6000 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 ::1:631 :::* LISTEN -
tcp6 0 0 ::1:25 :::* LISTEN -
tcp6 0 0 127.0.0.1:9092 :::* LISTEN -
tcp6 0 0 :::39429 :::* LISTEN -
tcp6 0 0 :::2181 :::* LISTEN -
tcp6 0 0 :::18630 :::* LISTEN -

Now I have configured an Edge Pipeline:

Once the pipeline is built I go to verify the pipeline and it errors out with:

kafka: client ran out of brokers to talk to (Is your cluster reachable?)

I have found multiple articles that touch on the same error code but not relating to my situation.

edit retag flag offensive close merge delete

Comments

1

Is Data Collector Edge running on the same machine as Zookeeper, Kafka and Data Collector?

metadaddy gravatar imagemetadaddy ( 2019-01-28 09:52:35 -0500 )edit

Nope, the Data Collector Edge is on a virtual-machine that is within my lab network.

LArchinal gravatar imageLArchinal ( 2019-01-28 15:52:50 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-01-29 10:41:13 -0500

metadaddy gravatar image

The problem is that Kafka is listening on 127.0.0.1:9092, meaning that it will only accept requests on port 9092 from 127.0.0.1 (localhost), so it is not accessible to Data Collector Edge running on another machine. You need to configure Kafka to listen on another interface, such as 0.0.0.0, so that it is accessible from Edge.

Note - you may need to change more than one Kafka setting. See Kafka Listeners - Explained for a discussion of the various settings in Kafka.

edit flag offensive delete link more

Comments

Ok, so I made some configuration changes: listener=INTERNAL://0.0.0.0:9092 advertised.listerner=EXTERNAL://192.168.100.110:9092 advertised.listener=INTERNAL://localhost:90902 inter.broker.listener=INTERNAL.

LArchinal gravatar imageLArchinal ( 2019-01-30 09:37:01 -0500 )edit

After making these changes, I was able to verify the pipeline successfully, pushed it to the edge, and started running it. I can now see the data in StreamSets, which is what I wanted. But I am getting this error.

LArchinal gravatar imageLArchinal ( 2019-01-30 09:39:09 -0500 )edit

kafka: Failed to produce message to topic StreamSets: dial tcp [::1]:9092: connectex: No connection could be made because the target machine actively refused it.

LArchinal gravatar imageLArchinal ( 2019-01-30 09:39:16 -0500 )edit
1

Sounds like either another Kafka config issue, or a firewall/networking problem?

metadaddy gravatar imagemetadaddy ( 2019-01-30 10:59:54 -0500 )edit

Ok, so Kafka problem, not StreamSets?

LArchinal gravatar imageLArchinal ( 2019-01-30 11:04:27 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-01-27 10:37:36 -0500

Seen: 39 times

Last updated: Jan 29