The configuration = was supplied but isn't a known config?

asked 2018-06-19 16:56:36 -0600

jsphar
56 ●3 ●4 ●8

updated 2018-06-20 13:26:02 -0600

5464 ●26 ●41 ●82 https://about.me/patpa...

I have StreamSets up and running in a Docker Linux environment and I am able to connect to it and add in some processors. Currently I just want to learn what this software can do but I am unable to use the Kafka Consumer to receive any messages. I get the following errors :

The configuration = was supplied but isn't a known config
The configuration schema.registry.url = was supplied but isn't a known config

The pipeline continues to run but when I produce messages from the Kafka Producer on another machine they are not making it into the consumer in Stream sets. I know the port 9092 is open on the specific ip address but still wondering why I am seeing these errors? Could it be because I am running inside a Docker image? Could be an issue with the Kafka being on Ubuntu running in EC2?

I checked the sdc.log file and found the following error that might is present multiple times:

WARN BaseWebSocket - WebSocket 'status' error: java.net.SocketTimeoutException: Timeout on Read

In the UI Log: I get the following:

WARN - The configuration schema.registry.url = [] was supplied but isn't a known config. 
INFO - Successfully initialized Kafka Consumer 
WARN - WebSocket 'alerts' error: java.net.SocketTimeoutException: Timeout on Read

Not sure but I guess it might be having trouble actually connecting to the Kafka Topic?

Any suggestions would be greatly appreciated as I have tried the DNS name, Amazon EC2 cloud compute name and the public ip address all without success.

edit retag flag offensive close merge delete

Comments

Is there a stack trace in sdc.log, or in the log UI under the Administration (gear) icon?

metadaddy ( 2018-06-19 17:12:11 -0600 )edit

So I checked the sdc.log file and found the following error that might is present multiple times: WARN BaseWebSocket - WebSocket 'status' error: java.net.SocketTimeoutException: Timeout on Read Not sure but I guess it might be having trouble actually connecting to the Kafka Topic?

jsphar ( 2018-06-20 08:36:12 -0600 )edit

In the UI Log: I get the following: WARN - The configuration schema.registry.url = [] was supplied but isn't a known config. INFO - Successfully initialized Kafka Consumer WARN - WebSocket 'alerts' error: java.net.SocketTimeoutException: Timeout on Read

jsphar ( 2018-06-20 08:50:20 -0600 )edit

I think the overall problem is that I do not have my Docker image opened to outside networks. Does this seem logical? If so, is it possible to use stream sets in this manner? Thanks, Jason

jsphar ( 2018-06-20 13:05:56 -0600 )edit

add a comment

0

answered 2018-06-20 13:37:44 -0600

metadaddy

5464 ●26 ●41 ●82 https://about.me/patpa...

The most likely problem here is connectivity. You should ensure that the Kafka endpoint is accessible from within Data Collector's Docker container. Try

docker exec -it <container_id> bash

Then, from within the container:

nc -v -z -w2 <kafka_host> 9092

edit flag offensive delete link

Comments

So I was able to run those two commands (thanks by the way) and the response was "open". These were run from the Docker CLI. To be perfectly honest I am not sure why I am getting that response. Any ideas. Jason

jsphar ( 2018-06-20 15:33:51 -0600 )edit

add a comment

0

answered 2020-02-06 14:56:30 -0600

tlochner95
21 ●2 ●4 ●6

updated 2020-02-06 14:57:45 -0600

I wanted to point out that we were running into a similar issue. We were using AWS' MSK to spin up a Kafka instance, and are running our Data Collectors inside a Docker container inside of an EC2 instance. We also have another EC2 instance which is our "client machine" to talk to our MSK cluster (per AWS' docs). We were able to get a functional Kafka Producer to write to a topic (once we manually created our topic from our client machine EC2 instance). However, we were unable to get a functional Kafka Consumer working on our SDC. We were seeing similar errors to the original post.

Our solution:

Our solution was to make the security group of our MSK cluster the same as our client machine and our Data Collector's EC2 instance's security groups, then, add an inbound rule to that security group allowing "all traffic" from itself (the security group). This got our Kafka Consumer to see the topic and consume data from it.

Note: The link posted also suggests to open up the security group to all traffic from itself.

edit flag offensive delete link

add a comment

The configuration = was supplied but isn't a known config?

Comments

2 Answers

Comments

Question Tools

Stats

Related questions

The configuration = was supplied but isn't a known config? edit

Comments

2 Answers

Comments

Question Tools

Stats

Related questions

The configuration = was supplied but isn't a known config?