Ask Your Question
0

Kafkaconsumer to JDBC Postgresql cannot send data

asked 2020-02-24 05:36:12 -0500

anonymous user

Anonymous

updated 2020-02-24 14:37:14 -0500

Hi,

I have a pipeline that sends data from kafka consumer to postgresql database. But it seems that these days the pipeline does not send any data tot he azure postgresql database. The main scenario is to get data from newly added sensor device and subscribe the data via kafka consumer and send it to the database (azure postgresql). I have a service that gets the data from the azure blob storage and subscribes it as a topic via a consumer. My kafka consumer has configuration as follows:

  • broker URI: localhost:9092
  • Zookeeper: localhost 9092
  • Max Batch Size (records): 25000000 (it exceeds the max range)

the pipeline is only for inserting data via kafka consumer to the database.

The pipeline looks as follows:

kafkaConsumer --> Javascript Evaluator--->JDBC Postgresql Producer.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2020-02-24 16:46:58 -0500

metadaddy gravatar image

The most likely problem here is that the Zookeeper port is wrong. The default Zookeeper port is 2181. It would never be the same as the Kafka broker port.

edit flag offensive delete link more

Comments

Hi, I changed the zookeeper port but still it is not working. Any idea? is it also related to Max Batch Size (records)?

rowshan.sathi@linde.com gravatar imagerowshan.sathi@linde.com ( 2020-03-04 03:24:54 -0500 )edit

Max batch size won't make any difference unless you also changed the limit in sdc.properties. Not sure why you raised it to 25000000 - that would require a lot of memory, since the entire batch must fit in memory.

metadaddy gravatar imagemetadaddy ( 2020-03-04 19:31:33 -0500 )edit

Are you seeing errors? Did this pipeline work in the past?

metadaddy gravatar imagemetadaddy ( 2020-03-04 19:32:10 -0500 )edit

Hi Yes it used to work before perfectly and passed data to the azure postgresql database perfectly. Any Idea. How to solve this problem. Thanks in advance.

rowshan.sathi@linde.com gravatar imagerowshan.sathi@linde.com ( 2020-05-11 03:03:24 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-02-24 05:35:23 -0500

Seen: 74 times

Last updated: Feb 24