Ask Your Question
1

Not able to push data from large .xml files present on an SFTP location to Kafka Topic

asked 2017-09-18 05:30:55 -0600

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I have an SFTP location and I have .xml files from which I want to transfer data to Kafka topics using StreamSets. For small .xml files, I'm able to push the data into Kafka, but as the size increases, I am not able to push the data anymore. What might be the issue?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2017-09-18 10:30:19 -0600

jeff gravatar image

There could be a few different settings at play. First, check the max record size on your origin (in the Data Format section) and ensure it's large enough to represent any record it may encounter. Also, on the Kafka side, you may need to tweak max.request.size (see documentation here) if the default is not large enough to contain an entire batch of your records in a single produce call.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-09-18 05:30:55 -0600

Seen: 761 times

Last updated: Sep 18 '17