Ask Your Question

Revision history [back]

We don't currently support Kafka as a destination for writing Whole File format. See https://streamsets.com/documentation/datacollector/2.7.1.1/help/index.html#Data_Formats/WholeFile.html#concept_nfc_qkh_xw for a list of destinations that support this.

However, it's worth mentioning that typically it's not a good idea to use Kafka to move large files around anyway. Kafka has a default maximum message size of 1MB, and although that's configurable, in general you won't get great performance if you're using it to store large files. Best practice is to put the files themselves in some shared data storage location (perhaps HDFS), and pass the location of the files via Kafka.