Ask Your Question

Send pictures into Kafka?

asked 2017-11-01 12:49:14 -0600

julrojas gravatar image

updated 2017-11-01 22:34:00 -0600

metadaddy gravatar image

Hello, I would like to read pictures from a folder, send them into kafka and read them. I have not managed to make it work.

My trials: Binary (Either / or /fileRef) Kafka does not admit it: BINARY_GENERATOR_00 - Record 'resolution.png' cannot convert field path '/' value to byte[] SDC Record: It gets into kafka, but trying to read it produces: KAFKA_37 - Cannot parse record from message 'images::0::1': com.fasterxml.jackson.databind.JsonMappingException: Cannot convert java.util.LinkedHashMap to File Ref Object (through reference chain: com.streamsets.datacollector.restapi.bean.RecordJson["value"]) Tried also some groovy to transform the record into bytes[], but my code did not work.

Any advice on how to send pictures through kafka using streamsets?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2017-11-01 22:32:34 -0600

We don't currently support Kafka as a destination for writing Whole File format. See for a list of destinations that support this.

However, it's worth mentioning that typically it's not a good idea to use Kafka to move large files around anyway. Kafka has a default maximum message size of 1MB, and although that's configurable, in general you won't get great performance if you're using it to store large files. Best practice is to put the files themselves in some shared data storage location (perhaps HDFS), and pass the location of the files via Kafka.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-11-01 12:49:14 -0600

Seen: 1,837 times

Last updated: Nov 01 '17