Ask Your Question
1

Send pictures into Kafka?

asked 2017-11-01 12:49:14 -0500

julrojas gravatar image

updated 2017-11-01 22:34:00 -0500

metadaddy gravatar image

Hello, I would like to read pictures from a folder, send them into kafka and read them. I have not managed to make it work.

My trials: Binary (Either / or /fileRef) Kafka does not admit it: BINARY_GENERATOR_00 - Record 'resolution.png' cannot convert field path '/' value to byte[] SDC Record: It gets into kafka, but trying to read it produces: KAFKA_37 - Cannot parse record from message 'images::0::1': com.fasterxml.jackson.databind.JsonMappingException: Cannot convert java.util.LinkedHashMap to File Ref Object (through reference chain: com.streamsets.datacollector.restapi.bean.RecordJson["value"]) Tried also some groovy to transform the record into bytes[], but my code did not work.

Any advice on how to send pictures through kafka using streamsets?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2017-11-01 22:32:34 -0500

We don't currently support Kafka as a destination for writing Whole File format. See https://streamsets.com/documentation/... for a list of destinations that support this.

However, it's worth mentioning that typically it's not a good idea to use Kafka to move large files around anyway. Kafka has a default maximum message size of 1MB, and although that's configurable, in general you won't get great performance if you're using it to store large files. Best practice is to put the files themselves in some shared data storage location (perhaps HDFS), and pass the location of the files via Kafka.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-11-01 12:49:14 -0500

Seen: 523 times

Last updated: Nov 01 '17