Ask Your Question

How do you encode a Kafka Message Key as avro?

asked 2020-03-27 16:48:11 -0500

meash-nrel gravatar image

I switched Kafka Producer setting for Message Key Format to Avro on the Data Format tab. I need to utilize an Avro key as I cannot set the Key to null (entered as a separate question here. If a key is present, the Confluent REST API and the Confluent Python library expect the key to be Avro format when the value is Avro format.

There is an option to Kafka tab for Message Key Format to Confluent, in order to embed the schema subject ID, which signals that this stage integrates with Schema Registry and should embed the proper schema id when serializing the key as Avro. However, there is NO setting to provide the Subject or Schema ID, like you can with the value's Avro schema lookup. How does it get the schema ID then? This seems an oversight on Streamsets' part.

When I switch to Avro format, the stage embeds a default function sequence in the Message Key field on the Kafka tab: ${avro:decode(record:attribute('avroKeySchema'),base64:decodeBytes(record:attribute('kafkaMessageKey')))}

Avro functions (avro:*) are not documented at all in Functions docs. The Kafka Producer stage docs don't cover this at all, nor does the Avro Data Format doc page, which is solely focused on value encoding not key encoding.

Is this function sequence a work around for the fact I cannot set schema registry subject or schema id?

How does this mechanism work? Does it actually integrate with the Schema Registry?

It would be so much easier if Streamsets had native settings in the Kafka Producer stage handling this.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-03-28 11:28:44 -0500

J3 gravatar image


I am not sure if this will help because I am using Confluent Cloud, however, if you take a look here. I think adding the Kafka Configuration and leaving the Kafka Message Key blank will work for you.

---J3 Out!

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-03-27 16:48:11 -0500

Seen: 163 times

Last updated: Mar 28