Ask Your Question

How to use schema registry with Kafka in StreamSets? [closed]

asked 2017-09-14 01:17:05 -0500

casel.chen gravatar image

updated 2017-09-20 13:05:02 -0500

LC gravatar image

Is there any tutorial or example on how to use schema registry with kafka in streamsets DC? Can it work with JDBC Consumer and MySQL Binlog to capture schema changes?

edit retag flag offensive reopen merge delete

Closed for the following reason the question is answered, right answer was accepted by casel.chen
close date 2017-11-08 03:56:28.909331

2 Answers

Sort by ยป oldest newest most voted

answered 2017-09-20 12:36:04 -0500

adam gravatar image

I don't think I've seen a tutorial yet. For now you'll have to refer to the documentation included in the Kafka stages for how to use it with Kafka.

It supports looking up the schema from the registry in the Kafka Consumer based on the embedded Schema ID as well as overriding it manually.

Typical usage would be to set the Key Deserializer and Message Deserializer to "Confluent" on the Kafka tab, and then in the Data Format tab choosing Avro and filling out the Schema Registry URL and using "Schema In Message/Data"

If you are looking to capture drift from SQL origins you can use the Avro Schema Generator Processor and then when sending data to your destination choose the Avro Data Format and configure the Schema Registry URL, set the schema to "In Record Header" and "Register Schema". You'll also need to specify the registry Subject for the schema.

This will register the updated schema with the registry (if a compatible change) or will error if incompatible.

Hope this helps to get you started.

edit flag offensive delete link more

answered 2017-11-08 03:55:57 -0500

casel.chen gravatar image

Here is the related tutorial:

edit flag offensive delete link more

Question Tools

1 follower


Asked: 2017-09-14 01:17:05 -0500

Seen: 367 times

Last updated: Nov 08 '17