Multitable CDC to Kafka in AVRO format not possible?

asked 2018-12-10 11:38:41 -0600

pwel gravatar image

updated 2018-12-13 01:28:58 -0600


no answer so far, hence I simplifiy my question:

  • I capture records of 100 tables in one CDC stage (possible with the CDC stages)
  • I need to write the records to 100 corresponding Kafka Topics (possible since I can derive the kafka topic from each record-header)
  • I need to use the AVRO recordtype for the Kafka Topics (doesn't seem to be possible?)

It's a common task in CDC to capture schematized data of many tables and it's also common to use AVRO in kafka as destination. How can I do that with SDC?

If it is not possible, I'd like to suggest implement something like this:

  • Allow to derive/define the Schema ID / Subject dynamically per record
  • load the avro schema from the schema-registry when accessing a record of a certain schema for the first time
  • Cache this schema (for a certain, configurable time - e.g. 60 minutes) within the pipeline
  • use it with the according record

Thanks in advance and Regards, Peter

edit retag flag offensive close merge delete