Ask Your Question
0

In Kafka Producer, can I use record attribute for schema subject?

asked 2020-07-24 05:58:28 -0500

paulu gravatar image

I would like to create a single pipeline that:

  1. Reads database changes using SQL Server Change Tracking origin
  2. For each table, writes change records to separate Kafka topic
    • Topic name: {table_name}
    • Data format: Avro
    • Schema location: Confluent Schema Registry
    • Schema lookup: By subject
    • Subject name: {table_name}-value

On the Kafka Producer destination I can select runtime topic selection and enter a topic expression, e.g. ${record:value('/Table')}.

I would like to do the same for the schema subject, e.g. ${str:concat(record:value('/Table'), '-value'}, but it will not allow the use of record function in the subject field. Is there another way to do this? There are many tables but would like to do it in single pipeline if possible.

Many thanks.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2020-07-24 13:30:23 -0500

iamontheinet gravatar image

Hi!

You're correct, EL for that particular attribute is not supported. You may file Jira for it https://issues.streamsets.com/. And I can't think of another way of doing it in the same pipeline.

Cheers, Dash

edit flag offensive delete link more

Comments

1

OK. Thanks Dash. Jira ticket raised. https://issues.streamsets.com/browse/SDC-15203

paulu gravatar imagepaulu ( 2020-07-26 10:44:57 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-07-24 05:58:28 -0500

Seen: 114 times

Last updated: Jul 26