Ask Your Question

How to use dynamic topics in Kafka Consumer origin?

asked 2018-10-24 18:59:07 -0500

casel.chen gravatar image

I want to consume dynamic topics whose values can be retrieved by a RESTful API (check per 5 minutes is OK), but how can I set those topic values in SDC's kafka consumer origin? Thanks!

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-10-25 09:15:56 -0500

iamontheinet gravatar image

Hi Casel!

You should be able to load them in as runtime parameters, properties or resources.

Cheers, Dash

edit flag offensive delete link more


Thanks for your answer. My understanding those runtime parameters are static, and cannot dynamic load from external system like jdbc query or restful api call, am I right?

casel.chen gravatar imagecasel.chen ( 2018-10-25 09:54:55 -0500 )edit

Yes... but you can have two pipelines with one retrieving topics (via HTTP Client, JDBC, etc.) and writing them to resource file and a second pipeline that loads those topics from that resource file. You're right though that it's not truly dynamic but this might be your only option.

iamontheinet gravatar imageiamontheinet ( 2018-10-25 10:22:55 -0500 )edit

Cool, thanks for your suggestion! I will have a try.

casel.chen gravatar imagecasel.chen ( 2018-10-25 10:25:21 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-10-24 18:59:07 -0500

Seen: 468 times

Last updated: Oct 25 '18