Ask Your Question

How to make stream selector work according to a list which get from other source?

asked 2018-04-12 21:36:13 -0500

casel.chen gravatar image

updated 2018-04-12 22:20:23 -0500

I need to process kafka messages which contains userId field and want to do stream separation according to dynamic changed userId list from our application (jdbc or http client is ok) then finally sink those specific userId streams to another kafka topic. Note not all userId need to do stream separation because there are lots of userIds which wasting resources.

I tried to add another "http client" origin, but SDC complains already existed "kafka consumer" origin. Then I tried to use "jdbc lookup" processor, it can add more additional fields according to the existing record fields, but what I need is to add one field which has a list of values. How to achieve it by SDC normally? Thanks!

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-04-12 22:25:08 -0500

casel.chen gravatar image

A workaround way is concat list values into one string by jdbc lookup (select group_concat(id) from myTable) then process in javascript evaluator by splitting it and determine if the list values contain current id or not by add a new field "isContained", finally process the records in "stream selector" according to the new field "isContained" to do stream separation. image description

Any better way is welcome!

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-04-12 21:36:13 -0500

Seen: 482 times

Last updated: Apr 12 '18