Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

How to make stream selector work according to a list which get from other source?

I need to process kafka messages which contains userId field and want to do stream separation according to dynamic changed userId list from our application (jdbc or http client is ok) then finally sink those specific userId streams to another kafka topic. Note not all userId need to do stream separation because there are lots of userIds which wasting resources.

How to achieve it by SDC? Thanks!

How to make stream selector work according to a list which get from other source?

I need to process kafka messages which contains userId field and want to do stream separation according to dynamic changed userId list from our application (jdbc or http client is ok) then finally sink those specific userId streams to another kafka topic. Note not all userId need to do stream separation because there are lots of userIds which wasting resources.

I tried to add another "http client" origin, but SDC complains already existed "kafka consumer" origin. Then I tried to use "jdbc lookup" processor, it can add more additional fields according to the existing record fields, but what I need is to add one field which has a list of values. A workaround way I can think is concating those values into one string return then process in javascript evaluator by splitting it.

How to achieve it by SDC? SDC normally? Thanks!

How to make stream selector work according to a list which get from other source?

I need to process kafka messages which contains userId field and want to do stream separation according to dynamic changed userId list from our application (jdbc or http client is ok) then finally sink those specific userId streams to another kafka topic. Note not all userId need to do stream separation because there are lots of userIds which wasting resources.

I tried to add another "http client" origin, but SDC complains already existed "kafka consumer" origin. Then I tried to use "jdbc lookup" processor, it can add more additional fields according to the existing record fields, but what I need is to add one field which has a list of values. A workaround way I can think is concating those values into one string return then process in javascript evaluator by splitting it.

How to achieve it by SDC normally? Thanks!

How to make stream selector work according to a list which get from other source?

I need to process kafka messages which contains userId field and want to do stream separation according to dynamic changed userId list from our application (jdbc or http client is ok) then finally sink those specific userId streams to another kafka topic. Note not all userId need to do stream separation because there are lots of userIds which wasting resources.

I tried to add another "http client" origin, but SDC complains already existed "kafka consumer" origin. Then I tried to use "jdbc lookup" processor, it can add more additional fields according to the existing record fields, but what I need is to add one field which has a list of values. How to achieve it by SDC normally? Thanks!