Ask Your Question

How to deduplicate kafka messages with SDC?

asked 2018-09-13 22:14:40 -0600

casel.chen gravatar image

Producer may send duplicate messages based on message key, how to deduplicate those kafka messages with SDC?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-09-14 10:07:48 -0600

metadaddy gravatar image

Use the Record Deduplicator processor - set Compare to 'Specified Fields' and set Fields to Compare to the message key. You can configure Max Records to Compare and Time to Compare as appropriate for your use case.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-09-13 22:14:40 -0600

Seen: 363 times

Last updated: Sep 14 '18