Ask Your Question
1

How to deduplicate kafka messages with SDC?

asked 2018-09-13 22:14:40 -0500

casel.chen gravatar image

Producer may send duplicate messages based on message key, how to deduplicate those kafka messages with SDC?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2018-09-14 10:07:48 -0500

metadaddy gravatar image

Use the Record Deduplicator processor - set Compare to 'Specified Fields' and set Fields to Compare to the message key. You can configure Max Records to Compare and Time to Compare as appropriate for your use case.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-09-13 22:14:40 -0500

Seen: 19 times

Last updated: Sep 14