Kinesis Consumer Library- Checkpoint behaviour

asked 2019-02-13 17:35:39 -0500

pmandumula gravatar image

updated 2019-02-14 12:01:31 -0500

Hi-- When SDC instance terminated or restarted -KCL origin is not reading from last known checkpoint position - it is reading from initial position in the shard.Basically it is reading all records from the shard- this is causing duplicates. Please let us know any solution for this scenario.

Current Settings : TRIM_HORIZON when KCL applications stops and starts it should read from Last checkpoint location- but when it run for first time(after dynamodb table got created) it will read all records from shard.

Today->02-14-2019, we repeated the test again - by stopping the task definition and restart the flow- it looks dynamodb tables is been deleted and created again- according to documentation if we use same application name it should use the existing dynamodb table.

Can StreamSets explain about this behaviour and is there any way not to delete and recreate lease table(dynamodb) when new docker image comes up?

Thanks Praveen

edit retag flag offensive close merge delete