Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

KCL Checkpoint behaviour

Hi-- When SDC instance terminated or restarted -KCL origin is not reading from last known checkpoint position - it is reading from initial position in the shard.Basically it is reading all records from the shard- this is causing duplicates.

Current Settings : TRIM_HORIZON when KCL applications stops and starts it should read from Last checkpoint location- but when it run for first time(after dynamodb table got created) it will read all records from shard.

Please let us know any solution for this scenario.

Thanks Praveen

KCL Checkpoint behaviour

Hi-- When SDC instance terminated or restarted -KCL origin is not reading from last known checkpoint position - it is reading from initial position in the shard.Basically it is reading all records from the shard- this is causing duplicates.

Current Settings : TRIM_HORIZON when KCL applications stops and starts it should read from Last checkpoint location- but when it run for first time(after dynamodb table got created) it will read all records from shard.

Please let us know any solution for this scenario.

Thanks Praveen

KCL Kinesis Consumer Library- Checkpoint behaviour

Hi-- When SDC instance terminated or restarted -KCL origin is not reading from last known checkpoint position - it is reading from initial position in the shard.Basically it is reading all records from the shard- this is causing duplicates.duplicates. Please let us know any solution for this scenario.

Current Settings : TRIM_HORIZON when KCL applications stops and starts it should read from Last checkpoint location- but when it run for first time(after dynamodb table got created) it will read all records from shard.

Please let us know Today->02-14-2019, we repeated the test again - by stopping the task definition and restart the flow- it looks dynamodb tables is been deleted and created again- according to documentation if we use same application name it should use the existing dynamodb table.

Can StreamSets explain about this behaviour and is there any solution for this scenario.way not to delete and recreate lease table(dynamodb) when new docker image comes up?

Thanks Praveen