Ask Your Question
1

how to provide keys to encryption stage?

asked 2019-05-15 01:06:08 -0600

mani gravatar image

updated 2019-05-16 10:38:25 -0600

metadaddy gravatar image

I am trying to pull date of birth from JSON and using encrypt/decrypt stage I wanted to push to Elasticsearch. I pass the secret key id, secret key and KMS ARN. When I try to validate the pipeline, I get following error:

 WARN   Stage 'EncryptandDecryptFields_01' initialization error: com.amazonaws.encryptionsdk.exception.AwsCryptoException: Can't use non-ARN key identifiers or aliases when no default region is set

com.amazonaws.encryptionsdk.exception.AwsCryptoException: Can't use non-ARN key identifiers or aliases when no default region is set
    at com.amazonaws.encryptionsdk.kms.KmsMasterKeyProvider$Builder.build(KmsMasterKeyProvider.java:250)
    at com.streamsets.pipeline.stage.processor.crypto.FieldEncryptProcessor.createCryptoMaterialsManager(FieldEncryptProcessor.java:142)
    at com.streamsets.pipeline.stage.processor.crypto.FieldEncryptProcessor.createProvider(FieldEncryptProcessor.java:112)
    at com.streamsets.pipeline.stage.processor.crypto.FieldEncryptProcessor.init(FieldEncryptProcessor.java:90)
    at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48)
    at com.streamsets.pipeline.api.base.configurablestage.DStage.init(DStage.java:36)
    at com.streamsets.datacollector.runner.StageRuntime.lambda$init$0(StageRuntime.java:211)
    at com.streamsets.datacollector.util.LambdaUtil.withClassLoaderInternal(LambdaUtil.java:148)
    at com.streamsets.datacollector.util.LambdaUtil.withClassLoader(LambdaUtil.java:44)
    at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:209)
    at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:123)
    at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:47)
    at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:403)
    at com.streamsets.datacollector.runner.Pipeline.lambda$init$0(Pipeline.java:393)
    at com.streamsets.datacollector.runner.PipeRunner.forEach(PipeRunner.java:166)
    at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:391)
    at com.streamsets.datacollector.runner.Pipeline.validateConfigs(Pipeline.java:219)
    at com.streamsets.datacollector.runner.preview.PreviewPipeline.validateConfigs(PreviewPipeline.java:60)
    at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.validateConfigs(SyncPreviewer.java:142)
    at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer$1.call(AsyncPreviewer.java:70)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-05-16 03:42:52 -0600

mani gravatar image

I got it, it was an initialization error. I asked my AWS admin to create new KMS and initialize it. He also provided new secret ids. It worked.

Thanks

edit flag offensive delete link more

Comments

Glad you got it working, and thanks for coming back to answer your own question!

metadaddy gravatar imagemetadaddy ( 2019-05-16 10:38:56 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-05-15 01:06:08 -0600

Seen: 70 times

Last updated: May 16