streamsets datacollector 3.10.1 can access S3 bucket but 3.11 & 3.13 cannot

asked 2020-02-08 08:19:47 -0500

nkim9098 gravatar image

updated 2020-02-08 19:55:58 -0500

StreamSets Data Collector v3.10.1 can download files from an AWS S3 bucket. However, the newer versions of StreamSets Data Collector versions 3.11 and 3.13 produce the error "com.streamsets.pipeline.api.StageException: S3_SPOOLDIR_26 - S3 runner failed. Reason com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied".

All three instances of SDC are trying to download files from the same bucket using same access key and secret key pair. What would be causing the SDC v3.11 which is the production level not be able to access the AWS S3 bucket while SDC v3.10.1 can? I can also access the AWS S3 bucket using S3CMD tool from S3TOOLS.ORG as well just fine. (https://s3tools.org/s3cmd).

Here is the complete error message.

com.streamsets.pipeline.api.StageException: S3_SPOOLDIR_26 - S3 runner failed. Reason com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: BF34ECDA4A4B41FE; S3 Extended Request ID: psjLZrOdn1uxrMWDS51K4FxxxxxxKQ/1uNbkV4xABBL9PLe6JxxxxxxxxxxxWFHrE1vyiPVW0gA=), S3 Extended Request ID: psjLZrOdn1uxrMWDS51K4FxxxxxxKQ/1uNbkV4xABBL9PLe6JxxxxxxxxxxxWFHrE1vyiPVW0gA= com.streamsets.pipeline.api.StageException: S3_SPOOLDIR_26 - S3 runner failed. Reason com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: BF34ECDA4A4B41FE; S3 Extended Request ID: psjLZrOdn1uxrMWDS51K4FxxxxxxKQ/1uNbkV4xABBL9PLe6JxxxxxxxxxxxWFHrE1vyiPVW0gA=), S3 Extended Request ID: psjLZrOdn1uxrMWDS51K4FxxxxxxKQ/1uNbkV4xABBL9PLe6JxxxxxxxxxxxWFHrE1vyiPVW0gA= at com.streamsets.pipeline.stage.origin.s3.AbstractAmazonS3Source.produce(AbstractAmazonS3Source.java:118) at com.streamsets.pipeline.api.base.configurablestage.DPushSource.produce(DPushSource.java:44) at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$1(StageRuntime.java:270) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:244) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:279) at com.streamsets.datacollector.runner.SourcePipe.process(SourcePipe.java:79) at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.runPushSource(PreviewPipelineRunner.java:236) at com.streamsets.datacollector.runner.preview.PreviewPipelineRunner.run(PreviewPipelineRunner.java:221) at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:543) at com.streamsets.datacollector.runner.preview.PreviewPipeline.run(PreviewPipeline.java:51) at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:230) at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$1(AsyncPreviewer.java:98) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226) at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226) at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.amazonaws.services ... (more)

edit retag flag offensive close merge delete