Ask Your Question
1

Reading files from a DigitalOcean Bucket (AWS S3 equivalent)

asked 2019-09-24 09:24:19 -0500

Jose A. gravatar image

updated 2019-09-24 09:24:35 -0500

I'm getting this error connecting to a bucket in DigitalOcean (using Amazon s3 Origin).

I'm able to connect and list the files, but when I try to open the file to process, I get the next error

2019-09-24 14:01:23,962 [user:*jose] [pipeline:1/1d7e19afc-8589-45ff-a491-a7b80e4d5875] [runner:] [thread:preview-pool-1-thread-1] [stage:] INFO  Pipeline - Processing lifecycle start event with stage
2019-09-24 14:01:24,689 [user:*jose] [pipeline:1/1d7e19afc-8589-45ff-a491-a7b80e4d5875] [runner:] [thread:Amazon S3 Runner - 0] [stage:AmazonS3_01] ERROR AmazonS3Runnable - Error processing object with key 'test_file.txt' offset 'test_file.txt::0::5dd39cab1c53c2c77cd352983f9641e1::1569333431960'
com.amazonaws.SdkClientException: Unable to execute HTTP request: s3.other.amazonaws.com
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1116)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1066)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4368)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4315)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1416)
    at com.streamsets.pipeline.stage.origin.s3.AmazonS3Util.getObjectRange(AmazonS3Util.java:263)
    at com.streamsets.pipeline.stage.origin.s3.AmazonS3Runnable.handleWholeFileDataFormat(AmazonS3Runnable.java:325)
    at com.streamsets.pipeline.stage.origin.s3.AmazonS3Runnable.produce(AmazonS3Runnable.java:177)
    at com.streamsets.pipeline.stage.origin.s3.AmazonS3Runnable.run(AmazonS3Runnable.java:115)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeRunnable.run(SafeScheduledExecutorService.java:188)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeRunnable.run(SafeScheduledExecutorService.java:188)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: s3.other.amazonaws.com
    at java.net.InetAddress.getAllByName0(InetAddress ...
(more)
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-09-27 14:54:12 -0500

metadaddy gravatar image

I've been through the Data Collector source and there is no mention of s3.other.amazonaws.com. I'm wondering if it's a bug in the underlying AWS S3 client SDK. I see there's already an issue - thanks for commenting there. If you're a StreamSets customer, please file a support ticket and we can investigate more deeply.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-09-24 09:24:19 -0500

Seen: 350 times

Last updated: Sep 27