Ask Your Question
1

open source sdc on kubernetes is failing.

asked 2019-07-23 00:27:59 -0600

eyesmoker gravatar image

updated 2019-07-23 09:06:57 -0600

metadaddy gravatar image

Hi I am trying option 1 (sdc as stateful kubernetes) from https://streamsets.com/blog/scaling-s... on AWS EKS and it is failing with following exception. Please help.

kubectl logs datacollector-0
Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTS
Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.streamsets.pipeline.BootstrapMain.bootstrap(BootstrapMain.java:351)
        at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:110)
Caused by: java.lang.RuntimeException: Could not create SDC ID file '/data/sdc.id': java.io.FileNotFoundException: /data/sdc.id (Permission denied)
        at com.streamsets.datacollector.main.StandaloneRuntimeInfo.getSdcId(StandaloneRuntimeInfo.java:82)
        at com.streamsets.datacollector.main.StandaloneRuntimeInfo.init(StandaloneRuntimeInfo.java:48)
        at com.streamsets.datacollector.main.RuntimeModule.provideRuntimeInfo(RuntimeModule.java:60)
        at com.streamsets.datacollector.main.RuntimeModule$$ModuleAdapter$ProvideRuntimeInfoProvidesAdapter.get(RuntimeModule$$ModuleAdapter.java:116)
        at com.streamsets.datacollector.main.RuntimeModule$$ModuleAdapter$ProvideRuntimeInfoProvidesAdapter.get(RuntimeModule$$ModuleAdapter.java:80)
        at dagger.internal.Linker$SingletonBinding.get(Linker.java:364)
        at com.streamsets.datacollector.execution.manager.standalone.StandaloneAndClusterPipelineManager$$InjectAdapter.injectMembers(StandaloneAndClusterPipelineManager$$InjectAdapter.java:94)
        at com.streamsets.datacollector.execution.manager.standalone.StandaloneAndClusterPipelineManager$$InjectAdapter.injectMembers(StandaloneAndClusterPipelineManager$$InjectAdapter.java:22)
        at dagger.ObjectGraph$DaggerObjectGraph.inject(ObjectGraph.java:281)
        at com.streamsets.datacollector.execution.manager.standalone.StandaloneAndClusterPipelineManager.<init>(StandaloneAndClusterPipelineManager.java:107)
        at com.streamsets.datacollector.main.MainStandalonePipelineManagerModule.<init>(MainStandalonePipelineManagerModule.java:49)
        at com.streamsets.datacollector.main.MainStandalonePipelineManagerModule$$ModuleAdapter.newModule(MainStandalonePipelineManagerModule$$ModuleAdapter.java:27)
        at com.streamsets.datacollector.main.MainStandalonePipelineManagerModule$$ModuleAdapter.newModule(MainStandalonePipelineManagerModule$$ModuleAdapter.java:16)
        at dagger.internal.Modules.loadModules(Modules.java:41)
        at dagger.ObjectGraph$DaggerObjectGraph.makeGraph(ObjectGraph.java:174)
        at dagger.ObjectGraph$DaggerObjectGraph.access$000(ObjectGraph.java:138)
        at dagger.ObjectGraph.create(ObjectGraph.java:129)
        at com.streamsets.datacollector.main.Main.<init>(Main.java:49)
        at com.streamsets.datacollector.main.DataCollectorMain.<init>(DataCollectorMain.java:28)
        at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:53)
        ... 6 more
Caused by: java.io.FileNotFoundException: /data/sdc.id (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at com.google.common.io.Files$FileByteSink.openStream(Files.java:200)
        at com.google.common.io.Files$FileByteSink.openStream(Files.java:188)
        at com.google.common.io.ByteSink$AsCharSink.openStream(ByteSink.java:148)
        at com.google.common.io.CharSink.write(CharSink.java:95)
        at com.google.common.io.Files.write(Files.java:320)
        at com.streamsets.datacollector.main.StandaloneRuntimeInfo.getSdcId(StandaloneRuntimeInfo.java:80)
        ... 25 more
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-07-23 09:12:15 -0600

metadaddy gravatar image

Looking at the two exceptions:

  • Caused by: java.lang.RuntimeException: Could not create SDC ID file '/data/sdc.id': java.io.FileNotFoundException: /data/sdc.id (Permission denied)
  • Caused by: java.io.FileNotFoundException: /data/sdc.id (Permission denied)

Data Collector is trying to create its sdc.id file, in the /data directory, but it can't write to the that directory. You should take a look at the permissions there.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-07-23 00:26:22 -0600

Seen: 93 times

Last updated: Jul 23