Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

open source sdc on kubernetes is failing.

Hi I am trying option 1 (sdc as stateful kubernetes) from https://streamsets.com/blog/scaling-streamsets-kubernetes/ on AWS Eks and it is failing with following exception. Please help.

`kubectl logs datacollector-0 Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTS Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.streamsets.pipeline.BootstrapMain.bootstrap(BootstrapMain.java:351) at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:110) Caused by: java.lang.RuntimeException: Could not create SDC ID file '/data/sdc.id': java.io.FileNotFoundException: /data/sdc.id (Permission denied) at com.streamsets.datacollector.main.StandaloneRuntimeInfo.getSdcId(StandaloneRuntimeInfo.java:82) at com.streamsets.datacollector.main.StandaloneRuntimeInfo.init(StandaloneRuntimeInfo.java:48) at com.streamsets.datacollector.main.RuntimeModule.provideRuntimeInfo(RuntimeModule.java:60) at com.streamsets.datacollector.main.RuntimeModule$$ModuleAdapter$ProvideRuntimeInfoProvidesAdapter.get(RuntimeModule$$ModuleAdapter.java:116) at com.streamsets.datacollector.main.RuntimeModule$$ModuleAdapter$ProvideRuntimeInfoProvidesAdapter.get(RuntimeModule$$ModuleAdapter.java:80) at dagger.internal.Linker$SingletonBinding.get(Linker.java:364) at com.streamsets.datacollector.execution.manager.standalone.StandaloneAndClusterPipelineManager$$InjectAdapter.injectMembers(StandaloneAndClusterPipelineManager$$InjectAdapter.java:94) at com.streamsets.datacollector.execution.manager.standalone.StandaloneAndClusterPipelineManager$$InjectAdapter.injectMembers(StandaloneAndClusterPipelineManager$$InjectAdapter.java:22) at dagger.ObjectGraph$DaggerObjectGraph.inject(ObjectGraph.java:281) at com.streamsets.datacollector.execution.manager.standalone.StandaloneAndClusterPipelineManager.<init>(StandaloneAndClusterPipelineManager.java:107) at com.streamsets.datacollector.main.MainStandalonePipelineManagerModule.<init>(MainStandalonePipelineManagerModule.java:49) at com.streamsets.datacollector.main.MainStandalonePipelineManagerModule$$ModuleAdapter.newModule(MainStandalonePipelineManagerModule$$ModuleAdapter.java:27) at com.streamsets.datacollector.main.MainStandalonePipelineManagerModule$$ModuleAdapter.newModule(MainStandalonePipelineManagerModule$$ModuleAdapter.java:16) at dagger.internal.Modules.loadModules(Modules.java:41) at dagger.ObjectGraph$DaggerObjectGraph.makeGraph(ObjectGraph.java:174) at dagger.ObjectGraph$DaggerObjectGraph.access$000(ObjectGraph.java:138) at dagger.ObjectGraph.create(ObjectGraph.java:129) at com.streamsets.datacollector.main.Main.<init>(Main.java:49) at com.streamsets.datacollector.main.DataCollectorMain.<init>(DataCollectorMain.java:28) at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:53) ... 6 more Caused by: java.io.FileNotFoundException: /data/sdc.id (Permission denied) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at com.google.common.io.Files$FileByteSink.openStream(Files.java:200) at com.google.common.io.Files$FileByteSink.openStream(Files.java:188) at com.google.common.io.ByteSink$AsCharSink.openStream(ByteSink.java:148) at com.google.common.io.CharSink.write(CharSink.java:95) at com.google.common.io.Files.write(Files.java:320) at com.streamsets.datacollector.main.StandaloneRuntimeInfo.getSdcId(StandaloneRuntimeInfo.java:80) ... 25 more

`

open source sdc on kubernetes is failing.

Hi I am trying option 1 (sdc as stateful kubernetes) from https://streamsets.com/blog/scaling-streamsets-kubernetes/ on AWS Eks EKS and it is failing with following exception. Please help.

`kubectl

kubectl logs datacollector-0
Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTS
Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.streamsets.pipeline.BootstrapMain.bootstrap(BootstrapMain.java:351)
        at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:110)
Caused by: java.lang.RuntimeException: Could not create SDC ID file '/data/sdc.id': java.io.FileNotFoundException: /data/sdc.id (Permission denied)
        at com.streamsets.datacollector.main.StandaloneRuntimeInfo.getSdcId(StandaloneRuntimeInfo.java:82)
        at com.streamsets.datacollector.main.StandaloneRuntimeInfo.init(StandaloneRuntimeInfo.java:48)
        at com.streamsets.datacollector.main.RuntimeModule.provideRuntimeInfo(RuntimeModule.java:60)
        at com.streamsets.datacollector.main.RuntimeModule$$ModuleAdapter$ProvideRuntimeInfoProvidesAdapter.get(RuntimeModule$$ModuleAdapter.java:116)
        at com.streamsets.datacollector.main.RuntimeModule$$ModuleAdapter$ProvideRuntimeInfoProvidesAdapter.get(RuntimeModule$$ModuleAdapter.java:80)
        at dagger.internal.Linker$SingletonBinding.get(Linker.java:364)
        at com.streamsets.datacollector.execution.manager.standalone.StandaloneAndClusterPipelineManager$$InjectAdapter.injectMembers(StandaloneAndClusterPipelineManager$$InjectAdapter.java:94)
        at com.streamsets.datacollector.execution.manager.standalone.StandaloneAndClusterPipelineManager$$InjectAdapter.injectMembers(StandaloneAndClusterPipelineManager$$InjectAdapter.java:22)
        at dagger.ObjectGraph$DaggerObjectGraph.inject(ObjectGraph.java:281)
        at com.streamsets.datacollector.execution.manager.standalone.StandaloneAndClusterPipelineManager.<init>(StandaloneAndClusterPipelineManager.java:107)
        at com.streamsets.datacollector.main.MainStandalonePipelineManagerModule.<init>(MainStandalonePipelineManagerModule.java:49)
        at com.streamsets.datacollector.main.MainStandalonePipelineManagerModule$$ModuleAdapter.newModule(MainStandalonePipelineManagerModule$$ModuleAdapter.java:27)
        at com.streamsets.datacollector.main.MainStandalonePipelineManagerModule$$ModuleAdapter.newModule(MainStandalonePipelineManagerModule$$ModuleAdapter.java:16)
        at dagger.internal.Modules.loadModules(Modules.java:41)
        at dagger.ObjectGraph$DaggerObjectGraph.makeGraph(ObjectGraph.java:174)
        at dagger.ObjectGraph$DaggerObjectGraph.access$000(ObjectGraph.java:138)
        at dagger.ObjectGraph.create(ObjectGraph.java:129)
        at com.streamsets.datacollector.main.Main.<init>(Main.java:49)
        at com.streamsets.datacollector.main.DataCollectorMain.<init>(DataCollectorMain.java:28)
        at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:53)
        ... 6 more
Caused by: java.io.FileNotFoundException: /data/sdc.id (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at com.google.common.io.Files$FileByteSink.openStream(Files.java:200)
        at com.google.common.io.Files$FileByteSink.openStream(Files.java:188)
        at com.google.common.io.ByteSink$AsCharSink.openStream(ByteSink.java:148)
        at com.google.common.io.CharSink.write(CharSink.java:95)
        at com.google.common.io.Files.write(Files.java:320)
        at com.streamsets.datacollector.main.StandaloneRuntimeInfo.getSdcId(StandaloneRuntimeInfo.java:80)
        ... 25 more

`

more