Ask Your Question
2

How to run docker image for other SDC_HOME values than setup in base image

asked 2019-12-11 16:42:25 -0500

anonymous user

Anonymous

As subject line specified, how can I override current/existing Environment/arguments to user-defined values.

e.g. I would like my streamsets datacollector configuration to be set as follow, in my child dockerfile instead of existing one in BASE.

ARG SDC_HOME="/app/streamsets/treamsets-datacollector"
ENV SDC_CONF=${SDC_HOME}/etc \
    SDC_DATA={SDC_HOME}/data \
    SDC_DIST=${SDC_HOME} \
    SDC_HOME=${SDC_HOME} \
    SDC_LOG=${SDC_HOME}/logs \
    SDC_RESOURCES=${SDC_HOME}/resources \
    USER_LIBRARIES_DIR=${SDC_HOME}/streamsets-datacollector-user-libs
ENV STREAMSETS_LIBRARIES_EXTRA_DIR="${SDC_DIST}/streamsets-libs-extras"

can you suggest some sample where we can have streamsets dc running on it's own path?

I created below docker file:

ARG SDC_VERSION=3.8.0
ARG SDC_HOME="/app/streamsets/treamsets-datacollector"
ARG SDC_URL="https://archives.streamsets.com/datacollector/3.8.0/tarball/streamsets-datacollector-core-3.8.0.tgz"
ARG SDC_USER=sdc
ARG SDC_UID=20159
ARG SDC_GID=20159

FROM streamsets/datacollector:${SDC_VERSION}

# override the base ENTRYPOINT
ENTRYPOINT []

ENV SDC_CONF=${SDC_HOME}/etc \
    SDC_DATA={SDC_HOME}/data \
    SDC_DIST=${SDC_HOME} \
    SDC_HOME=${SDC_HOME} \
    SDC_LOG=${SDC_HOME}/logs \
    SDC_RESOURCES=${SDC_HOME}/resources \
    USER_LIBRARIES_DIR=${SDC_HOME}/streamsets-datacollector-user-libs
ENV STREAMSETS_LIBRARIES_EXTRA_DIR="${SDC_DIST}/streamsets-libs-extras"

#RUN entrypoint hoping above value will refect in the conatiner
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["dc", "-exec"]

So after the build of this image, if I run my image (test1), I still see the old folders. How can I replace it with new ones?

docker run --restart on-failure -p 18630:18630 test1
Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTS
WARN: Security is enabled and was unable to verify policy file 'file:///opt/streamsets-datacollector-3.8.0/etc/sdc-security.policy'
Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
OpenJDK 64-Bit Server VM warning: Cannot open file **/opt/streamsets-datacollector-3.8.0/logs/gc.log** due to No such file or directory

Exception in thread "main" java.security.AccessControlException: access denied ("java.util.PropertyPermission" "streamsets.bootstrap.customInitializer.class" "read")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
        at java.security.AccessController.checkPermission(AccessController.java:884)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
        at java.lang.SecurityManager.checkPropertyAccess(SecurityManager.java:1294)
        at java.lang.System.getProperty(System.java:717)
        at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:107)
edit retag flag offensive close merge delete

Comments

Looks like a typo in the line ARG SDC_HOME="/app/streamsets/treamsets-datacollector" - 'treamsets'?

metadaddy gravatar imagemetadaddy ( 2019-12-11 17:11:54 -0500 )edit

yes, you are right but does it matter? @metadaddy. I would like to see all my packages should be extracted in this typo directory which is not happening at this moment. :-)

Prometheous2019 gravatar imagePrometheous2019 ( 2019-12-12 00:07:46 -0500 )edit

2 Answers

Sort by ยป oldest newest most voted
1

answered 2019-12-11 17:28:44 -0500

Dima gravatar image

updated 2019-12-11 17:34:36 -0500

Using one of our existing images with FROM won't work; by that point, we've already extracted all the folders into their default locations in the image and changing environment variables will just mean that those variables would be pointed to nonexistent locations. The way to do this would be simply to rerun a docker build using a command as follows:

docker build --no-cache -t streamsets/datacollector:3.12.0_new_home --build-arg SDC_URL=http://archives.streamsets.com/datacollector/3.12.0/tarball/streamsets-datacollector-core-3.12.0.tgz --build-arg SDC_VERSION=3.12.0 --build-arg SDC_HOME=/app/streamsets/streamsets-datacollector https://github.com/streamsets/datacollector-docker.git

Unfortunately, since /app doesn't exist in our base image, the part of our build scripts that creates directories fails when using the example location you specified. I'll file a Jira to fix this, but there would be further changes needed to our Dockerfile to support specifying the SDC_LOG directory (among others) at build time. In general, though, you wouldn't have to modify a Dockerfile yourself to accomplish this.

edit flag offensive delete link more

Comments

@Dima, So, I took your challenge and created a new Docker file, which is kind of better than my previous situation, but now Im running into error Missing CredentialStore definition 'streamsets-datacollector-aws-secrets-manager-credentialstore-lib::com_streamsets_datacollector_credential_aws

Prometheous2019 gravatar imagePrometheous2019 ( 2019-12-12 09:12:53 -0500 )edit

How can I or post my docker file, shall I create another issue and have a look? or what is the process?

Prometheous2019 gravatar imagePrometheous2019 ( 2019-12-12 09:14:36 -0500 )edit
0

answered 2019-12-12 00:07:50 -0500

Prometheous2019 gravatar image

thanks for your quick response and yes, I agree that there will be a lot needed to change in order to accomplish the desired situation. So, in summary, the current docker file won't allow the user-defined custom directory.

If so, then I would like to see a JIRA to customized this Docker file.

edit flag offensive delete link more

Comments

Feel free to create a Jira at https://issues.streamsets.com/

metadaddy gravatar imagemetadaddy ( 2019-12-12 10:08:38 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-12-11 16:42:25 -0500

Seen: 166 times

Last updated: Dec 12 '19