Ask Your Question
0

How to adjust Streamsets heap size on CentOS7 service?

asked 2018-11-20 17:00:23 -0600

meash-nrel gravatar image

updated 2018-11-26 10:30:01 -0600

I installed StreamSets via RPM on CentOS7 . I am trying to change the heap size, and adjusted the service file to add SDC_JAVA_OPTS, but it seems like the service is still loading default values from somewhere.

Adjusting /usr/lib/systemd/system/sdc.service, I added:

LimitNOFILE=100000
Environment="SDC_JAVA_OPTS=-Xmx10g -Xms10g -server"

I see these variables in the running service's Java parameters, however they are appended after the default values, which are overriding them -- so the process is still running at the default heap size.

Java params of the running process:

JAVA OPTS                      : -Djava.security.manager -Djava.security.policy=file:///etc/sdc/sdc-security.policy -Xmx1024m -Xms1024m -server -XX:-OmitStackTraceInFastThrow -Xmx10g -Xms10g -server -Dsdc.dist.dir=/opt/streamsets-datacollector -Dsdc.resources.dir=/opt/streamsets-datacollector/resources -Dsdc.hostname=snowy.hpc.nrel.gov -Dsdc.conf.dir=/etc/sdc -Dsdc.data.dir=/var/lib/sdc -Dsdc.log.dir=/var/log/sdc -javaagent:/opt/streamsets-datacollector/libexec/bootstrap-libs/main/streamsets-datacollector-bootstrap-3.5.1.jar -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/sdc/sdc_heapdump_1542753104.hprof -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/sdc/gc.log

Notice how my -Xmx10g part is there ... but after the -Xmx1024m default.

Where is this default getting loaded from and what is the proper way to override it in CentOS7 systemd?

I've adjusted env variables in service files before (eg Kafka service) and have never encountered it where it is appending env vars instead of overriding.

edit retag flag offensive close merge delete

Comments

Hi! After updating 'sdc.service', did you reload the systemd manager configuration by executing 'systemctl daemon-reload' -- before restarting the data collector service?

iamontheinet gravatar imageiamontheinet ( 2018-11-21 19:30:59 -0600 )edit

yes the systemctl is reloaded after each change of the service file, and you can see my changes are reflected in the options shown above -- the problem is that the default heap size (1024m) is being injected somewhere else besides my provided SDC_JAVA_OPTS .

meash-nrel gravatar imagemeash-nrel ( 2018-11-26 10:27:34 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2018-11-26 11:01:01 -0600

meash-nrel gravatar image

Editing the service file was the wrong approach. Yes, SDC_JAVA_OPTS can be provided there, but the sdc-env.sh file gets loaded, which puts the service-provided SDC_JAVA_OPTS after the provided defaults. To change the settings already set in the defaults, you must edit that file instead of setting them in the service.

I edited /opt/streamsets-datacollector/libexec/sdc-env.sh and adjusted SDC_JAVA_OPTS defaults:

export SDC_JAVA_OPTS="-Xmx10g -Xms10g -server -XX:-OmitStackTraceInFastThrow ${SDC_JAVA_OPTS}"

(I tried editing sdcd-env.sh first, thinking it was for service daemon, but it wasn't used by the service. Set it in sdc-env.sh instead).

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-11-20 17:00:23 -0600

Seen: 22 times

Last updated: Nov 26