Ask Your Question

Streamsets Always Crashes with Segmentation Faults

asked 2019-01-16 13:32:49 -0500

seamfaces gravatar image

updated 2019-01-17 11:07:27 -0500

metadaddy gravatar image

Hello everyone,

I have a standalone StreamSets setup that has been working quite well for some time until very recently it started crashing without warning. Attempts to restart it repeatedly result in more crashes with out a dump file or anything at all in the logs.

I got to find out hours later that the crash was due to segmentation fault (I know this has something to do with memory allocation). I have provisioned ample memory on the box but still the problem persists

There are currently only about five pipelines that have been setup and the default garbage collector is in use here.

Am I doing something wrong???

Memory Is Allocated As Follows

export SDC_JAVA_OPTS="-Xmx16g -Xms16g -server -XX:MaxDirectMemorySize=2g -XX:NewSize=4g -XX:MaxNewSize=4g -XX:-UseCompressedOops -XX:-OmitStackTraceInFastThrow -Doracle.jdbc.javaNetNio=true ${SDC_JAVA_OPTS}"
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2019-01-16 19:54:06 -0500

metadaddy gravatar image

A segmentation fault indicates that the JVM itself has crashed due to an internal error. Did you update Java around the time you saw the crashes?

edit flag offensive delete link more


No I did not.

seamfaces gravatar imageseamfaces ( 2019-01-17 05:21:11 -0500 )edit

Please, I have updated the question with the jvm memory allocation updates

seamfaces gravatar imageseamfaces ( 2019-01-17 07:00:44 -0500 )edit

With larger heap/memory size, you might want to use G1 garbage collector. Here's the command: export SDC_JAVA8_OPTS=${SDC_JAVA8_OPTS:-"-XX:+UseG1GC"} By default it uses Concurrent Mark Sweep (CMS) garbage collector.

iamontheinet gravatar imageiamontheinet ( 2019-01-17 10:56:42 -0500 )edit

Did that already, still the same problem

seamfaces gravatar imageseamfaces ( 2019-01-17 11:08:40 -0500 )edit

Which version of SDC are you running?

iamontheinet gravatar imageiamontheinet ( 2019-01-17 11:12:28 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-01-16 13:32:49 -0500

Seen: 180 times

Last updated: Jan 17