Out of memory error!!!

asked 2018-05-22 20:37:58 -0500

casel.chen gravatar image

updated 2018-05-25 10:13:40 -0500

metadaddy gravatar image

I run about 12 data pipelines with sdc 3.0.3.0 (export SDC_JAVA_OPTS="-Xmx2048m -Xms2048m -server ${SDC_JAVA_OPTS}") and after several hours sdc exit itself and report out of memory error in hs_err_pid31082.log file, see attached. What's wrong with it?

[root@ambari-agent2 ~]# top
top - 10:48:36 up 59 days, 18:14,  2 users,  load average: 0.50, 0.57, 0.64
Tasks:  99 total,   1 running,  98 sleeping,   0 stopped,   0 zombie
%Cpu(s): 20.7 us,  1.3 sy,  0.0 ni, 77.6 id,  0.2 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem :  8010688 total,   454672 free,  6141372 used,  1414644 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  1577036 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                     
12612 root      20   0 5163464 914288  56824 S  21.6 11.4   2:41.96 java    

Note: 12612 is sdc PID

[root@ambari-agent2 ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           7822        5993         447           0        1381        1543
Swap:             0           0           0

sdc.properties: https://gist.github.com/metadaddy/bed...

OOM error: https://gist.github.com/metadaddy/3b6...

edit retag flag offensive close merge delete

Comments

Tell us more about the server. How much physical memory does it have, and how much is being used under steady load? For instance: include the output of the top command when everything has started running but before the crash.

jeff gravatar imagejeff ( 2018-05-23 15:00:42 -0500 )edit
jeff gravatar imagejeff ( 2018-05-23 15:00:57 -0500 )edit

I supplement some materials you requested, please check!

casel.chen gravatar imagecasel.chen ( 2018-05-24 22:13:17 -0500 )edit

Please don't paste hundreds of lines of text into a question. I created two gists with the OOM output and sdc.properties file

metadaddy gravatar imagemetadaddy ( 2018-05-25 10:13:17 -0500 )edit