Support Questions
Find answers, ask questions, and share your expertise

HDP 2.5 - Storm process taking up all system memory and utilizing 100% of a CPU core.

Explorer

HDP 2.5, Ambari 2.4.1.0-22 running on CentOS 7.2.

This is after a fresh install, one of my data nodes keeps having this issue: capture.png

The process is running under the storm user, and it's running "python splitsentence.py", as you can see it's utilizing almost all of the memory on the system. When I stop All of the components on this host, this process does not get killed. What HDP process is launching this? How can I prevent this from happening?

Data node is only running HDFS / Flume / RegionServer / Metrics Monitor / Supervisor / NodeManager.

Thanks

1 REPLY 1

Explorer

This is the output I see when I dig into the process via lsof.

It looks like Supervisor kicks off this process.

[root@dldata ~]# lsof -l | grep 12863
python    12863          1002  cwd       DIR              253,0         6   67437098 /hadoop/storm/supervisor/stormdist/WordCountid10ac6aa9_date422216-1-1479850989/resources (deleted)
python    12863          1002  rtd       DIR              253,0      4096        128 /
python    12863          1002  txt       REG              253,0      7136  134417881 /usr/bin/python2.7
python    12863          1002  mem       REG              253,0     25696   67213514 /usr/lib64/python2.7/lib-dynload/binascii.so
python    12863          1002  mem       REG              253,0     39024   67246640 /usr/lib64/python2.7/lib-dynload/_json.so
python    12863          1002  mem       REG              253,0     38912   67130385 /usr/lib64/python2.7/lib-dynload/_struct.so
python    12863          1002  mem       REG              253,0     22856   67246638 /usr/lib64/python2.7/lib-dynload/_heapq.so
python    12863          1002  mem       REG              253,0     61976   67213527 /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so
python    12863          1002  mem       REG              253,0     47600   67213532 /usr/lib64/python2.7/lib-dynload/operator.so
python    12863          1002  mem       REG              253,0     33024   67213490 /usr/lib64/python2.7/lib-dynload/_collectionsmodule.so
python    12863          1002  mem       REG              253,0 106065056  134323023 /usr/lib/locale/locale-archive
python    12863          1002  mem       REG              253,0   2112384  201328477 /usr/lib64/libc-2.17.so
python    12863          1002  mem       REG              253,0   1141560  201328485 /usr/lib64/libm-2.17.so
python    12863          1002  mem       REG              253,0     14608  201328517 /usr/lib64/libutil-2.17.so
python    12863          1002  mem       REG              253,0     19520  201328483 /usr/lib64/libdl-2.17.so
python    12863          1002  mem       REG              253,0    142304  201328505 /usr/lib64/libpthread-2.17.so
python    12863          1002  mem       REG              253,0   1822536  201643978 /usr/lib64/libpython2.7.so.1.0
python    12863          1002  mem       REG              253,0    164440  201328470 /usr/lib64/ld-2.17.so
python    12863          1002    0r     FIFO                0,8       0t0    4569697 pipe
python    12863          1002    1w     FIFO                0,8       0t0    4569698 pipe
python    12863          1002    2w     FIFO                0,8       0t0    4569699 pipe

This is my childopts for Supervisor in Ambari (it's limited to 256m)

-Xmx256m _JAAS_PLACEHOLDER -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port={{jmxremote_port}} -javaagent:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Supervisor_JVM