Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudera Manager Java processes using lots of memory on Rocks Cluster only HDFS & Spark

Highlighted

Cloudera Manager Java processes using lots of memory on Rocks Cluster only HDFS & Spark

Explorer

Here is a HTOP CM is killing our server. Any ideas why?


Mem12134/32103MB] Tasks: 222, 1306 thr; 1 running
Swp[| 0/1999MB] Load average: 0.43 0.47 0.34
Uptime: 32 days, 17:46:16

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
7310 nobody 20 0 306M 16284 3876 S 9.2 0.0 6:20.40 /usr/sbin/gmond
25809 cloudera- 20 0 12.0G 1563M 102M S 5.0 4.9 1h21:15 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-SERVICEMONITOR-ourdomain
15527 root 20 0 112M 4436 1252 R 4.1 0.0 0:00.88 htop
16174 cloudera- 20 0 6875M 586M 22468 S 2.3 1.8 12:50.47 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-ourdomain.
12752 cloudera- 20 0 14.4G 1745M 22768 S 0.9 5.4 19:34.78 /usr/java/jdk1.7.0_67-cloudera/bin/java -cp .:lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar -server -Dlog4j
28096 cloudera- 20 0 14.4G 1745M 22768 S 0.9 5.4 0:50.07 /usr/java/jdk1.7.0_67-cloudera/bin/java -cp .:lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar -server -Dlog4j
26592 cloudera- 20 0 12.0G 1563M 102M S 0.9 4.9 2:17.96 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-SERVICEMONITOR-ourdomain
15127 hdfs 20 0 1325M 591M 18908 S 0.5 1.8 6:55.36 /usr/java/jdk1.7.0_67-cloudera/bin/java -Dproc_namenode -Xmx1000m -Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.prefe
27061 cloudera- 20 0 6875M 586M 22468 S 0.5 1.8 0:44.71 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-ourdomain.
18245 hdfs 20 0 1325M 591M 18908 S 0.5 1.8 0:02.31 /usr/java/jdk1.7.0_67-cloudera/bin/java -Dproc_namenode -Xmx1000m -Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.prefe
9972 mysql 20 0 6759M 610M 5032 S 0.5 1.9 1:56.81 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socke
17352 cloudera- 20 0 6875M 586M 22468 S 0.5 1.8 0:47.25 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-ourdomain.
3479 leeds 20 0 618M 68700 4044 S 0.5 0.2 36:27.01 /usr/libexec/gnome-settings-daemon
4456 condor 20 0 94552 8556 3476 S 0.5 0.0 41:35.74 condor_collector -f
15950 hdfs 20 0 1228M 354M 18748 S 0.5 1.1 1:55.50 /usr/java/jdk1.7.0_67-cloudera/bin/java -Dproc_secondarynamenode -Xmx1000m -Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.
26619 cloudera- 20 0 12.0G 1563M 102M S 0.5 4.9 1:05.96 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-SERVICEMONITOR-ourdomain
26623 cloudera- 20 0 12.0G 1563M 102M S 0.5 4.9 1:05.79 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-SERVICEMONITOR-ourdomain
26624 cloudera- 20 0 12.0G 1563M 102M S 0.5 4.9 1:06.15 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-SERVICEMONITOR-ourdomain
26617 cloudera- 20 0 12.0G 1563M 102M S 0.5 4.9 1:05.76 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-SERVICEMONITOR-ourdomain
26618 cloudera- 20 0 12.0G 1563M 102M S 0.5 4.9 1:05.87 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-SERVICEMONITOR-ourdomain
26621 cloudera- 20 0 12.0G 1563M 102M S 0.5 4.9 1:05.85 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-SERVICEMONITOR-ourdomain

3 REPLIES 3
Highlighted

Re: Cloudera Manager Java processes using lots of memory on Rocks Cluster only HDFS & Spark

Super Guru

Hello,

 

You say that Service Monitor is killing your server; can you share with us the back story of how you arrived at that conclusion?

What do you mean by "killing?"

What do you mean by "server?"

 

With these clarifications, the community at large will be better able to assist.

 

Regards,

 

Ben                                    

Highlighted

Re: Cloudera Manager Java processes using lots of memory on Rocks Cluster only HDFS & Spark

Explorer

Cloudera Manager (CDH 5.8.3) is using almost all of the availble memory of the head node of the Rocks cluster running on CentOS 6.8. I've disabled all of the services for now on the CM home page (/cmf/home) for now.

 

The output from htop I posted was supposed to demonstrate all of the Java/Cloudera processes that appear to run/spawn.

Highlighted

Re: Cloudera Manager Java processes using lots of memory on Rocks Cluster only HDFS & Spark

Explorer
Any other suggestions? What could be causing such memory usage "out of the box"?
Don't have an account?
Coming from Hortonworks? Activate your account here