Support Questions

cjervis · ‎09-12-2019

Is there anywhere to reduce the cloudera-scm-agent memory usage? Currently it's consuming 1.9G:

top - 00:49:46 up 10 days, 11:06, 2 users, load average: 0.62, 0.43, 0.50
Tasks: 195 total, 1 running, 194 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.5 us, 0.2 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 8008644 total, 3828140 free, 3228256 used, 952248 buff/cache
KiB Swap: 4063228 total, 3974644 free, 88584 used. 4180168 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24743 clouder+ 20 0 7107092 278440 5388 S 16.6 3.5 1329:59 java
13611 root 20 0 1260404 41488 2252 S 2.7 0.5 1:27.75 python2
23190 clouder+ 20 0 8526144 1.9g 47492 S 1.3 25.5 2470:33 java

Even with a stopped cluster.

Anyway to tweak this on smaller POC clusters?

Would anyone be able to ballpark memory settings for small Cloudera clusters and services?

Need to restart the agent and clear caches to free up memory. Wondering if that could be reduced instead.

Cheers,
TK

EricL · ‎09-12-2019

@TCloud ,

How did you determine 23190 is PID for CM agent? Can you run "service cloudera-scm-agent status" and confirm the PID for CM agent?

Also, can run "ps aux | grep 23190" to confirm if it really belong to CM agent.

Cheers
Eric

TCloud · ‎09-12-2019

Apologies. You're right. mysql connector running through the Cloudera user:

[root@cm-r01nn01 ~]# top
top - 02:06:24 up 19 days,  4:04,  1 user,  load average: 0.26, 0.28, 0.29
Tasks: 197 total,   1 running, 196 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.0 us,  0.5 sy,  0.0 ni, 98.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8008640 total,   286564 free,  5060212 used,  2661864 buff/cache
KiB Swap:  4063228 total,  4063228 free,        0 used.  2205572 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 3919 root      20   0 1260148  47188   8428 S   3.8  0.6   2:13.33 python2
13778 clouder+  20   0 8263556   2.8g  17472 S   3.8 37.1   2208:20 java
 3877 root      20   0 2680196  63480   9772 S   1.2  0.8   1:46.97 cmagent
 4160 root      20   0  619212  28772   9028 S   1.2  0.4   0:32.63 python2
 6281 httpfs    20   0 3860244 213008  29404 S   1.2  2.7   0:29.09 java
 7367 hive      20   0 2118308 312892  41972 S   1.2  3.9   1:09.55 java
14318 root      20   0  172308   2404   1628 R   1.2  0.0   0:00.49 top
    1 root      20   0   54952   7272   4268 S   0.0  0.1   6:44.58 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:01.41 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   1:27.50 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      rt   0       0      0      0 S   0.0  0.0   0:04.12 migration/0
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      20   0       0      0      0 S   0.0  0.0  29:36.83 rcu_sched
   10 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 lru-add-drain
   11 root      rt   0       0      0      0 S   0.0  0.0   0:12.34 watchdog/0
   12 root      rt   0       0      0      0 S   0.0  0.0   0:08.09 watchdog/1
   13 root      rt   0       0      0      0 S   0.0  0.0   0:02.97 migration/1
   14 root      20   0       0      0      0 S   0.0  0.0   1:21.07 ksoftirqd/1
   16 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   17 root      rt   0       0      0      0 S   0.0  0.0   0:07.84 watchdog/2
   18 root      rt   0       0      0      0 S   0.0  0.0   0:00.64 migration/2
   19 root      20   0       0      0      0 S   0.0  0.0   1:20.44 ksoftirqd/2
   21 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H
   22 root      rt   0       0      0      0 S   0.0  0.0   0:08.05 watchdog/3
   23 root      rt   0       0      0      0 S   0.0  0.0   0:00.39 migration/3
   24 root      20   0       0      0      0 S   0.0  0.0   1:20.37 ksoftirqd/3
   26 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0H
   27 root      rt   0       0      0      0 S   0.0  0.0   0:08.38 watchdog/4
   28 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 migration/4
[root@cm-r01nn01 ~]# ps -ef|grep -Ei 13778
clouder+ 13778     1  8 Aug25 ?        1-12:48:21 /usr/java/jdk1.8.0_181-cloudera/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -server -Dlog4j.configuration=file:/etc/cloudera-scm-server/log4j.properties -Dfile.encoding=UTF-8 -Dcmf.root.logger=INFO,LOGFILE -Dcmf.log.dir=/var/log/cloudera-scm-server -Dcmf.log.file=cloudera-scm-server.log -Dcmf.jetty.threshhold=WARN -Dcmf.schema.dir=/opt/cloudera/cm/schema -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dpython.home=/opt/cloudera/cm/python -XX:+HeapDumpOnOutOfMemoryError -Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:OnOutOfMemoryError=kill -9 %p com.cloudera.server.cmf.Main
root     14404  4844  0 02:06 pts/0    00:00:00 grep --color=auto -Ei 13778
[root@cm-r01nn01 ~]#

[root@cm-r01nn01 ~]# top
top - 02:06:24 up 19 days,  4:04,  1 user,  load average: 0.26, 0.28, 0.29
Tasks: 197 total,   1 running, 196 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.0 us,  0.5 sy,  0.0 ni, 98.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8008640 total,   286564 free,  5060212 used,  2661864 buff/cache
KiB Swap:  4063228 total,  4063228 free,        0 used.  2205572 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 3919 root      20   0 1260148  47188   8428 S   3.8  0.6   2:13.33 python2
13778 clouder+  20   0 8263556   2.8g  17472 S   3.8 37.1   2208:20 java
 3877 root      20   0 2680196  63480   9772 S   1.2  0.8   1:46.97 cmagent
 4160 root      20   0  619212  28772   9028 S   1.2  0.4   0:32.63 python2
 6281 httpfs    20   0 3860244 213008  29404 S   1.2  2.7   0:29.09 java
 7367 hive      20   0 2118308 312892  41972 S   1.2  3.9   1:09.55 java
14318 root      20   0  172308   2404   1628 R   1.2  0.0   0:00.49 top
    1 root      20   0   54952   7272   4268 S   0.0  0.1   6:44.58 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:01.41 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   1:27.50 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      rt   0       0      0      0 S   0.0  0.0   0:04.12 migration/0
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      20   0       0      0      0 S   0.0  0.0  29:36.83 rcu_sched
   10 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 lru-add-drain
   11 root      rt   0       0      0      0 S   0.0  0.0   0:12.34 watchdog/0
   12 root      rt   0       0      0      0 S   0.0  0.0   0:08.09 watchdog/1
   13 root      rt   0       0      0      0 S   0.0  0.0   0:02.97 migration/1
   14 root      20   0       0      0      0 S   0.0  0.0   1:21.07 ksoftirqd/1
   16 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   17 root      rt   0       0      0      0 S   0.0  0.0   0:07.84 watchdog/2
   18 root      rt   0       0      0      0 S   0.0  0.0   0:00.64 migration/2
   19 root      20   0       0      0      0 S   0.0  0.0   1:20.44 ksoftirqd/2
   21 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H
   22 root      rt   0       0      0      0 S   0.0  0.0   0:08.05 watchdog/3
   23 root      rt   0       0      0      0 S   0.0  0.0   0:00.39 migration/3
   24 root      20   0       0      0      0 S   0.0  0.0   1:20.37 ksoftirqd/3
   26 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0H
   27 root      rt   0       0      0      0 S   0.0  0.0   0:08.38 watchdog/4
   28 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 migration/4
[root@cm-r01nn01 ~]# ps -ef|grep -Ei 13778
clouder+ 13778     1  8 Aug25 ?        1-12:48:21 /usr/java/jdk1.8.0_181-cloudera/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -server -Dlog4j.configuration=file:/etc/cloudera-scm-server/log4j.properties -Dfile.encoding=UTF-8 -Dcmf.root.logger=INFO,LOGFILE -Dcmf.log.dir=/var/log/cloudera-scm-server -Dcmf.log.file=cloudera-scm-server.log -Dcmf.jetty.threshhold=WARN -Dcmf.schema.dir=/opt/cloudera/cm/schema -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dpython.home=/opt/cloudera/cm/python -XX:+HeapDumpOnOutOfMemoryError -Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:OnOutOfMemoryError=kill -9 %p com.cloudera.server.cmf.Main
root     14404  4844  0 02:06 pts/0    00:00:00 grep --color=auto -Ei 13778
[root@cm-r01nn01 ~]# systemctl status cloudera-scm-agent
â cloudera-scm-agent.service - Cloudera Manager Agent Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2019-09-13 00:55:44 EDT; 1h 11min ago
 Main PID: 3877 (cmagent)
   CGroup: /system.slice/cloudera-scm-agent.service
           ââ3877 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm agent

Sep 13 00:55:55 cm-r01nn01.mws.mds.xyz cm[3877]: warnings.warn(msg, RuntimeWarning)
Sep 13 01:10:58 cm-r01nn01.mws.mds.xyz cm[3877]: 2591-kms-KMS: added process group
Sep 13 01:10:58 cm-r01nn01.mws.mds.xyz cm[3877]: 2592-zookeeper-server: added process group
Sep 13 01:12:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2598-hdfs-HTTPFS: added process group
Sep 13 01:12:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2600-hdfs-SECONDARYNAMENODE: added process group
Sep 13 01:17:05 cm-r01nn01.mws.mds.xyz cm[3877]: [2019-09-13 01:17:05,746 pyinotify ERROR] add_watch: cannot watch /var/log/hive/audit WD=-1, Er...(ENOENT)
Sep 13 01:18:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2649-hive-HIVESERVER2: added process group
Sep 13 01:18:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2650-hive-WEBHCAT: added process group
Sep 13 01:54:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2649-hive-HIVESERVER2: stopped
Sep 13 01:54:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2649-hive-HIVESERVER2: removed process group
Hint: Some lines were ellipsized, use -l to show in full.
[root@cm-r01nn01 ~]# systemctl status cloudera-scm-agent -l
â cloudera-scm-agent.service - Cloudera Manager Agent Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2019-09-13 00:55:44 EDT; 1h 12min ago
 Main PID: 3877 (cmagent)
   CGroup: /system.slice/cloudera-scm-agent.service
           ââ3877 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm agent

Sep 13 00:55:55 cm-r01nn01.mws.mds.xyz cm[3877]: warnings.warn(msg, RuntimeWarning)
Sep 13 01:10:58 cm-r01nn01.mws.mds.xyz cm[3877]: 2591-kms-KMS: added process group
Sep 13 01:10:58 cm-r01nn01.mws.mds.xyz cm[3877]: 2592-zookeeper-server: added process group
Sep 13 01:12:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2598-hdfs-HTTPFS: added process group
Sep 13 01:12:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2600-hdfs-SECONDARYNAMENODE: added process group
Sep 13 01:17:05 cm-r01nn01.mws.mds.xyz cm[3877]: [2019-09-13 01:17:05,746 pyinotify ERROR] add_watch: cannot watch /var/log/hive/audit WD=-1, Errno=No such file or directory (ENOENT)
Sep 13 01:18:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2649-hive-HIVESERVER2: added process group
Sep 13 01:18:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2650-hive-WEBHCAT: added process group
Sep 13 01:54:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2649-hive-HIVESERVER2: stopped
Sep 13 01:54:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2649-hive-HIVESERVER2: removed process group
[root@cm-r01nn01 ~]#

EricL · ‎09-15-2019

13778 is Cloudera Manager Server process. What about 23190 you mentioned previously? You have not checked/confirmed which process this ID belongs to yet.

TCloud · ‎09-15-2019

I restarted things before capturing the second reply above to free space on the cluster. PID 13778 would be the same thing as the original PID 23190 I listed. The process always takes up 1.8+ GB. After a short while it climbs to 2.8GB.

EricL · ‎09-16-2019

That is Cloudera Manager Server then, not the agent. And it is requirement that you allocate a few GBs of memory for Cloudera Manager server, please refer to below documentation:

https://www.cloudera.com/documentation/enterprise/release-notes/topics/hardware_requirements_guide.h...

For a small cluster of less than 10 nodes, you still need at least 2GB of memory for CM server to function properly. So you 1.9GB is about right and you should not go anything less than that.

Cheers
Eric

Support Questions

Reduce Cloudera Agent Memory consuption