How to resolve space issue in cloudera management server

[#1]

Disk space is increasing in cloudera management server. Could some one help us to delete the correct files to free some space in root file system

Here is my fize greater than +100 MB

-sh-4.2$ sudo find / -xdev -type f -size +100M -exec ls -ltrh {} \; | sort -nk 5

-rw-r----- 1 cloudera-scm cloudera-scm 1.9G Jan 18 15:23 /opt/cloudera/parcel-repo/CDH-5.13.1-1.cdh5.13.1.p0.2-el7.parcel
-rw-r----- 1 cloudera-scm cloudera-scm 3.7G Jan 30 14:16 /opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.0.3.0-el7.parcel
-rw-rw-r-- 1 root root 3.8G Dec 13 15:21 /opt/cloudera/parcels/CDSW-1.2.2.p1.216803/images/cdsw_1.2.2_2cbfa5b.tar.gz
-rw-r----- 1 cloudera-scm cloudera-scm 3.9G Jan 20 18:35 /opt/cloudera/parcel-repo/CDSW-1.2.2.p1.216803-el7.parcel
-rw-r--r--. 1 root root 102M Jan 18 12:30 /usr/lib/locale/locale-archive
-rw-r--r-- 1 root root 104M Nov 9 12:38 /opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/jars/spark-assembly-1.6.0-cdh5.13.1-hadoop2.6.0-cdh5.13.1.jar
-rw-r--r-- 1 root root 106M Nov 9 12:41 /opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/jars/avro-tools-1.7.6-cdh5.13.1.jar
-rw------- 1 root root 117M Apr 17 08:27 /var/log/sssd/ldap_child.log-20180227
-rw-r--r-- 1 root root 118M Nov 14 16:32 /usr/share/cmf/cloudera-navigator-server/wars/nav-core-webapp-2.12.1.war
-rw-r----- 1 cloudera-scm cloudera-scm 120M Apr 17 08:34 /var/lib/cloudera-scm-navigator/solr/nav_elements/data/tlog/tlog.0000000000000018862
-rwxr-xr-x 1 root root 133M Dec 13 15:20 /opt/cloudera/parcels/CDSW-1.2.2.p1.216803/kubernetes/bin/kubelet
-rw-r----- 1 cloudera-scm cloudera-scm 147M Apr 17 08:41 /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR-ruduv-kmmgmt001.rush.edu.log.out
-rw-r--r-- 1 root root 151M Nov 9 12:39 /opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/jars/hbase-indexer-mr-1.5-cdh5.13.1-job.jar
-rw-r----- 1 cloudera-scm cloudera-scm 160M Apr 17 08:17 /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_t.fdt
-rw-r----- 1 cloudera-scm cloudera-scm 172M Apr 17 08:17 /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_u.fdt
-rw-r----- 1 cloudera-scm cloudera-scm 173M Jan 19 11:39 /opt/cloudera/parcel-repo/SPARK2-2.2.0.cloudera2-1.cdh5.12.0.p0.232957-el7.parcel
-rw-r----- 1 cloudera-scm cloudera-scm 201M Feb 25 13:15 /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR-ruduv-kmmgmt001.rush.edu.log.out.4
-rw-r----- 1 cloudera-scm cloudera-scm 201M Mar 11 11:58 /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR-ruduv-kmmgmt001.rush.edu.log.out.2
-rw-r----- 1 cloudera-scm cloudera-scm 201M Mar 20 06:27 /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR-ruduv-kmmgmt001.rush.edu.log.out.1
-rw-r----- 1 cloudera-scm cloudera-scm 201M Mar 9 14:24 /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR-ruduv-kmmgmt001.rush.edu.log.out.3
-rw-r--r-- 1 root root 202M Nov 9 12:37 /opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/debug/usr/lib/impala/sbin-debug/impalad.debug
-rw-r--r-- 1 root root 251M Nov 9 12:37 /opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/debug/usr/lib/impala/sbin-retail/impalad.debug
-rw------- 1 root root 299M Apr 16 03:44 /var/log/sssd/sssd_rush.edu.log-20180416
-rw-r----- 1 cloudera-scm cloudera-scm 342M Mar 28 10:11 /var/lib/cloudera-scm-eventserver/v3/_43pe.fdt
-rw-r----- 1 cloudera-scm cloudera-scm 346M Jan 18 15:23 /opt/cloudera/parcel-repo/KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel
-rw-r----- 1 cloudera-scm cloudera-scm 347M Mar 5 13:47 /var/lib/cloudera-scm-eventserver/v3/_2e3g.fdt

[#2]

@dpugazhe

Below are the usual excercise that we follow to reduce the log history, but ...

a. it is purly depending upon your client's business, if they are not demanding to keep longer log hisotry then you can try this
b. i've given below few samples, you don't need to reduce history for all the logs, pls do your own research to see which history file is taking more space and take action by reducing the max limit and size of the history file

CM -> HDFS -> Configuration -> search for the below

1. navigator.client.max_num_audit_log -> 'The default value is 10' - you can reduce it to 8 or 6 (it is recommended to have more history in general)
2. navigator.audit_log_max_file_size -> 'The default value is 100 MB' - you can reduce it to 80MB or 50MB

Note: You can try both --or-- any one

3. DataNode Max Log Size -> ' the default value is 200 MB' - you can reduce as needed
4. DataNode Maximum Log File Backups -> ' the default value is 10' - you can reduce as needed
5. NameNode Max Log Size -> 'the default value is 200 MB' - you can reduce as needed
6. NameNode Maximum Log File Backups -> 'the default value is 300' - you can reduce as needed

NOTE: I am repeating again, please consider the point a & b before you take action

[#3]

You should not delete anything under /opt/cloudera and /var/lib directories. If the contents of these directories are too high for your partitions, then you should consider extending them.

There is an exception in /var/lib/ but again you should not delete manually.

The only place you can delete files without issues is "/var/log/..." but this is a temporarily solution.

The "proper" way is to change "Max Log Size" and "Maximum Log File Backups" in Cloudera Manager, for each service is running on this machine.

Edit: I started writing before I see the reply from @saranvisa. I agree with this.

[#4]

Thanks Gekas. So I can delete the log files under the name of   cloudera-scm-% in the /var/log folder

It wont cause any issue  in UI right ?

This is error message I see in cloudera manager 

This role's Log Directory is on a filesystem with less than 5.0 GiB of its space free. /var/log/cloudera-scm-firehose (free: 4.0 GiB (8.02%), capacity: 50.0 GiB)

The "proper" way is to change "Max Log Size" and "Maximum Log File Backups" in Cloudera Manager, for each service is running on this machine.

Its set to 200 MB . Please advice 

[#5]

If you delete the log files that end with a number (e.g. mgmt-cmf-mgmt-NAVIGATORMETASERVER-my.node.com.log.out.1), then you will have no issue. Don't delete the ones with ".log" suffix, because they are open for writing by services.

Suppose that you have

Max Logs Size = 200MB

Maximum Log File Backups = 10

That means that each time your .log file reaches 200MB will be rolled, for ten times in total (will be copied to .log.1, .log.2,.....,.log.10). So in total, the log files of a service can occupy up to ((10* 200MB) + 200MB)=2200MB.

Depending on your disk size you can reduce only one or both of them, depending on what is better for you.

E.g. if you don't want so big files you can set "Max Log Size"=100MB, that means ((10*100MB) + 100MB) = 1100 MB.

So just reducing this parameter into half, you save 1100MB of disk space.

The same applies to all services.

[#6]

@dpugazhe Generally the / monut in the linux servers are small.

Could you share the df -h command output of you linux box?

I would  suggest you to change the location for the parcels and logs for example if you have larger mount in your linux box called /xxxxx, change the /var/lib and /var/log to /xxxx/hadoop/lib and /xxxx/hadoop/log and the same for the parcels, as you are using cloudera manager, these changes can be done quickly. so to do that.

1- Stop cloudera manager services.

2- Move the old logs to the new partition.

3- Delete the old logs.

4- Start cloudera manager services