Could I delete files in /var/lib?

[#1]

I'm running a one-note cdh4 system for test.

It's found that the size of /var/lib increases quickly. Could I delete those files?

[#2]

Hello,

Cloudera Manager stores information there, so deleting it would not be advised.

Check what directory or directories grow with something like:

du -sh /var/lib/* |sort -n

If you can isolate what is growing, perhaps we can suggest a way to limit that growth.

Ben

[#3]

/var/lib is usually used to store the state of the system. So, for example, if you have namenode running on a machine, the metadata for the namenode is written in that directory. Formatting the namenode will clean out a subdirectory of /var/lib, so in general, it's not a good idea to delete those files.

You should look a little more deeply into what's making that directory fill up. If they look like logs, it's likely ok to delete them but most of that directory contains things you don't want to delete from a functioning cluster.

[#4]

Thanks for reply.

I followed your suggestion and found that the size-growing diretory is "/var/lib/cloudera-scm-server-db/data/base"

Here is the detail size of directories in /var/lib/cloudera-scm-server-db/data/base

3.7M    ./1
3.7M    ./10792
3.7M    ./10793
8.3M    ./16385
8.7G    ./16387
84M    ./16389
911M    ./16391
3.7M    ./16393
3.7M    ./16395
5.7M    ./17569

Could this directories be deleted?

[#5]

Those files are actually your embedded PostgreSQL database that contains your Cloudera Manager data, so deleting would be highly destructive.

The embedded PostgreSQL is only intended for demonstration, so the way to go for you may be to configure an external database as described in:

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installat...

If you want to dig in and find out what is actually being stored in the PostgreSQL directories, I found this page that walks you through navigating the various subdirectories:

http://etutorials.org/SQL/Postgresql/Part+I+General+PostgreSQL+Use/Chapter+4.+Performance/How+Postgr...

-Ben

[#6]

If I format the hdfs, will these files clean up as @mgrover said?

If this is the case, how could I format the hdfs? Use the format command just as native hadoop?

[#7]

The point that mgrover was making in the previous post was that you should not delete anything.  Formatting will essentially destroy your hdfs data.

I think a good rule here is to not delete /var/lib data as you will likely corrupt or destroy key parts of your cluster.  In production, you would have an external database and monitor servers on other hosts, storing data in other directories.  There may be ways to free up space in your existing install, but none are trivial or without consequence.

Ben

[#8]

Exactly. All I was trying to do say that removing /var/lib files will
format your namenode. But there are definitely more files from many
different components (from CDH and Cloudera Manager) that put files in
/var/lib so formatting namenode is not necessarily going to help (and it's
going to delete all your HDFS data). In fact, looking at the sizes of
content in that directory, it's likely not going to help.

[#9]

The size of this directory is growing so quickly, about 1G per day even nothing has been done, which is unacceptable for a test machine.

Is such a growing speed normal?

Since this is a system for test, data in it is not so important.

Are there any simple ways in which I could refresh the cloudera system and clean up these files? Or I can only reinstall cloudera manager periodically?

[#10]

Without knowing what is growing or why, it is hard to say if this is normal or expected.  For an idle server with no notable problems and a few quiet nodes, I would not expect such growth. 

I would suggest finding out exactly what disk directory is growing the most; if it is /var/lib/cloudera-scm-server-db/data/base/16387 as you mentioned earlier, I would find out to which database that oid maps to.  Knowing what database is growing will at least narrow the possible causes and provide some direction to the investigation.

With the aid of the following steps:

http://etutorials.org/SQL/Postgresql/Part+I+General+PostgreSQL+Use/Chapter+4.+Performance/How+Postgr...

I was able to determine that in one of my test hosts, the "metrics" table in my "scm" database is taking the most space on disk, so I recommend taking a look at that to find out what database and table are growing.

[#11]

I have tried to figure out what is that oid maps to which database. However, I can't enter the PostgreSQL console.

When I run "psql" in the terminal, it return

psql: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

It seems that the embeded PostgreSQL is different from the native one, so all the solutions I found on the web don't work.

Any suggestion for this problem?

Here is some info that maybe helpful.

When I run "ps -elf | grep postmaster", it return

0 S 101      30238     1  0  75   0 - 540004 -     Jun19 ?        00:00:01 /usr/bin/postmaster -D /var/lib/cloudera-scm-server-db/data

When I run "ps -elf | grep post", it return

1 S 101       1255 30238  0  76   0 - 540765 -     10:08 ?        00:00:08 postgres: smon smon 192.168.28.40(33310) idle              
1 S 101       1256 30238  0  76   0 - 540749 -     10:08 ?        00:00:08 postgres: smon smon 192.168.28.40(33311) idle              
1 S 101       2085 30238  0  75   0 - 540765 -     10:23 ?        00:00:01 postgres: hmon hmon 192.168.28.40(33932) idle              
1 S 101       2086 30238  0  76   0 - 540772 -     10:23 ?        00:00:01 postgres: hmon hmon 192.168.28.40(33933) idle              
1 S 101       2932 30238  0  75   0 - 540765 -     10:38 ?        00:00:00 postgres: hmon hmon 192.168.28.40(58484) idle              
1 S 101       2935 30238  0  75   0 - 540749 -     10:38 ?        00:00:05 postgres: smon smon 192.168.28.40(58494) idle              
1 S 101       2936 30238  0  76   0 - 540749 -     10:38 ?        00:00:00 postgres: amon amon 192.168.28.40(58496) idle              
1 S 101       2937 30238  0  76   0 - 540749 -     10:38 ?        00:00:00 postgres: amon amon 192.168.28.40(58497) idle              
1 S 101       3804 30238  0  76   0 - 540749 -     10:53 ?        00:00:00 postgres: smon smon 192.168.28.40(33735) idle              
1 S 101       3806 30238  0  75   0 - 540772 -     10:53 ?        00:00:04 postgres: smon smon 192.168.28.40(33736) idle              
1 S 101       3807 30238  0  75   0 - 540778 -     10:53 ?        00:00:04 postgres: smon smon 192.168.28.40(33737) idle              
1 S 101       3808 30238  0  75   0 - 540765 -     10:53 ?        00:00:00 postgres: amon amon 192.168.28.40(33739) idle              
1 S 101       3809 30238  0  76   0 - 540749 -     10:53 ?        00:00:00 postgres: amon amon 192.168.28.40(33740) idle              
1 S 101       4718 30238  0  76   0 - 540749 -     11:08 ?        00:00:00 postgres: hmon hmon 192.168.28.40(38106) idle              
1 S 101       4719 30238  0  75   0 - 540749 -     11:08 ?        00:00:00 postgres: hmon hmon 192.168.28.40(38107) idle              
1 S 101       4720 30238  0  75   0 - 540765 -     11:08 ?        00:00:00 postgres: hmon hmon 192.168.28.40(38108) idle              
1 S 101       4722 30238  0  76   0 - 540765 -     11:08 ?        00:00:08 postgres: smon smon 192.168.28.40(38117) idle              
1 S 101       4723 30238  0  75   0 - 540772 -     11:08 ?        00:00:07 postgres: smon smon 192.168.28.40(38118) idle              
1 S 101       4724 30238  0  76   0 - 540749 -     11:08 ?        00:00:00 postgres: amon amon 192.168.28.40(38120) idle              
1 S 101       4725 30238  0  75   0 - 540749 -     11:08 ?        00:00:00 postgres: amon amon 192.168.28.40(38121) idle              
1 S 101       5637 30238  0  75   0 - 540765 -     11:23 ?        00:00:00 postgres: hmon hmon 192.168.28.40(33133) idle              
1 S 101       5638 30238  0  75   0 - 540660 -     11:23 ?        00:00:00 postgres: hmon hmon 192.168.28.40(33134) idle              
1 S 101       5639 30238  0  75   0 - 540749 -     11:23 ?        00:00:00 postgres: hmon hmon 192.168.28.40(33135) idle              
1 S 101       5641 30238  0  75   0 - 540749 -     11:23 ?        00:00:00 postgres: amon amon 192.168.28.40(33144) idle              
1 S 101       5642 30238  0  76   0 - 540749 -     11:23 ?        00:00:00 postgres: amon amon 192.168.28.40(33145) idle              
1 S 101       6789 30238  0  75   0 - 540749 -     11:38 ?        00:00:00 postgres: hmon hmon 192.168.28.40(45206) idle              
1 S 101       6791 30238  0  75   0 - 540749 -     11:38 ?        00:00:00 postgres: smon smon 192.168.28.40(45215) idle              
1 S 101       6792 30238  0  75   0 - 540749 -     11:38 ?        00:00:00 postgres: smon smon 192.168.28.40(45216) idle              
1 S 101       6793 30238  0  76   0 - 540749 -     11:38 ?        00:00:00 postgres: amon amon 192.168.28.40(45218) idle              
1 S 101       6794 30238  0  76   0 - 540749 -     11:38 ?        00:00:00 postgres: amon amon 192.168.28.40(45219) idle              
4 S root      6938  4666  0  78   0 - 15306 pipe_w 11:40 pts/0    00:00:00 grep post
1 S 101      19728 30238  0  75   0 - 540939 -     Jun19 ?        00:00:00 postgres: hive hive 192.168.28.40(55990) idle              
1 S 101      19730 30238  0  75   0 - 540642 -     Jun19 ?        00:00:00 postgres: hive hive 192.168.28.40(55993) idle              
4 S 101      21214  7449  0  80   0 - 204323 184466 Jun19 ?       00:12:37 /usr/java/jdk1.6.0_31/bin/java -server -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt1-HOSTMONITOR-master.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -Xms274384019 -Xmx274384019 -cp /var/run/cloudera-scm-agent/process/35-cloudera-mgmt-HOSTMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/* com.cloudera.cmon.firehose.Main --pipeline-type HOST_MONITORING --mgmt-home /usr/share/cmf
4 S 101      21230  7449  0  82   0 - 181839 futex_ Jun19 ?       00:07:19 /usr/java/jdk1.6.0_31/bin/java -server -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt1-EVENTSERVER-master.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Xms274384019 -Xmx274384019 -cp /var/run/cloudera-scm-agent/process/36-cloudera-mgmt-EVENTSERVER:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/* com.cloudera.cmf.eventcatcher.server.EventCatcherService
4 S 101      21258  7449  0  80   0 - 182312 futex_ Jun19 ?       00:08:11 /usr/java/jdk1.6.0_31/bin/java -server -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt1-ACTIVITYMONITOR-master.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -Xms274384019 -Xmx274384019 -cp /var/run/cloudera-scm-agent/process/37-cloudera-mgmt-ACTIVITYMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/* com.cloudera.cmon.firehose.Main --pipeline-type ACTIVITY_MONITORING_TREE --mgmt-home /usr/share/cmf
4 S 101      21280  7449  0  83   0 - 186964 184466 Jun19 ?       00:01:22 /usr/java/jdk1.6.0_31/bin/java -server -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt1-ALERTPUBLISHER-master.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Xms268435456 -Xmx268435456 -cp /var/run/cloudera-scm-agent/process/38-cloudera-mgmt-ALERTPUBLISHER:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/* com.cloudera.enterprise.alertpublisher.AlertPublisher
4 S 101      21296  7449  2  80   0 - 214066 184466 Jun19 ?       00:39:14 /usr/java/jdk1.6.0_31/bin/java -server -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt1-SERVICEMONITOR-master.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -XX:PermSize=128m -Dsun.rmi.transport.tcp.handshakeTimeout=10000 -Dsun.rmi.transport.tcp.responseTimeout=10000 -Xms274384019 -Xmx274384019 -cp /var/run/cloudera-scm-agent/process/39-cloudera-mgmt-SERVICEMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/* com.cloudera.cmon.firehose.Main --pipeline-type SERVICE_MONITORING --mgmt-home /usr/share/cmf
1 S 101      22524 30238  0  75   0 - 541013 -     Jun19 ?        00:00:44 postgres: scm scm 127.0.0.1(43782) idle                    
1 S 101      22525 30238  0  75   0 - 540876 -     Jun19 ?        00:00:19 postgres: scm scm 127.0.0.1(43783) idle                    
1 S 101      22526 30238  0  75   0 - 541243 -     Jun19 ?        00:00:39 postgres: scm scm 127.0.0.1(43784) idle                    
0 S 101      30238     1  0  75   0 - 540004 -     Jun19 ?        00:00:01 /usr/bin/postmaster -D /var/lib/cloudera-scm-server-db/data
1 S 101      30243 30238  0  75   0 - 27484 -      Jun19 ?        00:00:00 postgres: logger process                                   
1 S 101      30246 30238  0  75   0 - 540282 -     Jun19 ?        00:00:19 postgres: writer process                                   
1 S 101      30247 30238  0  75   0 - 28251 -      Jun19 ?        00:00:00 postgres: stats buffer process                             
1 S 101      30248 30247  0  75   0 - 28169 -      Jun19 ?        00:00:00 postgres: stats collector process                          
1 S 101      30291 30238  0  76   0 - 541311 -     Jun19 ?        00:00:21 postgres: scm scm 127.0.0.1(39839) idle                    
1 S 101      30292 30238  0  76   0 - 541536 -     Jun19 ?        00:00:25 postgres: scm scm 127.0.0.1(39840) idle                    
1 S 101      30293 30238  0  76   0 - 541180 -     Jun19 ?        00:00:06 postgres: scm scm 127.0.0.1(39841) idle                    
1 S 101      30294 30238  0  76   0 - 541260 -     Jun19 ?        00:00:20 postgres: scm scm 127.0.0.1(39842) idle                    
1 S 101      30295 30238  0  75   0 - 541407 -     Jun19 ?        00:00:25 postgres: scm scm 127.0.0.1(39843) idle

[#12]

Finally I found the solution in cloudera docunment !!!

I run "psql -U smon -p7432" and enter the console.

In the large-size directory /var/lib/cloudera-scm-server-db/data/base/16387, there are 872 files.

The largest files are 24M, and there are about 150 such files.

Here are relname of these oids.

         relname          |  oid  
--------------------------+-------
 cmon_ll_dp_2014_06_20_11 | 37437

 cmon_ll_dp_2014_06_20_10 | 37415

 cmon_ll_dp_2014_06_20_09 | 37393

Any idea about what it is? Thanks so much for your help !

[#13 · Accepted solution]

In the grand scheme, you should generally never, ever directly delete files on-disk that pertain to a database's data-files. It is advised to instead determine what application or process is writing to the database, and inserting that data. The appropriate place to address this is through the application that is writing them, not by deleting on-disk, or even by logging into postgres with psql and manually performing any activity.

In this case, I am fairly sure your Cloudera Manager deployment is using postgres running on port 7432, to house these databases:

Cloudera Manager

Activity Monitor

Service Monitor

Host Monitor

Reports Manager

None of these should be manually altered by logging into the database.

This is relevant for Cloudera Manager 4.x only:

Cause: 

Cloudera Manager's Management Services use various databases to store their gathered data. These databases should be located where sufficient space is available to accommodate their growth. Without proper consideration, their default locations could be in a location with insufficient space and a volume could fill to 100%.

Instructions

The rate of growth for these databases is solely controlled via purge/expiration configuration in each of the services (Activity Monitor, Service Monitor, Host Monitor) individually. This controls how many hours/days worth of monitoring data are kept, and will directly then influence how large the databases will grow. Use these settings to apply bound to their growth:

1. Host Monitor:
        * Host Monitor Data Expiration Period (hours) (default 168 hours, or 7 days)

2. Service Monitor:
        * Service Monitor Data Expiration Period (hours) (default 168 hours, or 7 days)

3. Activity Monitor:
        * Purge Activities Data at This Age (default 336 hours, or 14 days)
        * Purge Attempts Data at This Age (default 336 hours, or 14 days)
        * Purge MapReduce Service Data at This Age (default 336 hours, or 14 days)

You can adjust these values downward, save them, and then in the background services will expire data older than the new value you set.

For example, if you are using the cloudera-scm-server-db ("embedded postgres") database for Cloudera Manager, all these databases will consume space in the default path /var/lib/cloudera-scm-server-db/data. You may wish to halve the retention periods for all the above services so this location does not fill up.

This is also discussed briefly under the header titled "Configuring Cloudera Management Services Database Limits" in our documentation:http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.5.4/Cloudera-Manager-Enterpr...

TL;DR:

Please find and adjust the specific Cloudera Manager Management Services purging | expiration tunables that assist in controlling the size of the databases on-disk, or allocate more space to the partition where /var resides.

[#14]

@smark, thanks so much for your suggestion!
After adjusting those parameters and restart Cloudera Manager service, the size of that directory comes to 1G.

[#15]

Can we delete files/directory in /var/lib/cloudera-scm-server/commands ??

[#16]

@pchauras

Yes, you can but don't delete latest one from this dir because you will not be able to see recent commands in CM Web UI.

This command dir can be controlled by CM > Administration > Setting > Search for "Command Eviction Age"

And set it to some lower value like 90 days. 

To know more follow : https://www.cloudera.com/documentation/enterprise/5-5-x/topics/cm_ig_reqs_space.html#concept_tjd_4yc...

[#17]

After changing value of "Command Eviction Age" , does it requires restart of any service?

[#18]

@shravani CM server restart required. 

systemctl restart cloudera-scm-server