Could I delete files in /var/lib?

[#1]

I'm running a one-note cdh4 system for test.

It's found that the size of /var/lib increases quickly. Could I delete those files?

[#2]

Hello,

Cloudera Manager stores information there, so deleting it would not be advised.

Check what directory or directories grow with something like:

du -sh /var/lib/* |sort -n

If you can isolate what is growing, perhaps we can suggest a way to limit that growth.

Ben

[#3]

/var/lib is usually used to store the state of the system. So, for example, if you have namenode running on a machine, the metadata for the namenode is written in that directory. Formatting the namenode will clean out a subdirectory of /var/lib, so in general, it's not a good idea to delete those files.

You should look a little more deeply into what's making that directory fill up. If they look like logs, it's likely ok to delete them but most of that directory contains things you don't want to delete from a functioning cluster.

[#4]

Thanks for reply.

I followed your suggestion and found that the size-growing diretory is "/var/lib/cloudera-scm-server-db/data/base"

Here is the detail size of directories in /var/lib/cloudera-scm-server-db/data/base

3.7M    ./1
3.7M    ./10792
3.7M    ./10793
8.3M    ./16385
8.7G    ./16387
84M    ./16389
911M    ./16391
3.7M    ./16393
3.7M    ./16395
5.7M    ./17569

Could this directories be deleted?

[#5]

Those files are actually your embedded PostgreSQL database that contains your Cloudera Manager data, so deleting would be highly destructive.

The embedded PostgreSQL is only intended for demonstration, so the way to go for you may be to configure an external database as described in:

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installat...

If you want to dig in and find out what is actually being stored in the PostgreSQL directories, I found this page that walks you through navigating the various subdirectories:

http://etutorials.org/SQL/Postgresql/Part+I+General+PostgreSQL+Use/Chapter+4.+Performance/How+Postgr...

-Ben

[#6]

If I format the hdfs, will these files clean up as @mgrover said?

If this is the case, how could I format the hdfs? Use the format command just as native hadoop?

[#7]

The point that mgrover was making in the previous post was that you should not delete anything.  Formatting will essentially destroy your hdfs data.

I think a good rule here is to not delete /var/lib data as you will likely corrupt or destroy key parts of your cluster.  In production, you would have an external database and monitor servers on other hosts, storing data in other directories.  There may be ways to free up space in your existing install, but none are trivial or without consequence.

Ben

[#8]

Exactly. All I was trying to do say that removing /var/lib files will
format your namenode. But there are definitely more files from many
different components (from CDH and Cloudera Manager) that put files in
/var/lib so formatting namenode is not necessarily going to help (and it's
going to delete all your HDFS data). In fact, looking at the sizes of
content in that directory, it's likely not going to help.

[#9]

The size of this directory is growing so quickly, about 1G per day even nothing has been done, which is unacceptable for a test machine.

Is such a growing speed normal?

Since this is a system for test, data in it is not so important.

Are there any simple ways in which I could refresh the cloudera system and clean up these files? Or I can only reinstall cloudera manager periodically?

[#10]

Without knowing what is growing or why, it is hard to say if this is normal or expected.  For an idle server with no notable problems and a few quiet nodes, I would not expect such growth. 

I would suggest finding out exactly what disk directory is growing the most; if it is /var/lib/cloudera-scm-server-db/data/base/16387 as you mentioned earlier, I would find out to which database that oid maps to.  Knowing what database is growing will at least narrow the possible causes and provide some direction to the investigation.

With the aid of the following steps:

http://etutorials.org/SQL/Postgresql/Part+I+General+PostgreSQL+Use/Chapter+4.+Performance/How+Postgr...

I was able to determine that in one of my test hosts, the "metrics" table in my "scm" database is taking the most space on disk, so I recommend taking a look at that to find out what database and table are growing.