In the grand scheme, you should generally never, ever directly delete files on-disk that pertain to a database's data-files. It is advised to instead determine what application or process is writing to the database, and inserting that data. The appropriate place to address this is through the application that is writing them, not by deleting on-disk, or even by logging into postgres with psql and manually performing any activity.
In this case, I am fairly sure your Cloudera Manager deployment is using postgres running on port 7432, to house these databases:
None of these should be manually altered by logging into the database.
This is relevant for Cloudera Manager 4.x only:
Please find and adjust the specific Cloudera Manager Management Services purging | expiration tunables that assist in controlling the size of the databases on-disk, or allocate more space to the partition where /var resides.
Cloudera Manager stores information there, so deleting it would not be advised.
Check what directory or directories grow with something like:
du -sh /var/lib/* |sort -n
If you can isolate what is growing, perhaps we can suggest a way to limit that growth.
/var/lib is usually used to store the state of the system. So, for example, if you have namenode running on a machine, the metadata for the namenode is written in that directory. Formatting the namenode will clean out a subdirectory of /var/lib, so in general, it's not a good idea to delete those files.
You should look a little more deeply into what's making that directory fill up. If they look like logs, it's likely ok to delete them but most of that directory contains things you don't want to delete from a functioning cluster.
Thanks for reply.
I followed your suggestion and found that the size-growing diretory is "/var/lib/cloudera-scm-server-db/data/base"
Here is the detail size of directories in /var/lib/cloudera-scm-server-db/data/base
Could this directories be deleted?
Those files are actually your embedded PostgreSQL database that contains your Cloudera Manager data, so deleting would be highly destructive.
The embedded PostgreSQL is only intended for demonstration, so the way to go for you may be to configure an external database as described in:
If you want to dig in and find out what is actually being stored in the PostgreSQL directories, I found this page that walks you through navigating the various subdirectories:
The point that mgrover was making in the previous post was that you should not delete anything. Formatting will essentially destroy your hdfs data.
I think a good rule here is to not delete /var/lib data as you will likely corrupt or destroy key parts of your cluster. In production, you would have an external database and monitor servers on other hosts, storing data in other directories. There may be ways to free up space in your existing install, but none are trivial or without consequence.
The size of this directory is growing so quickly, about 1G per day even nothing has been done, which is unacceptable for a test machine.
Is such a growing speed normal?
Since this is a system for test, data in it is not so important.
Are there any simple ways in which I could refresh the cloudera system and clean up these files? Or I can only reinstall cloudera manager periodically?
Without knowing what is growing or why, it is hard to say if this is normal or expected. For an idle server with no notable problems and a few quiet nodes, I would not expect such growth.
I would suggest finding out exactly what disk directory is growing the most; if it is /var/lib/cloudera-scm-server-db/data/base/16387 as you mentioned earlier, I would find out to which database that oid maps to. Knowing what database is growing will at least narrow the possible causes and provide some direction to the investigation.
With the aid of the following steps:
I was able to determine that in one of my test hosts, the "metrics" table in my "scm" database is taking the most space on disk, so I recommend taking a look at that to find out what database and table are growing.