Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Could I delete files in /var/lib?

avatar
Contributor

I'm running a one-note cdh4 system for test.

 

It's found that the size of /var/lib increases quickly. Could I delete those files?

1 ACCEPTED SOLUTION

avatar
Super Collaborator

In the grand scheme, you should generally never, ever directly delete files on-disk that pertain to a database's data-files. It is advised to instead determine what application or process is writing to the database, and inserting that data. The appropriate place to address this is through the application that is writing them, not by deleting on-disk, or even by logging into postgres with psql and manually performing any activity.

 

In this case, I am fairly sure your Cloudera Manager deployment is using postgres running on port 7432, to house these databases:

Cloudera Manager

Activity Monitor

Service Monitor

Host Monitor

Reports Manager

 

None of these should be manually altered by logging into the database.

 

This is relevant for Cloudera Manager 4.x only:

Cause: 
Cloudera Manager's Management Services use various databases to store their gathered data. These databases should be located where sufficient space is available to accommodate their growth. Without proper consideration, their default locations could be in a location with insufficient space and a volume could fill to 100%.
Instructions
The rate of growth for these databases is solely controlled via purge/expiration configuration in each of the services (Activity Monitor, Service Monitor, Host Monitor) individually. This controls how many hours/days worth of monitoring data are kept, and will directly then influence how large the databases will grow. Use these settings to apply bound to their growth:

1. Host Monitor:
        * Host Monitor Data Expiration Period (hours) (default 168 hours, or 7 days)

2. Service Monitor:
        * Service Monitor Data Expiration Period (hours) (default 168 hours, or 7 days)

3. Activity Monitor:
        * Purge Activities Data at This Age (default 336 hours, or 14 days)
        * Purge Attempts Data at This Age (default 336 hours, or 14 days)
        * Purge MapReduce Service Data at This Age (default 336 hours, or 14 days)

You can adjust these values downward, save them, and then in the background services will expire data older than the new value you set.

For example, if you are using the cloudera-scm-server-db ("embedded postgres") database for Cloudera Manager, all these databases will consume space in the default path /var/lib/cloudera-scm-server-db/data. You may wish to halve the retention periods for all the above services so this location does not fill up.

This is also discussed briefly under the header titled "Configuring Cloudera Management Services Database Limits" in our documentation:http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.5.4/Cloudera-Manager-Enterpr...

 

 

TL;DR:

Please find and adjust the specific Cloudera Manager Management Services purging | expiration tunables that assist in controlling the size of the databases on-disk, or allocate more space to the partition where /var resides.

 

 

View solution in original post

17 REPLIES 17

avatar
Master Guru

Hello,

 

Cloudera Manager stores information there, so deleting it would not be advised.

 

Check what directory or directories grow with something like:

 

du -sh /var/lib/* |sort -n

 

If you can isolate what is growing, perhaps we can suggest a way to limit that growth.

 

Ben

 

avatar
Expert Contributor

/var/lib is usually used to store the state of the system. So, for example, if you have namenode running on a machine, the metadata for the namenode is written in that directory. Formatting the namenode will clean out a subdirectory of /var/lib, so in general, it's not a good idea to delete those files.

 

You should look a little more deeply into what's making that directory fill up. If they look like logs, it's likely ok to delete them but most of that directory contains things you don't want to delete from a functioning cluster.

avatar
Contributor

Thanks for reply.

I followed your suggestion and found that the size-growing diretory is "/var/lib/cloudera-scm-server-db/data/base"

Here is the detail size of directories in /var/lib/cloudera-scm-server-db/data/base

3.7M    ./1
3.7M    ./10792
3.7M    ./10793
8.3M    ./16385
8.7G    ./16387
84M    ./16389
911M    ./16391
3.7M    ./16393
3.7M    ./16395
5.7M    ./17569

 

Could this directories be deleted?

avatar
Master Guru

Those files are actually your embedded PostgreSQL database that contains your Cloudera Manager data, so deleting would be highly destructive.

 

The embedded PostgreSQL is only intended for demonstration, so the way to go for you may be to configure an external database as described in:

 

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installat...

 

If you want to dig in and find out what is actually being stored in the PostgreSQL directories, I found this page that walks you through navigating the various subdirectories:

 

http://etutorials.org/SQL/Postgresql/Part+I+General+PostgreSQL+Use/Chapter+4.+Performance/How+Postgr...

 

-Ben

 

avatar
Contributor
If I format the hdfs, will these files clean up as @mgrover said?

If this is the case, how could I format the hdfs? Use the format command just as native hadoop?

avatar
Master Guru

The point that mgrover was making in the previous post was that you should not delete anything.  Formatting will essentially destroy your hdfs data.

 

I think a good rule here is to not delete /var/lib data as you will likely corrupt or destroy key parts of your cluster.  In production, you would have an external database and monitor servers on other hosts, storing data in other directories.  There may be ways to free up space in your existing install, but none are trivial or without consequence.

 

Ben

 

avatar
Expert Contributor
Exactly. All I was trying to do say that removing /var/lib files will
format your namenode. But there are definitely more files from many
different components (from CDH and Cloudera Manager) that put files in
/var/lib so formatting namenode is not necessarily going to help (and it's
going to delete all your HDFS data). In fact, looking at the sizes of
content in that directory, it's likely not going to help.

avatar
Contributor

The size of this directory is growing so quickly, about 1G per day even nothing has been done, which is unacceptable for a test machine.

 

Is such a growing speed normal?

 

Since this is a system for test, data in it is not so important.

Are there any simple ways in which I could refresh the cloudera system and clean up these files? Or I can only reinstall cloudera manager periodically?

 

 

avatar
Master Guru

Without knowing what is growing or why, it is hard to say if this is normal or expected.  For an idle server with no notable problems and a few quiet nodes, I would not expect such growth. 

 

I would suggest finding out exactly what disk directory is growing the most; if it is /var/lib/cloudera-scm-server-db/data/base/16387 as you mentioned earlier, I would find out to which database that oid maps to.  Knowing what database is growing will at least narrow the possible causes and provide some direction to the investigation.

 

With the aid of the following steps:

 

http://etutorials.org/SQL/Postgresql/Part+I+General+PostgreSQL+Use/Chapter+4.+Performance/How+Postgr...

 

I was able to determine that in one of my test hosts, the "metrics" table in my "scm" database is taking the most space on disk, so I recommend taking a look at that to find out what database and table are growing.