Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How Clean HDFS in ambari?

avatar

Hi, I've been starting with Ambari and Sandbox Horton works, but I have a problem.

I've been developing some MapReduce and Pig script to test the framework and I always do rm in the result output in order to not waste space disk, but with the time (1 month) the hdfs disk is fully in the ambari metrics and I don't recovery space spite of remove and remove files.

4749-sin-titu1lo.jpg

1 ACCEPTED SOLUTION

avatar
Super Collaborator

@BRivas garriv

It takes some manual effort but, first change the working directory to / (cd /).

then do du -h on every folder to find the disk usage. (Example: "du -h var" or "du -h usr").

Locate the folder which is taking up all disk space and try to delete irrelevant files from that folder.

View solution in original post

5 REPLIES 5

avatar
Expert Contributor

If you do just rm you're actually moving your data to the Trash. In order to remove the data from HDFS and free space, when you do the rm you have to put the flag -skipTrash.

In order to delete the data from the trash, you can run:

hdfs dfs -expunge

avatar

Hi, thanks for the response. I´ve try but dont work. The problem is that the disk is filling while it is on but not store results. I think it may be because that is not where logs can be store, that is, every day metrics mark a 1 % stored more despite I dont use the machine. And now with the time is 99%, yesterday have 98% like show the picture.

4748-sin-titulo.jpg

avatar
Super Collaborator

@BRivas garriv

It takes some manual effort but, first change the working directory to / (cd /).

then do du -h on every folder to find the disk usage. (Example: "du -h var" or "du -h usr").

Locate the folder which is taking up all disk space and try to delete irrelevant files from that folder.

avatar

Thanks you so much, its works. The problem is that Atlas was comsuming 25GB of disk space in logs. You have any idea why this can happend? I never use Atlas app.

avatar