question How Clean HDFS in ambari? in Archives of Support Questions (Read Only)

How Clean HDFS in ambari?

bibiano_rivas_g — Mon, 19 Aug 2019 11:07:49 GMT

Hi, I've been starting with Ambari and Sandbox Horton works, but I have a problem.

I've been developing some MapReduce and Pig script to test the framework and I always do rm in the result output in order to not waste space disk, but with the time (1 month) the hdfs disk is fully in the ambari metrics and I don't recovery space spite of remove and remove files.

Re: How Clean HDFS in ambari?

mgaido — Wed, 01 Jun 2016 17:33:11 GMT

If you do just rm you're actually moving your data to the Trash. In order to remove the data from HDFS and free space, when you do the rm you have to put the flag -skipTrash.

In order to delete the data from the trash, you can run:

hdfs dfs -expunge

Re: How Clean HDFS in ambari?

bibiano_rivas_g — Mon, 19 Aug 2019 11:07:41 GMT

Hi, thanks for the response. I´ve try but dont work. The problem is that the disk is filling while it is on but not store results. I think it may be because that is not where logs can be store, that is, every day metrics mark a 1 % stored more despite I dont use the machine. And now with the time is 99%, yesterday have 98% like show the picture.

Re: How Clean HDFS in ambari?

mrizvi — Sat, 04 Jun 2016 01:59:09 GMT

@BRivas garriv

It takes some manual effort but, first change the working directory to / (cd /).

then do du -h on every folder to find the disk usage. (Example: "du -h var" or "du -h usr").

Locate the folder which is taking up all disk space and try to delete irrelevant files from that folder.

Re: How Clean HDFS in ambari?

bibiano_rivas_g — Sat, 04 Jun 2016 03:03:14 GMT

Thanks you so much, its works. The problem is that Atlas was comsuming 25GB of disk space in logs. You have any idea why this can happend? I never use Atlas app.

Re: How Clean HDFS in ambari?

divakarreddy_a — Sat, 04 Jun 2016 03:42:21 GMT

please check here:

https://community.hortonworks.com/questions/5148/atlas-generating-huge-log-files.html