How to fix host disk Consumptions

[#1]

I am getting alert in Ambari for Host disk full:

I checked my machine disk space, it's showing 100% full but i am not sure what should i delete as i have already deleted logs which were greater then 100MB size. But it's still showing same.

Can anyone help me in this? I am using kafka, hbase and spark-streaming.

Please find screenshot attached of my disk usage:

screen-shot-2016-03-06-at-13742-pm.png

screen-shot-2016-03-06-at-14231-pm.png

Regards,

~Vinti

[#2]

use the following two commands to investigate space issues

df -h
du -hs /*
du -hs /root/*
du -hs /var/log/*

[#3]

You can continue calling du on directories until you find directory withost space used.

[#4]

@Artem Ervits @Vinti Maheshwari As Artem suggested, you need to check which folder is using space.

First, figure out which components are running on this node. Then look at log directory for each component to determine which component is using most disk space. If there are no historical logs, you may need to limit log size.

[#5 · Accepted solution]

@Vinti Maheshwari

Start with /tmp

du -sh *

Remove files and directories from /tmp that are large in size and you don't need

/var/log -->

du -sh *

Pick big size directories and see whats consuming more space

As per screenshot , you are out of space on root i,e /

cd /

ls -lrth

look for any zip, .gz , rpm or any other files that you don't need and if you need them then move to other file system

[#6]

find / -size +1000000k will print out the results of a find command, but only those files found that are larger than .95 Gb

[#7]

a very useful cli tool for analyzing disk usage is ncdu: https://dev.yorhel.nl/ncdu

[#8]

Is /usr/hdp on root?

If you are using a GCE cluster you can setup such that /usr/hdp is on larger drive

# On each VM,

mkdir /grid/0/hdp

mkdir /usr/hdp

mount --bind /grid/0/hdp /usr/hdp

If you have done upgrades in the past and dont intend to go back to the older version, you could remove the old version bits from /usr/hdp/old-version

[#9]

/var/log/ Full and ?var/lib/ Full Out of Memory any suggestions to remove what files need to remove without affceting Cluster