Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to fix host disk Consumptions

avatar
Contributor

I am getting alert in Ambari for Host disk full:

I checked my machine disk space, it's showing 100% full but i am not sure what should i delete as i have already deleted logs which were greater then 100MB size. But it's still showing same.

Can anyone help me in this? I am using kafka, hbase and spark-streaming.

Please find screenshot attached of my disk usage:

screen-shot-2016-03-06-at-13742-pm.png

screen-shot-2016-03-06-at-14231-pm.png

Regards,

~Vinti

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Vinti Maheshwari

Start with /tmp

du -sh *

Remove files and directories from /tmp that are large in size and you don't need

/var/log -->

du -sh *

Pick big size directories and see whats consuming more space

As per screenshot , you are out of space on root i,e /

cd /

ls -lrth

look for any zip, .gz , rpm or any other files that you don't need and if you need them then move to other file system

View solution in original post

8 REPLIES 8

avatar
Master Mentor

use the following two commands to investigate space issues

df -h
du -hs /*
du -hs /root/*
du -hs /var/log/*

avatar
Master Mentor

You can continue calling du on directories until you find directory withost space used.

avatar

@Artem Ervits @Vinti Maheshwari As Artem suggested, you need to check which folder is using space.

First, figure out which components are running on this node. Then look at log directory for each component to determine which component is using most disk space. If there are no historical logs, you may need to limit log size.

avatar
Master Mentor

@Vinti Maheshwari

Start with /tmp

du -sh *

Remove files and directories from /tmp that are large in size and you don't need

/var/log -->

du -sh *

Pick big size directories and see whats consuming more space

As per screenshot , you are out of space on root i,e /

cd /

ls -lrth

look for any zip, .gz , rpm or any other files that you don't need and if you need them then move to other file system

avatar
Master Mentor

find / -size +1000000k will print out the results of a find command, but only those files found that are larger than .95 Gb

avatar
New Contributor

a very useful cli tool for analyzing disk usage is ncdu: https://dev.yorhel.nl/ncdu

avatar
Rising Star

Is /usr/hdp on root?

If you are using a GCE cluster you can setup such that /usr/hdp is on larger drive

# On each VM,

mkdir /grid/0/hdp

mkdir /usr/hdp

mount --bind /grid/0/hdp /usr/hdp

If you have done upgrades in the past and dont intend to go back to the older version, you could remove the old version bits from /usr/hdp/old-version

avatar
New Contributor

/var/log/ Full and ?var/lib/ Full Out of Memory any suggestions to remove what files need to remove without affceting Cluster