question Re: How to fix host disk Consumptions in Archives of Support Questions (Read Only)

How to fix host disk Consumptions

vinti_uiet — Mon, 07 Mar 2016 05:41:52 GMT

I am getting alert in Ambari for Host disk full:

I checked my machine disk space, it's showing 100% full but i am not sure what should i delete as i have already deleted logs which were greater then 100MB size. But it's still showing same.

Can anyone help me in this? I am using kafka, hbase and spark-streaming.

Please find screenshot attached of my disk usage:

screen-shot-2016-03-06-at-13742-pm.png

screen-shot-2016-03-06-at-14231-pm.png

Regards,

~Vinti

Re: How to fix host disk Consumptions

aervits — Mon, 07 Mar 2016 06:29:01 GMT

use the following two commands to investigate space issues

df -h
du -hs /*
du -hs /root/*
du -hs /var/log/*

Re: How to fix host disk Consumptions

aervits — Mon, 07 Mar 2016 06:29:40 GMT

You can continue calling du on directories until you find directory withost space used.

Re: How to fix host disk Consumptions

shishir_saxena4 — Mon, 07 Mar 2016 10:34:39 GMT

@Artem Ervits @Vinti Maheshwari As Artem suggested, you need to check which folder is using space.

First, figure out which components are running on this node. Then look at log directory for each component to determine which component is using most disk space. If there are no historical logs, you may need to limit log size.

Re: How to fix host disk Consumptions

nsabharwal — Mon, 07 Mar 2016 10:54:27 GMT

@Vinti Maheshwari

Start with /tmp

du -sh *

Remove files and directories from /tmp that are large in size and you don't need

/var/log -->

du -sh *

Pick big size directories and see whats consuming more space

As per screenshot , you are out of space on root i,e /

cd /

ls -lrth

look for any zip, .gz , rpm or any other files that you don't need and if you need them then move to other file system

Re: How to fix host disk Consumptions

nsabharwal — Mon, 07 Mar 2016 11:06:54 GMT

find / -size +1000000k will print out the results of a find command, but only those files found that are larger than .95 Gb

Re: How to fix host disk Consumptions

michael1 — Mon, 07 Mar 2016 13:49:12 GMT

a very useful cli tool for analyzing disk usage is ncdu: https://dev.yorhel.nl/ncdu

Re: How to fix host disk Consumptions

jluniya — Tue, 08 Mar 2016 03:16:10 GMT

Is /usr/hdp on root?

If you are using a GCE cluster you can setup such that /usr/hdp is on larger drive

# On each VM,

mkdir /grid/0/hdp

mkdir /usr/hdp

mount --bind /grid/0/hdp /usr/hdp

If you have done upgrades in the past and dont intend to go back to the older version, you could remove the old version bits from /usr/hdp/old-version

Re: How to fix host disk Consumptions

sunsrk109 — Fri, 12 Aug 2016 09:32:34 GMT

/var/log/ Full and ?var/lib/ Full Out of Memory any suggestions to remove what files need to remove without affceting Cluster