HDFS Disk Usage is 98% full

[#1]

I am using the "Hortonworks Data Platform (HDP®) 3.0.1 on Hortonworks Sandbox" using Docker in Hyper-V on Windows 10 1803 box. Pleas note have lots of disk space. memory, and cores on this machine. I have scan the forums and web so far nothing has worked. Here is the output from df -h:

[root@sandbox-hdp ~]# df -h
Filesystem  Size  Used Avail Use% Mounted on
overlay      59G   55G  1.4G  98% /
tmpfs        64M     0   64M   0% /dev
/dev/sda1    59G   55G  1.4G  98% /etc/hosts
shm          64M  4.0K   64M   1% /dev/shm
tmpfs       9.7G  355M  9.4G   4% /run
tmpfs       2.0G     0  2.0G   0% /run/user/1003
tmpfs       2.0G     0  2.0G   0% /run/user/0
tmpfs       2.0G     0  2.0G   0% /run/user/1004
tmpfs       2.0G     0  2.0G   0% /run/user/1011
tmpfs       2.0G     0  2.0G   0% /run/user/1009
tmpfs       2.0G     0  2.0G   0% /run/user/1019
tmpfs       2.0G     0  2.0G   0% /run/user/1002
tmpfs       2.0G     0  2.0G   0% /run/user/1021
tmpfs       2.0G     0  2.0G   0% /run/user/1001
tmpfs       2.0G     0  2.0G   0% /run/user/1017
tmpfs       2.0G     0  2.0G   0% /run/user/1005
tmpfs       2.0G     0  2.0G   0% /run/user/1023
tmpfs       2.0G     0  2.0G   0% /run/user/1000
tmpfs       2.0G     0  2.0G   0% /run/user/1012
tmpfs       2.0G     0  2.0G   0% /run/user/1013

It appears this line "/dev/sda1 59G 55G 1.4G 98% /etc/hosts" is the issue. So far haven't found any docs on to address space issue with that piece of the configuration.

Appreciate any help on get on this. I have just started using the sandbox when it ran out of space after about a week of running idle

[#2 · Accepted solution]

@Richard Wheeler
if you left the sandbox idling then for sure it MUST be the logs generated in /var/logs/{component}/ the HDP components continually generate logs with the components statuses and on the sandbox, it's mount on /

# du -a /var/log/ | sort -n -r | head -n 20 

Sample output

3363560 /var/log/
1966344 /var/log/kafka
494300  /var/log/ambari-metrics-collector
267092  /var/log/hadoop
265560  /var/log/hadoop/hdfs
171528  /var/log/hadoop/hdfs/hadoop-hdfs-namenode-test.tarta.se.log
159432  /var/log/ambari-agent
98756   /var/log/ambari-infra-solr
81932   /var/log/ambari-metrics-collector/ambari-metrics-collector.log.3
81932   /var/log/ambari-metrics-collector/ambari-metrics-collector.log.2
81932   /var/log/ambari-metrics-collector/ambari-metrics-collector.log.1
81928   /var/log/ambari-metrics-collector/ambari-metrics-collector.log.4
81924   /var/log/ambari-metrics-collector/ambari-metrics-collector.log.5
69956   /var/log/oozie
49116   /var/log/hbase
40056   /var/log/ranger
39136   /var/log/ranger/admin
36420   /var/log/hive
36232   /var/log/hadoop-yarn
36176   /var/log/hbase/hbase-hbase-regionserver-test.tarta.se.log

So you will need to delete the old files to regain some space. You can also run discretely a script in the cron !