question Re: HDFS storage space in Support Questions

HDFS storage space

cc_yang — Tue, 30 Apr 2024 01:51:44 GMT

I used the count command to view the HDFS directory of the cluster, and there is still 9T of storage space left. However, when I upload a 2MB file, I will report that the space is insufficient. Has anyone encountered this situation before

Re: HDFS storage space

cc_yang — Tue, 30 Apr 2024 02:01:37 GMT

My HDFS version number is 3.0.0+cdh6.3.2. Is this a bug?

Re: HDFS storage space

Asfahan — Mon, 13 May 2024 09:59:43 GMT

Hi Team,

Did you use hdfs dfsadmin -report command to check the dfs and non-dfs used, dfs is used by HDFS so i blieve 9TB is complete storage with dfs and nondfs. Now when you copying the file to hdfs have you enable the debug

HADOOP_ROOT_LOGGER=DEBUG,console

Also try to run the below command it will copy the file from local to hdfs

hdfs dfs -Ddfs.replication=#noofdatanodes -put /tmp/<3mbfile> /tmp

Re: HDFS storage space

sathishkr — Thu, 12 Dec 2024 17:48:53 GMT

@cc_yang It could be possible you may have enabled HDFS space quota to the directory and the directory may have reached to it hard limit, causing the file upload throws insufficient space message. Refer more about HDFS quota as below.

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html