HDFS capacity is 0 but all DataNode are live

[#1]

I have 2 DataNodes and both are live. However, the dashboard shows that the HDFS Disk Usage is n/a, and capacity is zero (see the screenshot).

I tried to put some files to hdfs through "hadoop fs -put hello.txt /", and got the error

 File /hello.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 2 datanode(s) running and no node(s) are excluded in this operation.

I guess this means the NameNode knows the existence of 2 DataNodes, and none is excluded from the "-put" operation?

I have checked the "dfs.datanode.data.dir", and it is pointing to the correct directory with 500G available space.

How can I resolve this issue?

[#2 · Accepted solution]

Hey @Yun Ding !
Could you check the following outputs?

hdfs dfs -du -h /
hdfs dfsadmin -report 
lsblk
df -h

And also check the value for this parameter on Ambari:
dfs.datanode.du.reserved

PS: Just in case, check the permission for the dfs.datanode.data.dir directory, should it be owned by hdfs:hadoop.

Hope this helps!

[#3]

@Vinicius Higa Murakami

it was due to the permission for the dfs.datanode.data.dir directory. Thanks!

[#4]

Good to know! 🙂

[#5]

Hi there,

is it dfs_datanode_data_dir_perm?

what's your previous value for it when it couldn't write?