question Re: HDFS capacity is 0 but all DataNode are live in Archives of Support Questions (Read Only)

HDFS capacity is 0 but all DataNode are live

yunding — Sun, 18 Aug 2019 08:00:08 GMT

I have 2 DataNodes and both are live. However, the dashboard shows that the HDFS Disk Usage is n/a, and capacity is zero (see the screenshot).

I tried to put some files to hdfs through "hadoop fs -put hello.txt /", and got the error

 File /hello.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 2 datanode(s) running and no node(s) are excluded in this operation.

I guess this means the NameNode knows the existence of 2 DataNodes, and none is excluded from the "-put" operation?

I have checked the "dfs.datanode.data.dir", and it is pointing to the correct directory with 500G available space.

How can I resolve this issue?

Re: HDFS capacity is 0 but all DataNode are live

vmurakami — Fri, 06 Jul 2018 13:10:16 GMT

Hey @Yun Ding !
Could you check the following outputs?

hdfs dfs -du -h /
hdfs dfsadmin -report 
lsblk
df -h

And also check the value for this parameter on Ambari:
dfs.datanode.du.reserved

PS: Just in case, check the permission for the dfs.datanode.data.dir directory, should it be owned by hdfs:hadoop.

Hope this helps!

Re: HDFS capacity is 0 but all DataNode are live

yunding — Fri, 06 Jul 2018 22:07:03 GMT

@Vinicius Higa Murakami

it was due to the permission for the dfs.datanode.data.dir directory. Thanks!

Re: HDFS capacity is 0 but all DataNode are live

vmurakami — Fri, 06 Jul 2018 22:56:09 GMT

Good to know! 🙂

Re: HDFS capacity is 0 but all DataNode are live

macweiyanen — Fri, 14 Jan 2022 06:28:01 GMT

Hi there,

is it dfs_datanode_data_dir_perm?

what's your previous value for it when it couldn't write?