hi, I have a hadoop cluster that has the following structure: 1 Name Node, 1 Secondary Name Node, and 10 Data nodes. Each data node has 8 hard disk drives, each having a capacity of 1 TB. Today, we received an alert saying that one hard disk (mounted in /dfs/data01) in a data node, is full. The rest of the disks are using only 3% of their total capacity. Running the balancer makes no difference. Why is it that only one disk is getting full and the others are barely used? Please help.
@Harsh J wrote:
What is the output of below, when run within the DN directory path under /dfs/data01?
find . -type f | grep dncp | xargs ls -lh
Also, what version of CDH5 are you on?
We are also facing the similer issue. One of the disk in the datanode is 100% and all other disks are around 75% used.
Please find the output below.
[root@r03wn17:/data/1]# find . -type f | grep dncp | xargs ls -lh
-rw-r--r-- 1 hdfs hdfs 509G May 17 11:02 ./dfs/dn/current/BP-396968526-10.37.42.241-1400126960303/dncp_block_verification.log.curr
-rw-r--r-- 1 hdfs hdfs 103G Apr 28 22:01 ./dfs/dn/current/BP-396968526-10.37.42.241-1400126960303/dncp_block_verification.log.prev
CDH version :
[root@r03wn17:/data/1]# hadoop version
Subversion http://github.com/cloudera/hadoop -r fe44e341b92cd4e63b4002b87e352f4205487972
Compiled by jenkins on 2015-10-15T00:26Z
Compiled with protoc 2.5.0
From source with checksum c17117c9ef53cb4be302ba73418afbd
This command was run using /opt/cloudera/parcels/CDH-5.3.8-1.cdh5.3.8.p0.5/jars/hadoop-common-2.5.0-cdh5.3.8.jar