Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Data Node host has one disk full

Highlighted

Data Node host has one disk full

New Contributor

hi, I have a hadoop cluster that has the following structure: 1 Name Node, 1 Secondary Name Node, and 10 Data nodes. Each data node has 8 hard disk drives, each having a capacity of 1 TB. Today, we received an alert saying that one hard disk (mounted in /dfs/data01) in a data node, is full. The rest of the disks are using only 3% of their total capacity. Running the balancer makes no difference. Why is it that only one disk is getting full and the others are barely used? Please help.

3 REPLIES 3
Highlighted

Re: Data Node host has one disk full

Master Guru
What is the output of below, when run within the DN directory path under /dfs/data01?

find . -type f | grep dncp | xargs ls -lh

Also, what version of CDH5 are you on?
Highlighted

Re: Data Node host has one disk full

Explorer

@Harsh J wrote:
What is the output of below, when run within the DN directory path under /dfs/data01?

find . -type f | grep dncp | xargs ls -lh

Also, what version of CDH5 are you on?

Hi Harsh

 

We are also facing the similer issue. One of the disk in the datanode is 100% and all other disks are around 75% used.

 

Please find the output below.

 

[root@r03wn17:/data/1]# find . -type f | grep dncp | xargs ls -lh
-rw-r--r-- 1 hdfs hdfs 509G May 17 11:02 ./dfs/dn/current/BP-396968526-10.37.42.241-1400126960303/dncp_block_verification.log.curr
-rw-r--r-- 1 hdfs hdfs 103G Apr 28 22:01 ./dfs/dn/current/BP-396968526-10.37.42.241-1400126960303/dncp_block_verification.log.prev

 

CDH version : 

 

[root@r03wn17:/data/1]# hadoop version
Hadoop 2.5.0-cdh5.3.8
Subversion http://github.com/cloudera/hadoop -r fe44e341b92cd4e63b4002b87e352f4205487972
Compiled by jenkins on 2015-10-15T00:26Z
Compiled with protoc 2.5.0
From source with checksum c17117c9ef53cb4be302ba73418afbd
This command was run using /opt/cloudera/parcels/CDH-5.3.8-1.cdh5.3.8.p0.5/jars/hadoop-common-2.5.0-cdh5.3.8.jar

 

Thanks

Vijith

Highlighted

Re: Data Node host has one disk full

Master Guru
The nature of the bug around the dncp files in 5.3.x match precisely your
situation. The first disk on the DN list will have the large growing file.
You can shut down the DN and safely delete both files and restart it.

This is fixed in 5.4 and above. See
https://issues.apache.org/jira/browse/HDFS-7532 for more.
Don't have an account?
Coming from Hortonworks? Activate your account here