question Re: Volume failure reported while disks seem fine in Archives of Support Questions (Read Only)

Volume failure reported while disks seem fine

Knut N — Tue, 16 Dec 2014 13:00:16 GMT

I just upgraded our cluster from CDH 5.0.1 to 5.2.1, using parcels and following the provided instructions.

After the upgrade has finished, the health test "Data Directory Status" is critical for one of the data nodes. The reported error message is "The DataNode has 1 volume failure(s)". By running 'hdfs dfsadmin -report' I can also confirm that the available HDFS space on that node is approximately 4 TB less than on the other nodes, indicating that one of the disks is not being used.

However, when checking the status of the actual disks and regular file system we can not find anything that seems wrong. All disks are mounted and seem to be working as they should. There is also an in_use.lock file in the dfs/nn directory on all of the disks.

How can I get more detailed information about which volume the DataNode is complaining about, and what the issue might be?

Best Regards

\Knut Nordin

Re: Volume failure reported while disks seem fine

GautamG — Wed, 17 Dec 2014 08:25:53 GMT

Couple of places you can check:
- Host Inspector in CM
- CM agent logs under /var/log/cloudera-scm-agent

Re: Volume failure reported while disks seem fine

Knut N — Wed, 17 Dec 2014 08:49:10 GMT

Thank you for the suggestions!

Host Inspector reports that everything is OK. I have checked the agent logs and also the log for the DataNode, but can not really find anything that gives any clues about this particular error.

Is there a command for making the DataNode report which physical directories it is using and what it thinks their status is?

Re: Volume failure reported while disks seem fine

Knut N — Thu, 18 Dec 2014 10:44:25 GMT

Not sure if this is helpful, but here are some charts of volume failures and disk capacity the faultly node, and for a healthy one as reference. The strange thing is that all disks seem to be in use on both nodes, but the total configured capacity is different.

Knut

Re: Volume failure reported while disks seem fine

GautamG — Fri, 19 Dec 2014 00:49:22 GMT

For a start, ensure your Cloudera Manager is also 5.2.1. We informally required that the versions be matched but it is now a requirement from 5.2 onwards. CM version should be equal or higher than CDH version.

Next, go to Cloudera Management services -> even search -> critical. See if the volume related error is listed. If so, please paste details here.

Re: Volume failure reported while disks seem fine

Knut N — Fri, 19 Dec 2014 07:53:01 GMT

I can confirm that CM is running version 5.2.1, so that should not be an issue.

When searching for critical log messages I came across these:

It appears that another namenode 56559@datanode_hostname has already locked the storage directory

datanode_hostname:50010:DataXceiver error processing WRITE_BLOCK operation src: /10.8.19.28:37029 dst: /10.8.19.14:50010

We have only ever had one namenode, and none on the datanode host, so I am a bit confused by what the first message really means. The second one I suspect might be a secondary failure of the first one?

Regards

Knut

Re: Volume failure reported while disks seem fine

Knut N — Fri, 02 Jan 2015 08:14:37 GMT

The source of this error has been found. It turned out that /etc/fstab on this node was badly configured, so that one of the disks was mounted twice as two separate data directories. Interestingly, this has not been causing any visible errors until upgrading to CDH 5.2.1. Nice that it was pointed out to us by this version though.

Re: Volume failure reported while disks seem fine

GautamG — Fri, 02 Jan 2015 08:24:29 GMT

Glad the issue is resolved, thanks for letting us know.

Re: Volume failure reported while disks seem fine

zeeshan.khan — Tue, 17 May 2016 14:13:38 GMT

Hi,

can you tell us please how it was pointed out ? where to check if this is the case ?

regards

Re: Volume failure reported while disks seem fine

bramd — Sat, 10 Dec 2016 23:28:00 GMT

can you tell us please how it was pointed out ? where to check if this is the case ?

Here's one way of checking for duplicate mounts:

mount |wc -l > a
mount |sort -u|uniq|wc -l > b
cmp a b

If there's no output, there are no duplicate mounts.

HTH

Re: Volume failure reported while disks seem fine

SSH007 — Mon, 12 Feb 2018 13:48:07 GMT

In my case I did moved the existing dn directory as it has very less data and started the services.

There was no disk failure or issue with fstab file or mounting.

Re: Volume failure reported while disks seem fine

Lambzee — Wed, 11 Sep 2019 22:40:00 GMT

Yes even in my case fstab issue was not there. Rather i could clearly find the issue from datanode logs under /var/log/ and fix the issue. I have a blog on the same as below. Please comment on blog if it helps.

https://itdoctorharsha.wordpress.com/2019/08/23/hadoop-data-volume-failures-and-solution-cloudera/