What causes a datanode to consider a volume as failed

[#1]

Terribly sorry, haven't seen that comment. Then, DN couldn't detect your disk as failed.

[#2]

@Thomas Larsson

If there are some folders which goes in Read-Only mode and the hard-disk still in mounted as a Read-Write, then this indicates that the mounting options for the filesystem needs to be verified.Check the mounting options which is set for the Harddisk. The harddisk should go in complete Read-Only mode when there is some I/O issues with some of the folders and partitions on the Harddisk.

HDFS will also see the Harddisk in read only mode and then based on the property value - "dfs.datanode.failed.volumes.tolerated" will act accordingly i.e. if the value for dfs.datanode.failed.volumes.tolerated is 0, it will stop the datanode process as soon as it find the Harddisk in Read-Only mode.

[#3]

Hi Ravi,

I'm not sure I understand what you mean. Is there a tool that could detect our type of disk error and automatically remount the drive in read-only mode? Or are you talking about something like the fstab mount options "errors=remount -ro"? The fstab options only means that if errors are encountered when the os tries to mount the drive for read-write mode, it should try to mount it as read-only. But this does not apply to our situation since our machine is not just starting up, its been up and running for a long while and then disk errors start to occur. If you mean some other tool or configuration that can detect and remount while a system is running, please share a link.

Best Regards

[#4]

@Thomas Larsson

The filesystem, when sees any Hard-disk error, should go into Read-Only mode. If it does not goes into read-only mode that is possible due to the mounting options.

The filesystem should be mounted with the option errors=remount-ro . This means that if an error is detected, the filesystem will go into read-only.

If the filesystem goes into Read-Only mode, then HDFS will also identify the same. Without OS identifying the filesystem as "read-only" HDFS will not be able to identify the same.

[#5 · Accepted solution]

Hi @Thomas Larsson, the DataNode will perform a simple disk check operation in response to certain IO errors. The disk check verifies that the DataNode's storage directory root is readable, writeable and executable. If either of these checks fails, the DataNode will mark the volume as failed.

HDFS failed disk detection can be better than it is today. We have seen instances where these checks are insufficient to detect volume failures. It is a hard problem in general since disks fail in byzantine ways where some but not all IOs may fail or a subset of directories on the disk become inaccessible.

[#6]

thanks nice

[#7]

Hi @Arpit Agarwal,

That is my understanding as well. Thanks for a short and to the point answer.