HDFS DN is writing into root-vg after disk failure

[#1]

Hey there, 

we're running a CDP PC Base Cluster in our datacenter. In the last months we saw serveral disks failing due to theire age. The Problem we had, is that, when a disk fails, HDFS DN and YARN NM creating directories in the root-vg of the node. During normal operations the DN has it directories on the grids (/grid/[0-16]/*).

Is there a parameter to prevent them of writing into the root-vg, when a grid points to the root-vg and not it's physical device under /dev/sd*?

Regards, 

Timo

[#2 · Accepted solution]

Hi @Timo ,

In Apache Hadoop, the directories where HDFS DataNodes and YARN NodeManagers store their data and logs are typically configured using the "dfs.datanode.data.dir" and "yarn.nodemanager.local-dirs" properties respectively.

To prevent HDFS DataNodes and YARN NodeManagers from writing data to the root-vg directory when disks fail, you should ensure that these properties are configured correctly to point to directories on the healthy disks or storage volumes.

-> Configure HDFS DataNode Data Directories:Set the "dfs.datanode.data.dir" property in "hdfs-site.xml" to specify the directories where HDFS DataNodes should store their data. Make sure to list the directories on the healthy disks or storage volumes.

-> Configure YARN NodeManager Local Directories:

Set the "yarn.nodemanager.local-dirs" property in "yarn-site.xml" to specify the directories where YARN NodeManagers should store their local data and logs. Again, ensure that these directories are on the healthy disks or storage volumes.

Regards,

Chethan YM

[#3]

@Timo, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.