question Re: dfs.data.dir question in Archives of Support Questions (Read Only)

dfs.data.dir question

SathishNarayanan — Fri, 16 Sep 2022 08:49:14 GMT

If I have several slave nodes with varying number of drives mounted in each of them, then how does dfs.data.dir property setup works? If I include the directory name that is available in one of the datanode that is not available in other in this property, then does hadoop skip this value/drive for that particular datanode where this directory is missing?

Thanks in advance!

Re: dfs.data.dir question

Bryan — Fri, 18 Oct 2013 15:49:11 GMT

No, this isn't how it works. If you use the same configuration:

/mnt, /mnt2, /mnt3, /mnt4, /mnt5

A host who just has 3 drives (/mnt, /mnt2, /mnt3) will fail to start, depending on the value of dfs.datanode.failed.volumes.tolerated (default 0). You're going to need to set up each server properly with the right value for dfs.data.dir. For this (and other) reason(s), a homogenous cluster setup is usually preferred.