Why do we allocate 32 GB (recommended) for /dev partition to the Hadoop master node

[#1]

Hi All,

We have 2* 80 GB 6G SATA SSD RAID-1 Hard-drive for Master node, According to Hortonworks recommendation in one of the blog-post, it suggests 32 GB for /dev partition. Our client is not interested in providing 32 GB for /dev partition, since the maximum disk capacity is 2*80 GB.

Full Master configuration details:

2x E5-2670V2 2.5GHz-25MB 10C CPU

448GB PC3-14900L RAM

2x 80GB 6G SATA SSD RAID 1

I would like to ask why we need 32 GB for /dev partition for Hadoop master nodes, also please suggest the recommended partition by considering the above hard-drive limits.

Thanks !

[#2]

Hi @Uday Vakalapudi,

HDP do not require any special sizing for /dev as long as we follow the OS recommendations. The "baseline" we present includes general Linux requirements for system partitions. So in this case I recommend to follow the requirements for /dev issued by the operating system vendor.

[#3]

Hi @Mats Johansson,

Thanks for your reply!

Yes, /dev directory contains the special device files for all the devices that's it. However, do we need to explicitly mention any sizing numbers for the partitions /dev/sda1, /dev/sda2 ..etc. What kind of information will store under those /dev partitions w.r.t HDP configuration setup.

Thanks!

[#4 · Accepted solution]

I found it on RedHat documentation, There is no one right answer to this question. It depends on your needs and requirements. 😞

/dev/:

This is the name of the directory in which all device files reside. Because partitions reside on hard disks, and hard disks are devices, the files representing all possible partitions reside in /dev/

Red Hat Enterprise Linux uses a naming scheme that is file-based, with file names in the form of /dev/xxyN.

Where,

xx:

The first two letters of the partition name indicate the type of device on which the partition resides, usually sd.

y:

This letter indicates which device the partition is on. For example, /dev/sda for the first hard disk, /dev/sdb for the second, and so on.

N:

The final number denotes the partition. The first four (primary or extended) partitions are numbered 1 through 4. Logical partitions start at 5. So, for example, /dev/sda3 is the third primary or extended partition on the first hard disk, and /dev/sdb6 is the second logical partition on the second hard disk.

In Red Hat Enterprise Linux each partition is used to form part of the storage necessary to support a single set of files and directories. Mounting a partition makes its storage available starting at the specified directory (known as a mount point).

For example, if partition /dev/sda5 is mounted on /usr/, that would mean that all files and directories under /usr/ physically reside on /dev/sda5. So the file /usr/share/doc/FAQ/txt/Linux-FAQ would be stored on /dev/sda5, while the file /etc/gdm/custom.conf would not. It is also possible that one or more directories below /usr/ would be mount points for other partitions. For instance, a partition (say, /dev/sda7) could be mounted on /usr/local/, meaning that /usr/local/man/whatis would then reside on /dev/sda7 rather than /dev/sda5.

Generally speaking, the disk spacing for /dev partition depends on number and size of the partitions (both primary and logical)to be used by operating system. However, there is no one right answer to this question. It depends on your needs and requirements.

Conclusion: Red Hat recommends that, unless you have a reason for doing otherwise, you should at least create the following partitions: swap, /boot/, and / (root).