Created on 12-06-2018 09:39 PM
Performance of a cluster running on Hadoop can be impacted by the OS partitioning. This document is intended to understand the best practices to setup the “/var” folder/partition with optimum size.
Lets try to approach this problem by asking some important questions.
From OS perspective, “/var” is commonly used for constantly changing files i.e. variable. The short form of which is “var”.
Example of such files could be the log file, mail, transient file, the printer spool, temporary files, cached data, etc.
For example - “/var/tmp” holds the temporary files between system reboots.
On any node (Hadoop or non-Hadoop), /var directory holds content for a number of applications. It also is used to store downloaded update packages on a temporary basis.
The PackageKit update software downloads updated packages to /var/cache/yum/ by default. /var/ partition should be large enough to download package updates.
An example of application which uses /var is MySql, which by default uses “/var/lib/mysql” as the MySql directory location.
Question 2 - How can /var folder run out of disk space?
/var is much more susceptible to filling up - by accident or by attack.
Some of the directories which can be affected by this is “/var/log”, “/var/tmp”, “/var/crash” etc.
If there is a serious OS issue, the logging can increase tremendously. If the disk space is set too low, like 10GB, this excessive logging can fill in the “disk” space for /var.
Question 3 - Common issue to expect on a Hadoop cluster if “/var” is out of disk space.
/var has been seen to be easily filled by a (possibly misbehaved) application, and that if it wasn't separate from /, the filling of / could cause a kernel panic.
“/var” folder has some very important file/folders locations which are used by default by many kernel and OS applications.
For example –
“/var” is very important from Hadoop perspective to keep all the service running. Running out of Disk space on “/var” can cause Hadoop and dependent services to fail to run on that node.Question 4 - How is the setup of “/var” in the clusters on my cluster?
The size of “/var” therefore plays important role if /var/crash can be too low for saving the “crashdump” logs.
If there is a OS crash (Kernel Panic etc.) then the crashdump will never be captured complete, since the size of “/var” is too low i.e. 10 GB or 50GB. Without the complete crashdump logs, there can never be a complete analysis of the cause of Kernel Crash.Answer - Recommendations on the optimum setup of “/var”.