Community Articles

Find and share helpful community-sourced technical articles.
Labels (2)

Virtual Memory swapping can have a large impact on the performance of a Hadoop system. Because of the memory requirements of YARN containers and processes running on the nodes in a cluster, swapping process out of memory to disk can cause serious performance limitations. As such, the historical recommendations for setting the swappiness, or propensity to swap out a process, on a Hadoop system has been to disable swap altogether. With newer versions of the Linux kernel, Out Of Memory (OOM) situations can be more likely to indiscriminately kill important processes to reclaim valuable physical memory on the system with a swappiness of 0.

In order to prevent the system from swapping processes too frequently, but still allow for emergency swapping (instead of killing processes), the recommendation is now to set swappiness to 1 on Linux systems. This will still allow swapping, but with the least possible aggressiveness (for comparison, the default value for swappiness is 60).

To change the swappiness on a running machine, use the following command:

echo "1" > /proc/sys/vm/swappiness

To ensure the swappiness is set appropriately on reboot, use the following command:

echo "vm.swappiness=1" >> /etc/sysctl.conf
16,147 Views
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.
Version history
Last update:
‎09-16-2022 01:34 AM
Updated by:
Contributors
Top Kudoed Authors