swap warning

Hello @Malrashed 

Thank you for reaching out to the community

Could you please confirm what services are giving a swap warning? If possible please attach a screenshot to further understand the issue

Hello @upadhyayk04 ,

All the services affected except Hue ,hive, hive on Tez , Yarn (MR2 included) and Nifi

please find attached screenshot.

Hello @Malrashed 

In addition to the last update

Also, check the current threshold you have set for the process . There could be a possibility that it could be set to a very low value like KB's if that is the case then you can tweak those

CM UI >> HDFS >> Configuration >> Process Swap Memory Thresholds

Also, share the output of free -mh to check the free memory and swap consumed. You might need to swapoff and swapon

dears @upadhyayk04 @yagoaparecidoti 

I would like to ask about the recommendations from Cloudera about :

1- What is the optimal swap value for all services?

2- Is it recommended to enable swap? if so, under what conditions?

Best Regards,

Hello @Malrashed 

Thank you for your update

Are you talking about enabling swap threshold on services or actual swap on operating system?

hi @Malrashed,

we do not use swap on Cloudera cluster hosts.

we disable 100% or keep the value 1 in the vm.swappiness parameter.

reference:
https://docs.cloudera.com/cdp-reference-architectures/latest/cdp-pvc-base-ra/topics/ra-cdpdc-swapiness.html

Hello @Malrashed 

Thank you for your update

I think my colleague has already shared the answer to your query. In general for host level swap configuration you can refer the below redhat document

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-swapspace

if swap is available on a host in linux, the Kernel can elect to store things in swap at any time. The system does not have to be out of memory for the Kernel place. As an example a significant number of the pages used by an application during its startup phase may only be used for initialisation and then never used again. The system can swap out those pages and free the memory for other applications or even for the disk cache. That being said sometimes running into swap space is un-avoidable unless the swapiness value is set to 0. Cloudera Manager is set to alert on any usage at all. Cloudera officially recommends setting the swapiness value to 1, but this can lead to the Swap Memory alerts that you're experiencing.

POSSIBLE SOLUTIONS:
There are two options we have. We can raise the warning threshold, or change the swapiness value to 0. It's really up to you, but most customers raise the warning threshold because with newer versions of the Linux kernel, the swappiness value of 0 now means to "never swap" and in Out of Memory (OOM) situations the kernel is more likely to indiscriminately kill important processes needed reclaim RAM. This may be fine on worker nodes but certainly not Master nodes.

In order to fix the error you can do one of the following:

To Raise the Threshold:

1.) CM => Search for "host_memswap_thresholds" (Host Memory Swapping Thresholds) => Specify for both Warning and Critical and put the following:

Warning: 51200
Critical : 102400

The thresholds are specified in pages (256 = 1MB)
So for warning the Memory limit = 200 MB and Critical Memory limit = 400 MB


To Change Swapiness Value

1.) In order to change the swap space on a running machine you must issue first the following:
echo "0" > /proc/sys/vm/swappiness

2.) To make the swap parameter persistent upon a reboot, use the following command:
echo "vm.swappiness=0" >> /etc/sysctl.conf
(NOTE: Also vi the sysctl.conf file and delete any vm.swappiness=1 entries)

3.) Issue the command: swapoff -a then swapon -a
This will turn off swap space completely and get rid of the warnings, then turn swap back on.


NOTE : As per the official document [Ref1] Cloudera recommends swapiness to 1 so please check on that in your environment.

[Ref1] : https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/managing-clusters/topics/cm-setting-vmswappiness-linux-kernel-parameter.html

I hope this helps

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Dear @upadhyayk04 ,

I will raise the threshold to be as what you recommended, but the host threshold already 200 ( warning).

Can I change the Process Swap Memory Threshold? to fix my issue ?