swap warning

[#1]

Hello Cloudera Support,

We are receiving warnings in Cloudera Manager related to swapping activity on one or more cluster nodes. However, swap has already been disabled across all our hosts, and we've verified that no active swap space is configured.

Despite this, Cloudera Manager continues to raise the warning.

We would appreciate your assistance in clarifying the following:

1. What conditions or metrics trigger this warning?

2. Are there additional configurations or steps we should take to prevent this alert?

Please let us know if any diagnostics or host information is required from our end.

Thank you for your support.

(Swap has been disabled across all nodes via OS configuration.)

Best regards,
Modhi Alrashed

[#2]

Hello @Malrashed 

Thank you for reaching out to the community

Could you please confirm what services are giving a swap warning? If possible please attach a screenshot to further understand the issue

[#3]

Hello @upadhyayk04 ,

All the services affected except Hue ,hive, hive on Tez , Yarn (MR2 included) and Nifi

please find attached screenshot.

[#4]

hi @Malrashed ,

did you completely disable swap at the operating system level?

comment out the swap line in the fstab file.
run the swap off command.
the free command returns swap 0 (zero).

after disabling, did you restart the Cloudera agent?

[#5]

Hello @Malrashed 

In addition to the last update

Also, check the current threshold you have set for the process . There could be a possibility that it could be set to a very low value like KB's if that is the case then you can tweak those

CM UI >> HDFS >> Configuration >> Process Swap Memory Thresholds

Also, share the output of free -mh to check the free memory and swap consumed. You might need to swapoff and swapon

[#6]

dears @upadhyayk04 @yagoaparecidoti 

I would like to ask about the recommendations from Cloudera about :

1- What is the optimal swap value for all services?

2- Is it recommended to enable swap? if so, under what conditions?

Best Regards,

[#7]

Hello @Malrashed 

Thank you for your update

Are you talking about enabling swap threshold on services or actual swap on operating system?

[#8]

Dear @upadhyayk04 

Thank you for your response. I was actually referring to both.

[#9]

hi @Malrashed,

we do not use swap on Cloudera cluster hosts.

we disable 100% or keep the value 1 in the vm.swappiness parameter.

reference:
https://docs.cloudera.com/cdp-reference-architectures/latest/cdp-pvc-base-ra/topics/ra-cdpdc-swapine...

best regards 🙂

[#10]

best regards 🙂

[#11]

Hello @Malrashed 

Thank you for your update

I think my colleague has already shared the answer to your query. In general for host level swap configuration you can refer the below redhat document

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/storage_administration_guid...

if swap is available on a host in linux, the Kernel can elect to store things in swap at any time. The system does not have to be out of memory for the Kernel place. As an example a significant number of the pages used by an application during its startup phase may only be used for initialisation and then never used again. The system can swap out those pages and free the memory for other applications or even for the disk cache. That being said sometimes running into swap space is un-avoidable unless the swapiness value is set to 0. Cloudera Manager is set to alert on any usage at all. Cloudera officially recommends setting the swapiness value to 1, but this can lead to the Swap Memory alerts that you're experiencing.

POSSIBLE SOLUTIONS:
There are two options we have. We can raise the warning threshold, or change the swapiness value to 0. It's really up to you, but most customers raise the warning threshold because with newer versions of the Linux kernel, the swappiness value of 0 now means to "never swap" and in Out of Memory (OOM) situations the kernel is more likely to indiscriminately kill important processes needed reclaim RAM. This may be fine on worker nodes but certainly not Master nodes.

In order to fix the error you can do one of the following:

To Raise the Threshold:

1.) CM => Search for "host_memswap_thresholds" (Host Memory Swapping Thresholds) => Specify for both Warning and Critical and put the following:

Warning: 51200
Critical : 102400

The thresholds are specified in pages (256 = 1MB)
So for warning the Memory limit = 200 MB and Critical Memory limit = 400 MB


To Change Swapiness Value

1.) In order to change the swap space on a running machine you must issue first the following:
echo "0" > /proc/sys/vm/swappiness

2.) To make the swap parameter persistent upon a reboot, use the following command:
echo "vm.swappiness=0" >> /etc/sysctl.conf
(NOTE: Also vi the sysctl.conf file and delete any vm.swappiness=1 entries)

3.) Issue the command: swapoff -a then swapon -a
This will turn off swap space completely and get rid of the warnings, then turn swap back on.


NOTE : As per the official document [Ref1] Cloudera recommends swapiness to 1 so please check on that in your environment.

[Ref1] : https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/managing-clusters/topics/cm-setting-vmswappin...

I hope this helps

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.